Package org.apache.beam.sdk.schemas.transforms

Class Group

  • @Experimental(SCHEMAS)
public class Group
extends java.lang.Object
    A generic grouping transform for schema PCollections.

    When used without a combiner, this transforms simply acts as a GroupByKey but without the need for the user to explicitly extract the keys. For example, consider the following input type: 

    @DefaultSchema(JavaFieldSchema.class)
 public class UserPurchase {
   public String userId;
   public String country;
   public long cost;
   public double transactionDuration;
 }

 PCollection<UserPurchase> purchases = readUserPurchases();

    You can group all purchases by user and country as follows: 

    @DefaultSchema(JavaFieldSchema.class)
 PCollection<Row> byUser = purchases.apply(Group.byFieldNames("userId', "country"));

    However often an aggregation of some form is desired. The builder methods inside the Group class allows building up separate aggregations for every field (or set of fields) on the input schema, and generating an output schema based on these aggregations. For example: 

    
 PCollection<Row> aggregated = purchases
      .apply(Group.byFieldNames("userId', "country")
          .aggregateField("cost", Sum.ofLongs(), "total_cost")
          .aggregateField("cost", Top.<Long>largestLongsFn(10), "top_purchases")
          .aggregateField("cost", ApproximateQuantilesCombineFn.create(21),
              Field.of("transactionDurations", FieldType.array(FieldType.INT64)));

    The result will be a new row schema containing the fields total_cost, top_purchases, and transactionDurations, containing the sum of all purchases costs (for that user and country), the top ten purchases, and a histogram of transaction durations. The schema will also contain a key field, which will be a row containing userId and country.

    Note that usually the field type can be automatically inferred from the Combine.CombineFn passed in. However sometimes it cannot be inferred, due to Java type erasure, in which case a Schema.Field object containing the field type must be passed in. This is currently the case for ApproximateQuantilesCombineFn in the above example.

    • Constructor Detail

      • Group

        public Group()

    • Method Detail

      • globally

        public static <T> Group.Global<T> globally()
        Returns a transform that groups all elements in the input PCollection. The returned transform contains further builder methods to control how the grouping is done.

      • byFieldNames

        public static <T> Group.ByFields<T> byFieldNames​(java.lang.String... fieldNames)
        Returns a transform that groups all elements in the input PCollection keyed by the list of fields specified. The output of this transform will be a KV keyed by a Row containing the specified extracted fields. The returned transform contains further builder methods to control how the grouping is done.

      • byFieldIds

        public static <T> Group.ByFields<T> byFieldIds​(java.lang.Integer... fieldIds)
        Returns a transform that groups all elements in the input PCollection keyed by the list of fields specified. The output of this transform will have a key field of type Row containing the specified extracted fields. It will also have a value field of type Row containing the specified extracted fields. The returned transform contains further builder methods to control how the grouping is done.

      • byFieldAccessDescriptor

        public static <T> Group.ByFields<T> byFieldAccessDescriptor​(FieldAccessDescriptor fieldAccess)
        Returns a transform that groups all elements in the input PCollection keyed by the fields specified. The output of this transform will have a key field of type Row containing the specified extracted fields. It will also have a value field of type Row containing the specified extracted fields. The returned transform contains further builder methods to control how the grouping is done.