Package org.apache.beam.sdk.transforms

Class WithFailures

  • @Experimental(WITH_EXCEPTIONS)
public class WithFailures
extends java.lang.Object
    A collection of utilities for writing transforms that can handle exceptions raised during processing of elements.

    Consuming transforms such as MapElements.MapWithFailures follow the general pattern of taking in a user-defined exception handler of type ProcessFunction<ExceptionElement<InputT>, FailureOutputT> where the input WithFailures.ExceptionElement contains an exception along with the input element that was being processed when the exception was raised. This handler is responsible for producing some output element that captures relevant details of the failure and can be encoded as part of a failure output PCollection. Transforms can then package together their output and failure collections in a WithFailures.Result that avoids users needing to interact with TupleTags and indexing into a PCollectionTuple.

    Exception handlers can narrow their scope by rethrowing the passed WithFailures.ExceptionElement.exception() and catching only specific subclasses of Exception. Unhandled exceptions will generally bubble up to a top-level Pipeline.PipelineExecutionException that halts progress.

    Users can take advantage of WithFailures.Result.failuresTo(List) for fluent chaining of transforms that handle exceptions: 

    
 PCollection<Integer> input = ...
 List<PCollection<Map<String, String>> failureCollections = new ArrayList<>();
 input.apply(MapElements.via(...).exceptionsVia(...))
      .failuresTo(failureCollections)
      .apply(MapElements.via(...).exceptionsVia(...))
      .failuresTo(failureCollections);
 PCollection<Map<String, String>> failures = PCollectionList.of(failureCollections)
      .apply("FlattenFailureCollections", Flatten.pCollections());

    • Nested Class Summary

      Nested Classes 
      Modifier and Type Class Description
      static class  WithFailures.ExceptionAsMapHandler<T>
      A simple handler that extracts information from an exception to a Map<String, String> and returns a KV where the key is the input element that failed processing, and the value is the map of exception attributes.
      static class  WithFailures.ExceptionElement<T>
      The value type passed as input to exception handlers.
      static class  WithFailures.Result<OutputT extends POutput,​FailureElementT>
      An intermediate output type for PTransforms that allows an output collection to live alongside a collection of elements that failed the transform.
      static class  WithFailures.ThrowableHandler<T>
      A handler that holds onto the Throwable that led to the exception, returning it along with the original value as a KV.

    • Constructor Summary

      Constructors 
      Constructor Description
      WithFailures()  

    • Method Summary

      • Methods inherited from class java.lang.Object

        clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

    • Constructor Detail

      • WithFailures

        public WithFailures()