Package org.apache.flink.streaming.api.graph

Class StreamGraph

  • All Implemented Interfaces:
    org.apache.flink.api.dag.Pipeline
    @Internal
public class StreamGraph
extends Object
implements org.apache.flink.api.dag.Pipeline
    Class representing the streaming topology. It contains all the information necessary to build the jobgraph for the execution.

    • Constructor Detail

      • StreamGraph

        public StreamGraph​(org.apache.flink.configuration.Configuration jobConfiguration,
                   org.apache.flink.api.common.ExecutionConfig executionConfig,
                   CheckpointConfig checkpointConfig,
                   SavepointRestoreSettings savepointRestoreSettings)

    • Method Detail

      • clear

        public void clear()
        Remove all registered nodes etc.

      • getExecutionConfig

        public org.apache.flink.api.common.ExecutionConfig getExecutionConfig()

      • getJobConfiguration

        public org.apache.flink.configuration.Configuration getJobConfiguration()

      • setSavepointRestoreSettings

        public void setSavepointRestoreSettings​(SavepointRestoreSettings savepointRestoreSettings)

      • getJobName

        public String getJobName()

      • setJobName

        public void setJobName​(String jobName)

      • setLineageGraph

        public void setLineageGraph​(LineageGraph lineageGraph)

      • setStateBackend

        public void setStateBackend​(StateBackend backend)

      • setCheckpointStorage

        public void setCheckpointStorage​(CheckpointStorage checkpointStorage)

      • getUserArtifacts

        public Collection<org.apache.flink.api.java.tuple.Tuple2<String,​org.apache.flink.api.common.cache.DistributedCache.DistributedCacheEntry>> getUserArtifacts()

      • setGlobalStreamExchangeMode

        public void setGlobalStreamExchangeMode​(GlobalStreamExchangeMode globalExchangeMode)

      • setSlotSharingGroupResource

        public void setSlotSharingGroupResource​(Map<String,​ResourceProfile> slotSharingGroupResources)

      • hasFineGrainedResource

        public boolean hasFineGrainedResource()

      • setAllVerticesInSameSlotSharingGroupByDefault

        public void setAllVerticesInSameSlotSharingGroupByDefault​(boolean allVerticesInSameSlotSharingGroupByDefault)
        Set whether to put all vertices into the same slot sharing group by default.
        Parameters:
        allVerticesInSameSlotSharingGroupByDefault - indicates whether to put all vertices into the same slot sharing group by default.

      • isAllVerticesInSameSlotSharingGroupByDefault

        public boolean isAllVerticesInSameSlotSharingGroupByDefault()
        Gets whether to put all vertices into the same slot sharing group by default.
        Returns:
        whether to put all vertices into the same slot sharing group by default.

      • isEnableCheckpointsAfterTasksFinish

        public boolean isEnableCheckpointsAfterTasksFinish()

      • setEnableCheckpointsAfterTasksFinish

        public void setEnableCheckpointsAfterTasksFinish​(boolean enableCheckpointsAfterTasksFinish)

      • isChainingEnabled

        public boolean isChainingEnabled()

      • isChainingOfOperatorsWithDifferentMaxParallelismEnabled

        public boolean isChainingOfOperatorsWithDifferentMaxParallelismEnabled()

      • isIterative

        public boolean isIterative()

      • addSource

        public <IN,​OUT> void addSource​(Integer vertexID,
                                     @Nullable
                                     String slotSharingGroup,
                                     @Nullable
                                     String coLocationGroup,
                                     SourceOperatorFactory<OUT> operatorFactory,
                                     org.apache.flink.api.common.typeinfo.TypeInformation<IN> inTypeInfo,
                                     org.apache.flink.api.common.typeinfo.TypeInformation<OUT> outTypeInfo,
                                     String operatorName)

      • addLegacySource

        public <IN,​OUT> void addLegacySource​(Integer vertexID,
                                           @Nullable
                                           String slotSharingGroup,
                                           @Nullable
                                           String coLocationGroup,
                                           StreamOperatorFactory<OUT> operatorFactory,
                                           org.apache.flink.api.common.typeinfo.TypeInformation<IN> inTypeInfo,
                                           org.apache.flink.api.common.typeinfo.TypeInformation<OUT> outTypeInfo,
                                           String operatorName)

      • addSink

        public <IN,​OUT> void addSink​(Integer vertexID,
                                   @Nullable
                                   String slotSharingGroup,
                                   @Nullable
                                   String coLocationGroup,
                                   StreamOperatorFactory<OUT> operatorFactory,
                                   org.apache.flink.api.common.typeinfo.TypeInformation<IN> inTypeInfo,
                                   org.apache.flink.api.common.typeinfo.TypeInformation<OUT> outTypeInfo,
                                   String operatorName)

      • addOperator

        public <IN,​OUT> void addOperator​(Integer vertexID,
                                       @Nullable
                                       String slotSharingGroup,
                                       @Nullable
                                       String coLocationGroup,
                                       StreamOperatorFactory<OUT> operatorFactory,
                                       org.apache.flink.api.common.typeinfo.TypeInformation<IN> inTypeInfo,
                                       org.apache.flink.api.common.typeinfo.TypeInformation<OUT> outTypeInfo,
                                       String operatorName)

      • addCoOperator

        public <IN1,​IN2,​OUT> void addCoOperator​(Integer vertexID,
                                                    String slotSharingGroup,
                                                    @Nullable
                                                    String coLocationGroup,
                                                    StreamOperatorFactory<OUT> taskOperatorFactory,
                                                    org.apache.flink.api.common.typeinfo.TypeInformation<IN1> in1TypeInfo,
                                                    org.apache.flink.api.common.typeinfo.TypeInformation<IN2> in2TypeInfo,
                                                    org.apache.flink.api.common.typeinfo.TypeInformation<OUT> outTypeInfo,
                                                    String operatorName)

      • addMultipleInputOperator

        public <OUT> void addMultipleInputOperator​(Integer vertexID,
                                           String slotSharingGroup,
                                           @Nullable
                                           String coLocationGroup,
                                           StreamOperatorFactory<OUT> operatorFactory,
                                           List<org.apache.flink.api.common.typeinfo.TypeInformation<?>> inTypeInfos,
                                           org.apache.flink.api.common.typeinfo.TypeInformation<OUT> outTypeInfo,
                                           String operatorName)

      • addVirtualSideOutputNode

        public void addVirtualSideOutputNode​(Integer originalId,
                                     Integer virtualId,
                                     org.apache.flink.util.OutputTag outputTag)
        Adds a new virtual node that is used to connect a downstream vertex to only the outputs with the selected side-output OutputTag.
        Parameters:
        originalId - ID of the node that should be connected to.
        virtualId - ID of the virtual node.
        outputTag - The selected side-output OutputTag.

      • addVirtualPartitionNode

        public void addVirtualPartitionNode​(Integer originalId,
                                    Integer virtualId,
                                    StreamPartitioner<?> partitioner,
                                    StreamExchangeMode exchangeMode)
        Adds a new virtual node that is used to connect a downstream vertex to an input with a certain partitioning.

        When adding an edge from the virtual node to a downstream node the connection will be made to the original node, but with the partitioning given here.

        Parameters:
        originalId - ID of the node that should be connected to.
        virtualId - ID of the virtual node.
        partitioner - The partitioner

      • getSlotSharingGroup

        public String getSlotSharingGroup​(Integer id)
        Determines the slot sharing group of an operation across virtual nodes.

      • addEdge

        public void addEdge​(Integer upStreamVertexID,
                    Integer downStreamVertexID,
                    int typeNumber)

      • setParallelism

        public void setParallelism​(Integer vertexID,
                           int parallelism)

      • isDynamic

        public boolean isDynamic()

      • setParallelism

        public void setParallelism​(Integer vertexId,
                           int parallelism,
                           boolean parallelismConfigured)

      • setDynamic

        public void setDynamic​(boolean dynamic)

      • setMaxParallelism

        public void setMaxParallelism​(int vertexID,
                              int maxParallelism)

      • setResources

        public void setResources​(int vertexID,
                         org.apache.flink.api.common.operators.ResourceSpec minResources,
                         org.apache.flink.api.common.operators.ResourceSpec preferredResources)

      • setManagedMemoryUseCaseWeights

        public void setManagedMemoryUseCaseWeights​(int vertexID,
                                           Map<org.apache.flink.core.memory.ManagedMemoryUseCase,​Integer> operatorScopeUseCaseWeights,
                                           Set<org.apache.flink.core.memory.ManagedMemoryUseCase> slotScopeUseCases)

      • setOneInputStateKey

        public void setOneInputStateKey​(Integer vertexID,
                                org.apache.flink.api.java.functions.KeySelector<?,​?> keySelector,
                                org.apache.flink.api.common.typeutils.TypeSerializer<?> keySerializer)

      • setTwoInputStateKey

        public void setTwoInputStateKey​(Integer vertexID,
                                org.apache.flink.api.java.functions.KeySelector<?,​?> keySelector1,
                                org.apache.flink.api.java.functions.KeySelector<?,​?> keySelector2,
                                org.apache.flink.api.common.typeutils.TypeSerializer<?> keySerializer)

      • setMultipleInputStateKey

        public void setMultipleInputStateKey​(Integer vertexID,
                                     List<org.apache.flink.api.java.functions.KeySelector<?,​?>> keySelectors,
                                     org.apache.flink.api.common.typeutils.TypeSerializer<?> keySerializer)

      • setBufferTimeout

        public void setBufferTimeout​(Integer vertexID,
                             long bufferTimeout)

      • setSerializers

        public void setSerializers​(Integer vertexID,
                           org.apache.flink.api.common.typeutils.TypeSerializer<?> in1,
                           org.apache.flink.api.common.typeutils.TypeSerializer<?> in2,
                           org.apache.flink.api.common.typeutils.TypeSerializer<?> out)

      • setInputFormat

        public void setInputFormat​(Integer vertexID,
                           org.apache.flink.api.common.io.InputFormat<?,​?> inputFormat)

      • setOutputFormat

        public void setOutputFormat​(Integer vertexID,
                            org.apache.flink.api.common.io.OutputFormat<?> outputFormat)

      • setTransformationUID

        public void setTransformationUID​(Integer nodeId,
                                 String transformationId)

      • getStreamEdges

        @VisibleForTesting
public List<StreamEdge> getStreamEdges​(int sourceId)

      • getStreamEdges

        @VisibleForTesting
public List<StreamEdge> getStreamEdges​(int sourceId,
                                       int targetId)

      • getStreamEdgesOrThrow

        @VisibleForTesting
@Deprecated
public List<StreamEdge> getStreamEdgesOrThrow​(int sourceId,
                                              int targetId)
        Deprecated.

      • getLoopTimeout

        public long getLoopTimeout​(Integer vertexID)

      • createIterationSourceAndSink

        public org.apache.flink.api.java.tuple.Tuple2<StreamNode,​StreamNode> createIterationSourceAndSink​(int loopId,
                                                                                                        int sourceId,
                                                                                                        int sinkId,
                                                                                                        long timeout,
                                                                                                        int parallelism,
                                                                                                        int maxParallelism,
                                                                                                        org.apache.flink.api.common.operators.ResourceSpec minResources,
                                                                                                        org.apache.flink.api.common.operators.ResourceSpec preferredResources)

      • getIterationSourceSinkPairs

        public Set<org.apache.flink.api.java.tuple.Tuple2<StreamNode,​StreamNode>> getIterationSourceSinkPairs()

      • getJobGraph

        @VisibleForTesting
public JobGraph getJobGraph()
        Gets the assembled JobGraph with a random JobID.

      • getJobGraph

        public JobGraph getJobGraph​(ClassLoader userClassLoader,
                            @Nullable
                            org.apache.flink.api.common.JobID jobID)
        Gets the assembled JobGraph with a specified JobID.

      • getStreamingPlanAsJSON

        public String getStreamingPlanAsJSON()

      • setJobType

        public void setJobType​(JobType jobType)

      • getJobType

        public JobType getJobType()

      • isAutoParallelismEnabled

        public boolean isAutoParallelismEnabled()

      • setAutoParallelismEnabled

        public void setAutoParallelismEnabled​(boolean autoParallelismEnabled)

      • getVertexDescriptionMode

        public org.apache.flink.configuration.PipelineOptions.VertexDescriptionMode getVertexDescriptionMode()

      • setVertexDescriptionMode

        public void setVertexDescriptionMode​(org.apache.flink.configuration.PipelineOptions.VertexDescriptionMode mode)

      • setVertexNameIncludeIndexPrefix

        public void setVertexNameIncludeIndexPrefix​(boolean includePrefix)

      • isVertexNameIncludeIndexPrefix

        public boolean isVertexNameIncludeIndexPrefix()

      • registerJobStatusHook

        public void registerJobStatusHook​(org.apache.flink.core.execution.JobStatusHook hook)
        Registers the JobStatusHook.

      • getJobStatusHooks

        public List<org.apache.flink.core.execution.JobStatusHook> getJobStatusHooks()

      • setSupportsConcurrentExecutionAttempts

        public void setSupportsConcurrentExecutionAttempts​(Integer vertexId,
                                                   boolean supportsConcurrentExecutionAttempts)

      • setAttribute

        public void setAttribute​(Integer vertexId,
                         org.apache.flink.api.common.attribute.Attribute attribute)