Class PutSQL
java.lang.Object
org.apache.nifi.components.AbstractConfigurableComponent
org.apache.nifi.processor.AbstractSessionFactoryProcessor
org.apache.nifi.processors.standard.PutSQL
- All Implemented Interfaces:
ConfigurableComponent
,Processor
@SupportsBatching
@SeeAlso(ConvertJSONToSQL.class)
@InputRequirement(INPUT_REQUIRED)
@Tags({"sql","put","rdbms","database","update","insert","relational"})
@CapabilityDescription("Executes a SQL UPDATE or INSERT command. The content of an incoming FlowFile is expected to be the SQL command to execute. The SQL command may use the ? to escape parameters. In this case, the parameters to use must exist as FlowFile attributes with the naming convention sql.args.N.type and sql.args.N.value, where N is a positive integer. The sql.args.N.type is expected to be a number indicating the JDBC Type. The content of the FlowFile is expected to be in UTF-8 format.")
@ReadsAttribute(attribute="fragment.identifier",description="If the <Support Fragment Transactions> property is true, this attribute is used to determine whether or not two FlowFiles belong to the same transaction.") @ReadsAttribute(attribute="fragment.count",description="If the <Support Fragment Transactions> property is true, this attribute is used to determine how many FlowFiles are needed to complete the transaction.") @ReadsAttribute(attribute="fragment.index",description="If the <Support Fragment Transactions> property is true, this attribute is used to determine the order that the FlowFiles in a transaction should be evaluated.") @ReadsAttribute(attribute="sql.args.N.type",description="Incoming FlowFiles are expected to be parametrized SQL statements. The type of each Parameter is specified as an integer that represents the JDBC Type of the parameter.") @ReadsAttribute(attribute="sql.args.N.value",description="Incoming FlowFiles are expected to be parametrized SQL statements. The value of the Parameters are specified as sql.args.1.value, sql.args.2.value, sql.args.3.value, and so on. The type of the sql.args.1.value Parameter is specified by the sql.args.1.type attribute.") @ReadsAttribute(attribute="sql.args.N.format",description="This attribute is always optional, but default options may not always work for your data. Incoming FlowFiles are expected to be parametrized SQL statements. In some cases a format option needs to be specified, currently this is only applicable for binary data types, dates, times and timestamps. Binary Data Types (defaults to \'ascii\') - ascii: each string character in your attribute value represents a single byte. This is the format provided by Avro Processors. base64: the string is a Base64 encoded string that can be decoded to bytes. hex: the string is hex encoded with all letters in upper case and no \'0x\' at the beginning. Dates/Times/Timestamps - Date, Time and Timestamp formats all support both custom formats or named format (\'yyyy-MM-dd\',\'ISO_OFFSET_DATE_TIME\') as specified according to java.time.format.DateTimeFormatter. If not specified, a long value input is expected to be an unix epoch (milli seconds from 1970/1/1), or a string value in \'yyyy-MM-dd\' format for Date, \'HH:mm:ss.SSS\' for Time (some database engines e.g. Derby or MySQL do not support milliseconds and will truncate milliseconds), \'yyyy-MM-dd HH:mm:ss.SSS\' for Timestamp is used.")
@WritesAttributes(@WritesAttribute(attribute="sql.generated.key",description="If the database generated a key for an INSERT statement and the Obtain Generated Keys property is set to true, this attribute will be added to indicate the generated key, if possible. This feature is not supported by all database vendors."))
public class PutSQL
extends AbstractSessionFactoryProcessor
-
Nested Class Summary
Nested ClassesModifier and TypeClassDescriptionprivate static class
A simple, immutable data structure to hold a List of FlowFiles and an indicator as to whether or not those FlowFiles represent a "fragmented transaction" - that is, a collection of FlowFiles that all must be executed as a single transaction (we refer to it as a fragment transaction because the information for that transaction, including SQL and the parameters, is fragmented across multiple FlowFiles).private static class
private static class
private static interface
private static class
A simple, immutable data structure to hold a Prepared Statement and a List of FlowFiles for which that statement should be evaluated.(package private) static class
A FlowFileFilter that is responsible for ensuring that the FlowFiles returned either belong to the same "fragmented transaction" (i.e., 1 transaction whose information is fragmented across multiple FlowFiles) or that none of the FlowFiles belongs to a fragmented transaction -
Field Summary
FieldsModifier and TypeFieldDescription(package private) static final PropertyDescriptor
(package private) static final PropertyDescriptor
(package private) static final PropertyDescriptor
private static final String
private static final String
private static final String
private ExceptionHandler
<PutSQL.FunctionContext> private final PartialFunctions.FetchFlowFiles
<PutSQL.FunctionContext> private static final String
private static final String
private static final String
(package private) final PutGroup.GroupFlowFiles
<PutSQL.FunctionContext, Connection, PutSQL.StatementFlowFileEnclosure> private final PutSQL.GroupingFunction
private final PutSQL.GroupingFunction
private final PutSQL.GroupingFunction
private final PartialFunctions.InitConnection
<PutSQL.FunctionContext, Connection> (package private) static final PropertyDescriptor
(package private) final PutGroup.PutFlowFiles
<PutSQL.FunctionContext, Connection, PutSQL.StatementFlowFileEnclosure> (package private) static final Relationship
(package private) static final Relationship
(package private) static final Relationship
(package private) static final PropertyDescriptor
(package private) static final PropertyDescriptor
(package private) static final PropertyDescriptor
-
Constructor Summary
Constructors -
Method Summary
Modifier and TypeMethodDescriptionprivate FlowFile
addErrorAttributesToFlowFile
(ProcessSession session, FlowFile flowFile, Exception exception) addErrorAttributesToFlowFilesInGroup
(ProcessSession session, List<FlowFile> flowFilesOnRelationship, List<FlowFile> flowFilesInGroup, Exception exception) void
protected final Collection
<ValidationResult> customValidate
(ValidationContext context) private String
Returns the key that was generated from the given statement, ornull
if no key was generated or it could not be determined.getFlowFilesOnRelationship
(RoutingResult result, Relationship relationship) private String
getSQL
(ProcessSession session, FlowFile flowFile) Determines the SQL statement that should be executed for the given FlowFileprotected List
<PropertyDescriptor> (package private) boolean
isFragmentedTransactionReady
(List<FlowFile> flowFiles, Long transactionTimeoutMillis) Determines which relationship the given FlowFiles should go to, based on a transaction timing out or transaction information not being present.onBatchUpdateError
(ProcessContext context, ProcessSession session, RoutingResult result) onFlowFileError
(ProcessContext context, ProcessSession session, RoutingResult result) onGroupError
(ProcessContext context, ProcessSession session, RoutingResult result) void
onTrigger
(ProcessContext context, ProcessSessionFactory sessionFactory) private PutSQL.FlowFilePoll
pollFlowFiles
(ProcessContext context, ProcessSession session, PutSQL.FunctionContext functionContext, RoutingResult result) Pulls a batch of FlowFiles from the incoming queues.Methods inherited from class org.apache.nifi.processor.AbstractSessionFactoryProcessor
getControllerServiceLookup, getIdentifier, getLogger, getNodeTypeProvider, init, initialize, isConfigurationRestored, isScheduled, toString, updateConfiguredRestoredTrue, updateScheduledFalse, updateScheduledTrue
Methods inherited from class org.apache.nifi.components.AbstractConfigurableComponent
equals, getPropertyDescriptor, getPropertyDescriptors, getSupportedDynamicPropertyDescriptor, hashCode, onPropertyModified, validate
Methods inherited from class java.lang.Object
clone, finalize, getClass, notify, notifyAll, wait, wait, wait
Methods inherited from interface org.apache.nifi.components.ConfigurableComponent
getPropertyDescriptor, getPropertyDescriptors, onPropertyModified, validate
Methods inherited from interface org.apache.nifi.processor.Processor
isStateful, migrateProperties, migrateRelationships
-
Field Details
-
CONNECTION_POOL
-
SQL_STATEMENT
-
AUTO_COMMIT
-
SUPPORT_TRANSACTIONS
-
TRANSACTION_TIMEOUT
-
BATCH_SIZE
-
OBTAIN_GENERATED_KEYS
-
REL_SUCCESS
-
REL_RETRY
-
REL_FAILURE
-
FRAGMENT_ID_ATTR
-
FRAGMENT_INDEX_ATTR
-
FRAGMENT_COUNT_ATTR
-
ERROR_MESSAGE_ATTR
- See Also:
-
ERROR_CODE_ATTR
- See Also:
-
ERROR_SQL_STATE_ATTR
- See Also:
-
process
-
adjustError
-
exceptionHandler
-
fetchFlowFiles
-
initConnection
-
groupFragmentedTransaction
-
groupFlowFilesBySQLBatch
-
groupFlowFilesBySQL
-
groupFlowFiles
final PutGroup.GroupFlowFiles<PutSQL.FunctionContext,Connection, groupFlowFilesPutSQL.StatementFlowFileEnclosure> -
putFlowFiles
final PutGroup.PutFlowFiles<PutSQL.FunctionContext,Connection, putFlowFilesPutSQL.StatementFlowFileEnclosure>
-
-
Constructor Details
-
PutSQL
public PutSQL()
-
-
Method Details
-
getSupportedPropertyDescriptors
- Overrides:
getSupportedPropertyDescriptors
in classAbstractConfigurableComponent
-
customValidate
- Overrides:
customValidate
in classAbstractConfigurableComponent
-
getRelationships
- Specified by:
getRelationships
in interfaceProcessor
- Overrides:
getRelationships
in classAbstractSessionFactoryProcessor
-
onFlowFileError
private ExceptionHandler.OnError<PutSQL.FunctionContext,FlowFile> onFlowFileError(ProcessContext context, ProcessSession session, RoutingResult result) -
onGroupError
private ExceptionHandler.OnError<RollbackOnFailure,PartialFunctions.FlowFileGroup> onGroupError(ProcessContext context, ProcessSession session, RoutingResult result) -
getFlowFilesOnRelationship
-
addErrorAttributesToFlowFilesInGroup
-
onBatchUpdateError
private ExceptionHandler.OnError<PutSQL.FunctionContext,PutSQL.StatementFlowFileEnclosure> onBatchUpdateError(ProcessContext context, ProcessSession session, RoutingResult result) -
constructProcess
-
onTrigger
public void onTrigger(ProcessContext context, ProcessSessionFactory sessionFactory) throws ProcessException - Throws:
ProcessException
-
pollFlowFiles
private PutSQL.FlowFilePoll pollFlowFiles(ProcessContext context, ProcessSession session, PutSQL.FunctionContext functionContext, RoutingResult result) Pulls a batch of FlowFiles from the incoming queues. If no FlowFiles are available, returnsnull
. Otherwise, a List of FlowFiles will be returned. If all FlowFiles pulled are not eligible to be processed, the FlowFiles will be penalized and transferred back to the input queue and an empty List will be returned. Otherwise, if the Support Fragmented Transactions property is true, all FlowFiles that belong to the same transaction will be sorted in the order that they should be evaluated.- Parameters:
context
- the process context for determining propertiessession
- the process session for pulling FlowFiles- Returns:
- a FlowFilePoll containing a List of FlowFiles to process, or
null
if there are no FlowFiles to process
-
determineGeneratedKey
Returns the key that was generated from the given statement, ornull
if no key was generated or it could not be determined.- Parameters:
stmt
- the statement that generated a key- Returns:
- the key that was generated from the given statement, or
null
if no key was generated, or it could not be determined.
-
getSQL
Determines the SQL statement that should be executed for the given FlowFile- Parameters:
session
- the session that can be used to access the given FlowFileflowFile
- the FlowFile whose SQL statement should be executed- Returns:
- the SQL that is associated with the given FlowFile
-
isFragmentedTransactionReady
boolean isFragmentedTransactionReady(List<FlowFile> flowFiles, Long transactionTimeoutMillis) throws IllegalArgumentException Determines which relationship the given FlowFiles should go to, based on a transaction timing out or transaction information not being present. If the FlowFiles should be processed and not transferred to any particular relationship yet, will returnnull
- Parameters:
flowFiles
- the FlowFiles whose relationship is to be determinedtransactionTimeoutMillis
- the maximum amount of time (in milliseconds) that we should wait for all FlowFiles in a transaction to be present before routing to failure- Returns:
- the appropriate relationship to route the FlowFiles to, or
null
if the FlowFiles should instead be processed - Throws:
IllegalArgumentException
-
addErrorAttributesToFlowFile
private FlowFile addErrorAttributesToFlowFile(ProcessSession session, FlowFile flowFile, Exception exception)
-