org.apache.nifi.processors.standard

Class TailFile

java.lang.Object

org.apache.nifi.components.AbstractConfigurableComponent

org.apache.nifi.processor.AbstractSessionFactoryProcessor

org.apache.nifi.processor.AbstractProcessor

org.apache.nifi.processors.standard.TailFile

All Implemented Interfaces:

ConfigurableComponent, Processor

@TriggerSerially
 @InputRequirement(value=INPUT_FORBIDDEN)
 @Tags(value={"tail","file","log","text","source"})
 @CapabilityDescription(value="\"Tails\" a file, or a list of files, ingesting data from the file as it is written to the file. The file is expected to be textual. Data is ingested only when a new line is encountered (carriage return or new-line character or combination). If the file to tail is periodically \"rolled over\", as is generally the case with log files, an optional Rolling Filename Pattern can be used to retrieve data from files that have rolled over, even if the rollover occurred while NiFi was not running (provided that the data still exists upon restart of NiFi). It is generally advisable to set the Run Schedule to a few seconds, rather than running with the default value of 0 secs, as this Processor will consume a lot of resources if scheduled very aggressively. At this time, this Processor does not support ingesting files that have been compressed when \'rolled over\'.")
 @Stateful(scopes={LOCAL,CLUSTER},
          description="Stores state about where in the Tailed File it left off so that on restart it does not have to duplicate data. State is stored either local or clustered depend on the <File Location> property.")
 @WritesAttribute(attribute="tailfile.original.path",
                 description="Path of the original file the flow file comes from.")
 @Restricted(restrictions=@Restriction(requiredPermission=READ_FILESYSTEM,explanation="Provides operator the ability to read from any file that NiFi has access to."))
public class TailFile
extends AbstractProcessor

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
(package private) static class	TailFile.NulCharacterEncounteredException
(package private) static class	TailFile.TailFileObject
(package private) static class	TailFile.TailFileState A simple Java class to hold information about our state so that we can maintain this state across multiple invocations of the Processor

Field Summary

Fields

Modifier and Type	Field and Description
(package private) static PropertyDescriptor	BASE_DIRECTORY
(package private) static PropertyDescriptor	FILENAME
private AtomicBoolean	isMultiChanging
private AtomicLong	lastLookup
(package private) static PropertyDescriptor	LINE_START_PATTERN
private ByteArrayOutputStream	linesBuffer
private Pattern	lineStartPattern
(package private) static AllowableValue	LOCATION_LOCAL
(package private) static AllowableValue	LOCATION_REMOTE
(package private) static PropertyDescriptor	LOOKUP_FREQUENCY
(package private) static String	MAP_PREFIX
(package private) static PropertyDescriptor	MAX_BUFFER_LENGTH
private long	maxBufferBytes
(package private) static PropertyDescriptor	MAXIMUM_AGE
(package private) static PropertyDescriptor	MODE
(package private) static AllowableValue	MODE_MULTIFILE
(package private) static AllowableValue	MODE_SINGLEFILE
private static byte[]	NEW_LINE_BYTES
(package private) static PropertyDescriptor	POST_ROLLOVER_TAIL_PERIOD
(package private) static PropertyDescriptor	RECURSIVE
(package private) static Relationship	REL_SUCCESS
private boolean	requireStateLookup
(package private) static PropertyDescriptor	REREAD_ON_NUL
(package private) static PropertyDescriptor	ROLLING_FILENAME_PATTERN
(package private) static AllowableValue	START_BEGINNING_OF_TIME
(package private) static AllowableValue	START_CURRENT_FILE
(package private) static AllowableValue	START_CURRENT_TIME
(package private) static PropertyDescriptor	START_POSITION
(package private) static PropertyDescriptor	STATE_LOCATION
private Map<String,TailFile.TailFileObject>	states

Constructor Summary

Constructors

Constructor and Description
TailFile()

Method Summary

Modifier and Type	Method and Description
private void	cleanReader(TailFile.TailFileObject tfo)
void	cleanup(ProcessContext context)
void	compileRegex(ProcessContext context)
private TailFile.TailFileState	consumeFileFully(File file, ProcessContext context, ProcessSession session, TailFile.TailFileObject tfo) Creates a new FlowFile that contains the entire contents of the given file and transfers that FlowFile to success.
private FileChannel	createReader(File file, long position)
protected Collection<ValidationResult>	customValidate(ValidationContext context)
private void	flushByteArrayOutputStream(ByteArrayOutputStream baos, OutputStream out, Checksum checksum, boolean ignoreRegex)
private void	flushLinesBuffer(OutputStream out, Checksum checksum)
long	getCurrentTimeMs()
private List<String>	getFilesToTail(String baseDir, String fileRegex, boolean isRecursive, long maxAge) Method to list the files to tail according to the given base directory and using the user-provided regular expression
Set<Relationship>	getRelationships()
private List<File>	getRolledOffFiles(ProcessContext context, long minTimestamp, String tailFilePath) Returns a list of all Files that match the following criteria: Filename matches the Rolling Filename Pattern Filename does not match the actual file being tailed The Last Modified Time on the file is equal to or later than the given minimum timestamp
(package private) Map<String,TailFile.TailFileObject>	getState()
private Scope	getStateScope(ProcessContext context)
protected List<PropertyDescriptor>	getSupportedPropertyDescriptors()
private void	initStates(List<String> filesToTail, Map<String,String> statesMap, boolean isCleared, String startPosition)
private List<String>	lookup(ProcessContext context)
void	onPrimaryNodeChange()
void	onPropertyModified(PropertyDescriptor descriptor, String oldValue, String newValue)
void	onTrigger(ProcessContext context, ProcessSession session)
private void	persistState(Map<String,String> state, ProcessSession session, ProcessContext context)
private void	persistState(TailFile.TailFileObject tfo, ProcessSession session, ProcessContext context)
private void	processTailFile(ProcessContext context, ProcessSession session, String tailFile)
private long	readLines(FileChannel reader, ByteBuffer buffer, OutputStream out, Checksum checksum, Boolean reReadOnNul)
private long	readLines(FileChannel reader, ByteBuffer buffer, OutputStream out, Checksum checksum, Boolean reReadOnNul, boolean readFully) Read new lines from the given FileChannel, copying it to the given Output Stream.
private boolean	recoverRolledFiles(ProcessContext context, ProcessSession session, String tailFile, List<File> rolledOffFiles, Long expectedChecksum, long position) Finds any files that have rolled over and have not yet been ingested by this Processor.
private boolean	recoverRolledFiles(ProcessContext context, ProcessSession session, String tailFile, Long expectedChecksum, long timestamp, long position) Finds any files that have rolled over and have not yet been ingested by this Processor.
void	recoverState(ProcessContext context)
private void	recoverState(ProcessContext context, List<String> filesToTail, Map<String,String> map)
private void	recoverState(ProcessContext context, Map<String,String> stateValues, String filePath) Updates member variables to reflect the "expected recovery checksum" and seek to the appropriate location in the tailed file, updating our checksum, so that we are ready to proceed with the onTrigger(ProcessContext, ProcessSession) call.
private void	resetState(String filePath)
private boolean	tailRolledFile(ProcessContext context, ProcessSession session, String tailFile, Long expectedChecksum, long position, TailFile.TailFileObject tfo, File fileToTail, boolean readFully, boolean tailingPostRollover)

Methods inherited from class org.apache.nifi.processor.AbstractProcessor

onTrigger

Methods inherited from class org.apache.nifi.processor.AbstractSessionFactoryProcessor

getControllerServiceLookup, getIdentifier, getLogger, getNodeTypeProvider, init, initialize, isConfigurationRestored, isScheduled, toString, updateConfiguredRestoredTrue, updateScheduledFalse, updateScheduledTrue

Methods inherited from class org.apache.nifi.components.AbstractConfigurableComponent

equals, getPropertyDescriptor, getPropertyDescriptors, getSupportedDynamicPropertyDescriptor, hashCode, validate

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.apache.nifi.components.ConfigurableComponent

getPropertyDescriptor, getPropertyDescriptors, validate

Field Detail

MAP_PREFIX

static final String MAP_PREFIX

See Also:

Constant Field Values

NEW_LINE_BYTES

private static final byte[] NEW_LINE_BYTES

LOCATION_LOCAL

static final AllowableValue LOCATION_LOCAL

LOCATION_REMOTE

static final AllowableValue LOCATION_REMOTE

MODE_SINGLEFILE

static final AllowableValue MODE_SINGLEFILE

MODE_MULTIFILE

static final AllowableValue MODE_MULTIFILE

START_BEGINNING_OF_TIME

static final AllowableValue START_BEGINNING_OF_TIME

START_CURRENT_FILE

static final AllowableValue START_CURRENT_FILE

START_CURRENT_TIME

static final AllowableValue START_CURRENT_TIME

BASE_DIRECTORY

static final PropertyDescriptor BASE_DIRECTORY

MODE

static final PropertyDescriptor MODE

FILENAME

static final PropertyDescriptor FILENAME

ROLLING_FILENAME_PATTERN

static final PropertyDescriptor ROLLING_FILENAME_PATTERN

POST_ROLLOVER_TAIL_PERIOD

static final PropertyDescriptor POST_ROLLOVER_TAIL_PERIOD

STATE_LOCATION

static final PropertyDescriptor STATE_LOCATION

START_POSITION

static final PropertyDescriptor START_POSITION

RECURSIVE

static final PropertyDescriptor RECURSIVE

LOOKUP_FREQUENCY

static final PropertyDescriptor LOOKUP_FREQUENCY

MAXIMUM_AGE

static final PropertyDescriptor MAXIMUM_AGE

REREAD_ON_NUL

static final PropertyDescriptor REREAD_ON_NUL

LINE_START_PATTERN

static final PropertyDescriptor LINE_START_PATTERN

MAX_BUFFER_LENGTH

static final PropertyDescriptor MAX_BUFFER_LENGTH

REL_SUCCESS

static final Relationship REL_SUCCESS

states

private volatile Map<String,TailFile.TailFileObject> states

lastLookup

private volatile AtomicLong lastLookup

isMultiChanging

private volatile AtomicBoolean isMultiChanging

requireStateLookup

private volatile boolean requireStateLookup

linesBuffer

private volatile ByteArrayOutputStream linesBuffer

lineStartPattern

private volatile Pattern lineStartPattern

maxBufferBytes

private volatile long maxBufferBytes

Constructor Detail

TailFile

public TailFile()

Method Detail

getSupportedPropertyDescriptors

protected List<PropertyDescriptor> getSupportedPropertyDescriptors()

Overrides:

getSupportedPropertyDescriptors in class AbstractConfigurableComponent

getRelationships

public Set<Relationship> getRelationships()

Specified by:

getRelationships in interface Processor

Overrides:

getRelationships in class AbstractSessionFactoryProcessor

onPropertyModified

public void onPropertyModified(PropertyDescriptor descriptor,
                               String oldValue,
                               String newValue)

Specified by:

onPropertyModified in interface ConfigurableComponent

Overrides:

onPropertyModified in class AbstractConfigurableComponent

customValidate

protected Collection<ValidationResult> customValidate(ValidationContext context)

Overrides:

customValidate in class AbstractConfigurableComponent

onPrimaryNodeChange

@OnPrimaryNodeStateChange
public void onPrimaryNodeChange()

lookup

private List<String> lookup(ProcessContext context)

compileRegex

@OnScheduled
public void compileRegex(ProcessContext context)

recoverState

@OnScheduled
public void recoverState(ProcessContext context)
                               throws IOException

Throws:

IOException

initStates

private void initStates(List<String> filesToTail,
                        Map<String,String> statesMap,
                        boolean isCleared,
                        String startPosition)

recoverState

private void recoverState(ProcessContext context,
                          List<String> filesToTail,
                          Map<String,String> map)
                   throws IOException

Throws:

IOException

getFilesToTail

private List<String> getFilesToTail(String baseDir,
                                    String fileRegex,
                                    boolean isRecursive,
                                    long maxAge)

Method to list the files to tail according to the given base directory and using the user-provided regular expression

Parameters:

baseDir - base directory to recursively look into

fileRegex - expression regular used to match files to tail

isRecursive - true if looking for file recursively, false otherwise

Returns:

List of files to tail

recoverState

private void recoverState(ProcessContext context,
                          Map<String,String> stateValues,
                          String filePath)
                   throws IOException

Updates member variables to reflect the "expected recovery checksum" and seek to the appropriate location in the tailed file, updating our checksum, so that we are ready to proceed with the onTrigger(ProcessContext, ProcessSession) call.

Parameters:

context - the ProcessContext

stateValues - the values that were recovered from state that was previously stored. This Map should be populated with the keys defined in TailFile.TailFileState.StateKeys.

filePath - the file of the file for which state must be recovered

Throws:

IOException - if unable to seek to the appropriate location in the tailed file.

resetState

private void resetState(String filePath)

cleanup

@OnStopped
public void cleanup(ProcessContext context)

cleanReader

private void cleanReader(TailFile.TailFileObject tfo)

onTrigger

public void onTrigger(ProcessContext context,
                      ProcessSession session)
               throws ProcessException

Specified by:

onTrigger in class AbstractProcessor

Throws:

ProcessException

processTailFile

private void processTailFile(ProcessContext context,
                             ProcessSession session,
                             String tailFile)

readLines

private long readLines(FileChannel reader,
                       ByteBuffer buffer,
                       OutputStream out,
                       Checksum checksum,
                       Boolean reReadOnNul)
                throws IOException

Throws:

IOException

readLines

private long readLines(FileChannel reader,
                       ByteBuffer buffer,
                       OutputStream out,
                       Checksum checksum,
                       Boolean reReadOnNul,
                       boolean readFully)
                throws IOException

Read new lines from the given FileChannel, copying it to the given Output Stream. The Checksum is used in order to later determine whether or not data has been consumed.

Parameters:

reader - The FileChannel to read data from

buffer - the buffer to use for copying data

out - the OutputStream to copy the data to

checksum - the Checksum object to use in order to calculate checksum for recovery purposes

reReadOnNul - If set to 'true', ASCII NUL characters will be treated as temporary values and a NulCharacterEncounteredException is thrown. This allows the caller to re-attempt a read from the same position. If set to 'false' these characters will be treated as regular content.

readFully - If set to 'true' the last chunk of bytes after the last whole line will be also written to the OutputStream

Returns:

The new position after the lines have been read

Throws:

IOException - if an I/O error occurs.

flushByteArrayOutputStream

private void flushByteArrayOutputStream(ByteArrayOutputStream baos,
                                        OutputStream out,
                                        Checksum checksum,
                                        boolean ignoreRegex)
                                 throws IOException

Throws:

IOException

flushLinesBuffer

private void flushLinesBuffer(OutputStream out,
                              Checksum checksum)
                       throws IOException

Throws:

IOException

getRolledOffFiles

private List<File> getRolledOffFiles(ProcessContext context,
                                     long minTimestamp,
                                     String tailFilePath)
                              throws IOException

Returns a list of all Files that match the following criteria:

• Filename matches the Rolling Filename Pattern
• Filename does not match the actual file being tailed
• The Last Modified Time on the file is equal to or later than the given minimum timestamp

The List that is returned will be ordered by file timestamp, providing the oldest file first.

Parameters:

context - the ProcessContext to use in order to determine Processor configuration

minTimestamp - any file with a Last Modified Time before this timestamp will not be returned

Returns:

a list of all Files that have rolled over

Throws:

IOException - if unable to perform the listing of files

getStateScope

private Scope getStateScope(ProcessContext context)

persistState

private void persistState(TailFile.TailFileObject tfo,
                          ProcessSession session,
                          ProcessContext context)

persistState

private void persistState(Map<String,String> state,
                          ProcessSession session,
                          ProcessContext context)

createReader

private FileChannel createReader(File file,
                                 long position)

getState

Map<String,TailFile.TailFileObject> getState()

recoverRolledFiles

private boolean recoverRolledFiles(ProcessContext context,
                                   ProcessSession session,
                                   String tailFile,
                                   Long expectedChecksum,
                                   long timestamp,
                                   long position)

Finds any files that have rolled over and have not yet been ingested by this Processor. Each of these files that is found will be ingested as its own FlowFile. If a file is found that has been partially ingested, the rest of the file will be ingested as a single FlowFile but the data that already has been ingested will not be ingested again.

Parameters:

context - the ProcessContext to use in order to obtain Processor configuration.

session - the ProcessSession to use in order to interact with FlowFile creation and content.

expectedChecksum - the checksum value that is expected for the oldest file from offset 0 through <position>.

timestamp - the latest Last Modified Timestamp that has been consumed. Any data that was written before this data will not be ingested.

position - the byte offset in the file being tailed, where tailing last left off.

Returns:

true if the file being tailed has rolled over, false otherwise

recoverRolledFiles

private boolean recoverRolledFiles(ProcessContext context,
                                   ProcessSession session,
                                   String tailFile,
                                   List<File> rolledOffFiles,
                                   Long expectedChecksum,
                                   long position)

Finds any files that have rolled over and have not yet been ingested by this Processor. Each of these files that is found will be ingested as its own FlowFile. If a file is found that has been partially ingested, the rest of the file will be ingested as a single FlowFile but the data that already has been ingested will not be ingested again.

Parameters:

context - the ProcessContext to use in order to obtain Processor configuration.

session - the ProcessSession to use in order to interact with FlowFile creation and content.

expectedChecksum - the checksum value that is expected for the oldest file from offset 0 through <position>.

position - the byte offset in the file being tailed, where tailing last left off.

Returns:

true if the file being tailed has rolled over, false otherwise

tailRolledFile

private boolean tailRolledFile(ProcessContext context,
                               ProcessSession session,
                               String tailFile,
                               Long expectedChecksum,
                               long position,
                               TailFile.TailFileObject tfo,
                               File fileToTail,
                               boolean readFully,
                               boolean tailingPostRollover)
                        throws IOException

Throws:

IOException

consumeFileFully

private TailFile.TailFileState consumeFileFully(File file,
                                                ProcessContext context,
                                                ProcessSession session,
                                                TailFile.TailFileObject tfo)
                                         throws IOException

Creates a new FlowFile that contains the entire contents of the given file and transfers that FlowFile to success. This method will commit the given session and emit an appropriate Provenance Event.

Parameters:

file - the file to ingest

context - the ProcessContext

session - the ProcessSession

tfo - the current state

Returns:

the new, updated state that reflects that the given file has been ingested.

Throws:

IOException

getCurrentTimeMs

public long getCurrentTimeMs()