Package org.apache.camel.component.file

Class GenericFileConsumer<T>

java.lang.Object
org.apache.camel.support.service.BaseService
org.apache.camel.support.service.ServiceSupport
org.apache.camel.support.DefaultConsumer
org.apache.camel.support.ScheduledPollConsumer
org.apache.camel.support.ScheduledBatchPollingConsumer
org.apache.camel.component.file.GenericFileConsumer<T>
All Implemented Interfaces:
AutoCloseable, Runnable, org.apache.camel.BatchConsumer, org.apache.camel.Consumer, org.apache.camel.EndpointAware, org.apache.camel.health.HealthCheckAware, org.apache.camel.PollingConsumerPollingStrategy, org.apache.camel.RouteAware, org.apache.camel.Service, org.apache.camel.ShutdownableService, org.apache.camel.spi.HostedService, org.apache.camel.spi.RouteIdAware, org.apache.camel.spi.ShutdownAware, org.apache.camel.spi.ShutdownPrepared, org.apache.camel.StatefulService, org.apache.camel.Suspendable, org.apache.camel.SuspendableService
Direct Known Subclasses:
FileConsumer
public abstract class GenericFileConsumer<T> extends org.apache.camel.support.ScheduledBatchPollingConsumer
Base class for file consumers.

  • Field Summary

    Fields
    Modifier and Type
    Field
    Description
    protected org.apache.camel.Processor
    customProcessor
     
    protected boolean
    eagerLimitMaxMessagesPerPoll
     
    protected GenericFileEndpoint<T>
    endpoint
     
    protected GenericFileOperations<T>
    operations
     
    protected int
    pendingExchanges
     
    protected boolean
    prepareOnStartup
     
    protected GenericFileProcessStrategy<T>
    processStrategy
     
    protected org.apache.camel.ShutdownRunningTask
    shutdownRunningTask
     

    Fields inherited from class org.apache.camel.support.ScheduledBatchPollingConsumer

    maxMessagesPerPoll

    Fields inherited from class org.apache.camel.support.service.BaseService

    BUILT, FAILED, INITIALIZED, INITIALIZING, lock, NEW, SHUTDOWN, SHUTTING_DOWN, STARTED, STARTING, status, STOPPED, STOPPING, SUSPENDED, SUSPENDING

  • Constructor Summary

    Constructors
    Modifier
    Constructor
    Description
    protected
    GenericFileConsumer(GenericFileEndpoint<T> endpoint, org.apache.camel.Processor processor, GenericFileOperations<T> operations, GenericFileProcessStrategy<T> processStrategy)
     

  • Method Summary

    Modifier and Type
    Method
    Description
    void
    afterPoll()
     
    long
    beforePoll(long timeout)
     
    boolean
    canPollMoreFiles(List<?> fileList)
    Whether or not we can continue polling for more files
    protected abstract org.apache.camel.Exchange
    createExchange(GenericFile<T> file)
    Creates the exchange from the polled file
    protected boolean
    customProcessExchange(org.apache.camel.Exchange exchange, org.apache.camel.Processor processor)
    Processes the exchange using a custom processor.
    protected void
    doInit()
     
    protected void
    doStart()
     
    protected void
    doStop()
     
    protected String
    evaluateFileExpression(org.apache.camel.Exchange exchange)
     
    org.apache.camel.Processor
    getCustomProcessor()
     
    protected abstract Supplier<String>
    getRelativeFilePath(String endpointPath, String path, String absolutePath, T file)
    Geta the relative path from the given file, calculated from the starting path, current path, and current absolute path
    protected boolean
    hasExtExlusions(String fname)
     
    protected boolean
    hasExtInclusions(String fname)
     
    protected boolean
    ignoreCannotRetrieveFile(String name, org.apache.camel.Exchange exchange, Exception cause)
    Whether to ignore if the file cannot be retrieved.
    boolean
    isEagerLimitMaxMessagesPerPoll()
     
    protected abstract boolean
    isMatched(Supplier<GenericFile<T>> file, String doneFileName, T[] files)
    Strategy to perform file matching based on endpoint configuration in terms of done file name.
    protected boolean
    isMatched(org.apache.camel.Exchange dynamic, Supplier<GenericFile<T>> file, String name, String absoluteFilePath, Supplier<String> relativeFilePath, boolean isDirectory, T[] files)
    Strategy to perform file matching based on endpoint configuration.
    protected boolean
    isMatchedHiddenFile(Supplier<GenericFile<T>> file, String name, boolean isDirectory)
    Strategy to perform hidden file matching based on endpoint configuration.
    boolean
    isRetrieveFile()
    Whether the consumer should retrieve/download files.
    protected boolean
    isValidFile(org.apache.camel.Exchange dynamic, Supplier<GenericFile<T>> file, String name, String absoluteFilePath, Supplier<String> relativeFilePath, boolean isDirectory, T[] files)
    Strategy for validating if the given remote file should be included or not
    void
    onInit()
     
    protected int
    poll()
     
    protected int
    poll(org.apache.camel.Exchange dynamic)
    Poll for files
    protected abstract boolean
    pollDirectory(org.apache.camel.Exchange dynamic, String fileName, List<GenericFile<T>> fileList, int depth)
    Polls the given directory for files to process
    protected void
    postPollCheck(int polledMessages)
    Override if required.
    protected boolean
    prePollCheck()
    Override if required.
    int
    processBatch(Queue<Object> exchanges)
     
    protected boolean
    processExchange(org.apache.camel.Exchange exchange)
    Processes the exchange
    protected void
    removeExcessiveInProgressFiles(Deque<org.apache.camel.Exchange> exchanges, int limit)
    Drain any in progress files as we are done with this batch
    protected void
    removeExcessiveInProgressFiles(List<GenericFile<T>> files)
    Drain any in progress files as we are done with the files
    void
    setCustomProcessor(org.apache.camel.Processor processor)
    Use a custom processor to process the exchange.
    void
    setEagerLimitMaxMessagesPerPoll(boolean eagerLimitMaxMessagesPerPoll)
     
    void
    setOperations(GenericFileOperations<T> operations)
    Sets the operations to be used.
    void
    setRetrieveFile(boolean retrieveFile)
    Whether the consumer should retrieve/download files.
    protected abstract void
    updateFileHeaders(GenericFile<T> file, org.apache.camel.Message message)
    Updates the information on Message after we have acquired read-lock and can begin process the file.

    Methods inherited from class org.apache.camel.support.ScheduledBatchPollingConsumer

    deferShutdown, getMaxMessagesPerPoll, getPendingExchangesSize, isBatchAllowed, prepareShutdown, processEmptyMessage, setMaxMessagesPerPoll

    Methods inherited from class org.apache.camel.support.ScheduledPollConsumer

    addLastErrorDetail, afterConfigureScheduler, doBuild, doShutdown, doSuspend, forceConsumerAsReady, getBackoffCounter, getBackoffErrorThreshold, getBackoffIdleThreshold, getBackoffMultiplier, getCounter, getDelay, getErrorCounter, getInitialDelay, getLastError, getLastErrorDetails, getPollStrategy, getRepeatCount, getRunLoggingLevel, getScheduledExecutorService, getScheduler, getSchedulerProperties, getSuccessCounter, getTimeUnit, initialHealthCheckState, isConsumerReady, isFirstPollDone, isGreedy, isPollAllowed, isPolling, isSchedulerStarted, isSendEmptyMessageWhenIdle, isStartScheduler, isUseFixedDelay, run, setBackoffErrorThreshold, setBackoffIdleThreshold, setBackoffMultiplier, setDelay, setGreedy, setInitialDelay, setPollStrategy, setRepeatCount, setRunLoggingLevel, setScheduledExecutorService, setScheduler, setSchedulerProperties, setSendEmptyMessageWhenIdle, setStartScheduler, setTimeUnit, setUseFixedDelay, startScheduler

    Methods inherited from class org.apache.camel.support.DefaultConsumer

    createExchange, createUoW, defaultConsumerCallback, doneUoW, getAsyncProcessor, getEndpoint, getExceptionHandler, getHealthCheck, getProcessor, getRoute, getRouteId, handleException, handleException, handleException, isHostedService, releaseExchange, setExceptionHandler, setHealthCheck, setRoute, setRouteId, toString

    Methods inherited from class org.apache.camel.support.service.BaseService

    build, doFail, doLifecycleChange, doResume, fail, getInternalLock, getStatus, init, isBuild, isInit, isNew, isRunAllowed, isShutdown, isStarted, isStarting, isStartingOrStarted, isStopped, isStopping, isStoppingOrStopped, isSuspended, isSuspending, isSuspendingOrSuspended, resume, shutdown, start, stop, suspend

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

    Methods inherited from interface org.apache.camel.Consumer

    createExchange, defaultConsumerCallback, getProcessor, releaseExchange

    Methods inherited from interface org.apache.camel.EndpointAware

    getEndpoint

    Methods inherited from interface org.apache.camel.health.HealthCheckAware

    getHealthCheck, setHealthCheck

    Methods inherited from interface org.apache.camel.Service

    build, close, init, start, stop

    Methods inherited from interface org.apache.camel.ShutdownableService

    shutdown

    Methods inherited from interface org.apache.camel.StatefulService

    getStatus, isRunAllowed, isStarted, isStarting, isStopped, isStopping, isSuspending

    Methods inherited from interface org.apache.camel.SuspendableService

    isSuspended, resume, suspend

  • Field Details

    • endpoint

      protected GenericFileEndpoint<T> endpoint

    • operations

      protected GenericFileOperations<T> operations

    • processStrategy

      protected GenericFileProcessStrategy<T> processStrategy

    • shutdownRunningTask

      protected volatile org.apache.camel.ShutdownRunningTask shutdownRunningTask

    • pendingExchanges

      protected volatile int pendingExchanges

    • customProcessor

      protected org.apache.camel.Processor customProcessor

    • eagerLimitMaxMessagesPerPoll

      protected boolean eagerLimitMaxMessagesPerPoll

    • prepareOnStartup

      protected volatile boolean prepareOnStartup

  • Constructor Details

  • Method Details

    • getCustomProcessor

      public org.apache.camel.Processor getCustomProcessor()

    • setCustomProcessor

      public void setCustomProcessor(org.apache.camel.Processor processor)
      Use a custom processor to process the exchange.

      Only set this if you need to do custom processing, instead of the regular processing.

      This is for example used to browse file endpoints by leveraging the file consumer to poll the directory to gather the list of exchanges. But to avoid processing the files regularly we can use a custom processor.

      Parameters:
      processor - a custom processor

    • isEagerLimitMaxMessagesPerPoll

      public boolean isEagerLimitMaxMessagesPerPoll()

    • setEagerLimitMaxMessagesPerPoll

      public void setEagerLimitMaxMessagesPerPoll(boolean eagerLimitMaxMessagesPerPoll)

    • createExchange

      protected abstract org.apache.camel.Exchange createExchange(GenericFile<T> file)
      Creates the exchange from the polled file

    • poll

      protected int poll() throws Exception
      Specified by:
      poll in class org.apache.camel.support.ScheduledPollConsumer
      Throws:
      Exception

    • poll

      protected int poll(org.apache.camel.Exchange dynamic) throws Exception
      Poll for files
      Overrides:
      poll in class org.apache.camel.support.ScheduledPollConsumer
      Throws:
      Exception

    • processBatch

      public int processBatch(Queue<Object> exchanges)

    • removeExcessiveInProgressFiles

      protected void removeExcessiveInProgressFiles(Deque<org.apache.camel.Exchange> exchanges, int limit)
      Drain any in progress files as we are done with this batch
      Parameters:
      exchanges - the exchanges
      limit - the limit

    • removeExcessiveInProgressFiles

      protected void removeExcessiveInProgressFiles(List<GenericFile<T>> files)
      Drain any in progress files as we are done with the files
      Parameters:
      files - the files

    • canPollMoreFiles

      public boolean canPollMoreFiles(List<?> fileList)
      Whether or not we can continue polling for more files
      Parameters:
      fileList - the current list of gathered files
      Returns:
      true to continue, false to stop due hitting maxMessagesPerPoll limit

    • prePollCheck

      protected boolean prePollCheck() throws Exception
      Override if required. Perform some checks (and perhaps actions) before we poll.
      Returns:
      true to poll, false to skip this poll.
      Throws:
      Exception

    • postPollCheck

      protected void postPollCheck(int polledMessages)
      Override if required. Perform some checks (and perhaps actions) after we have polled.
      Parameters:
      polledMessages - number of polled messages

    • pollDirectory

      protected abstract boolean pollDirectory(org.apache.camel.Exchange dynamic, String fileName, List<GenericFile<T>> fileList, int depth)
      Polls the given directory for files to process
      Parameters:
      fileName - current directory or file
      fileList - current list of files gathered
      depth - the current depth of the directory (will start from 0)
      Returns:
      whether or not to continue polling, false means the maxMessagesPerPoll limit has been hit

    • setOperations

      public void setOperations(GenericFileOperations<T> operations)
      Sets the operations to be used.

      Can be used to set a fresh operations in case of recovery attempts

      Parameters:
      operations - the operations

    • ignoreCannotRetrieveFile

      protected boolean ignoreCannotRetrieveFile(String name, org.apache.camel.Exchange exchange, Exception cause)
      Whether to ignore if the file cannot be retrieved.

      By default an GenericFileOperationFailedException is thrown if the file cannot be retrieved.

      This method allows to suppress this and just ignore that.

      Parameters:
      name - the file name
      exchange - the exchange
      cause - optional exception occurred during retrieving file
      Returns:
      true to ignore, false is the default.

    • processExchange

      protected boolean processExchange(org.apache.camel.Exchange exchange)
      Processes the exchange
      Parameters:
      exchange - the exchange
      Returns:
      true if the file was started to be processed, false if the file was not started to be processed, for some reason (not found, or aborted etc)

    • updateFileHeaders

      protected abstract void updateFileHeaders(GenericFile<T> file, org.apache.camel.Message message)
      Updates the information on Message after we have acquired read-lock and can begin process the file.
      Parameters:
      file - the file
      message - the Camel message to update its headers

    • isRetrieveFile

      public boolean isRetrieveFile()
      Whether the consumer should retrieve/download files. If false then the message body is null as no file is retrieved.

    • setRetrieveFile

      public void setRetrieveFile(boolean retrieveFile)
      Whether the consumer should retrieve/download files. If false then the message body is null as no file is retrieved.

    • customProcessExchange

      protected boolean customProcessExchange(org.apache.camel.Exchange exchange, org.apache.camel.Processor processor)
      Processes the exchange using a custom processor.
      Parameters:
      exchange - the exchange
      processor - the custom processor

    • isValidFile

      protected boolean isValidFile(org.apache.camel.Exchange dynamic, Supplier<GenericFile<T>> file, String name, String absoluteFilePath, Supplier<String> relativeFilePath, boolean isDirectory, T[] files)
      Strategy for validating if the given remote file should be included or not
      Parameters:
      file - the file
      name - the file name
      isDirectory - whether the file is a directory or a file
      files - files in the directory
      Returns:
      true to include the file, false to skip it

    • isMatchedHiddenFile

      protected boolean isMatchedHiddenFile(Supplier<GenericFile<T>> file, String name, boolean isDirectory)
      Strategy to perform hidden file matching based on endpoint configuration.

      Will always return false for certain files/folders:

      • Starting with a dot (hidden)

    • getRelativeFilePath

      protected abstract Supplier<String> getRelativeFilePath(String endpointPath, String path, String absolutePath, T file)
      Geta the relative path from the given file, calculated from the starting path, current path, and current absolute path

    • isMatched

      protected boolean isMatched(org.apache.camel.Exchange dynamic, Supplier<GenericFile<T>> file, String name, String absoluteFilePath, Supplier<String> relativeFilePath, boolean isDirectory, T[] files)
      Strategy to perform file matching based on endpoint configuration.

      Will always return false for certain files/folders:

      • Starting with a dot
      • lock files
      And then true for directories.
      Parameters:
      file - the file
      name - the file name
      absoluteFilePath - the absolute file name
      relativeFilePath - the relative file name
      isDirectory - whether the file is a directory or a file
      files - files in the directory
      Returns:
      true if the file is matched, false if not

    • hasExtInclusions

      protected boolean hasExtInclusions(String fname)

    • hasExtExlusions

      protected boolean hasExtExlusions(String fname)

    • isMatched

      protected abstract boolean isMatched(Supplier<GenericFile<T>> file, String doneFileName, T[] files)
      Strategy to perform file matching based on endpoint configuration in terms of done file name.
      Parameters:
      file - the file
      doneFileName - the done file name (without any paths)
      files - files in the directory
      Returns:
      true if the file is matched, false if not

    • evaluateFileExpression

      protected String evaluateFileExpression(org.apache.camel.Exchange exchange)

    • doInit

      protected void doInit() throws Exception
      Overrides:
      doInit in class org.apache.camel.support.ScheduledPollConsumer
      Throws:
      Exception

    • doStart

      protected void doStart() throws Exception
      Overrides:
      doStart in class org.apache.camel.support.ScheduledPollConsumer
      Throws:
      Exception

    • doStop

      protected void doStop() throws Exception
      Overrides:
      doStop in class org.apache.camel.support.ScheduledPollConsumer
      Throws:
      Exception

    • onInit

      public void onInit() throws Exception
      Specified by:
      onInit in interface org.apache.camel.PollingConsumerPollingStrategy
      Overrides:
      onInit in class org.apache.camel.support.ScheduledPollConsumer
      Throws:
      Exception

    • beforePoll

      public long beforePoll(long timeout) throws Exception
      Specified by:
      beforePoll in interface org.apache.camel.PollingConsumerPollingStrategy
      Overrides:
      beforePoll in class org.apache.camel.support.ScheduledPollConsumer
      Throws:
      Exception

    • afterPoll

      public void afterPoll() throws Exception
      Specified by:
      afterPoll in interface org.apache.camel.PollingConsumerPollingStrategy
      Overrides:
      afterPoll in class org.apache.camel.support.ScheduledPollConsumer
      Throws:
      Exception