File2 has been edited by Claus Ibsen (May 23, 2009).

(View changes)

Content:

File Component

The File component provides access to file systems; allowing files to be processed by any other Camel Components or messages from other components can be saved to disk.

URI format

file:directoryName[?options]

or

file://directoryName[?options]

Where directoryName represents the underlying file directory.

Only directories

Camel 2.0 only support endpoints configured with a starting directory. So the directoryName must be a directory.
If you want to consume a single file only, you can use the fileName option to only select your filename, e.g. by just setting fileName=thefilename.

In Camel 1.x you could also configure a file and this caused more harm than good as it could lead to confusing situations.

URI Options

Common

Name Default Value Description
bufferSize 128kb Write buffer sized in bytes.
fileName null Use Expression such as File Language to dynamically set the filename. For consumers its used as a filename filter. For producers its used to evaluate the filename to write. If an expression is set it take precedents over the CamelFileName header. (Note: The header itself can also be an Expression). The expression options supports both String and Expression types. If the expression is a String type then its always evaluated using the File Language. If the expression is an Expression type then this type is of course used as it - this allows for instance also to use OGNL as expression. For the consumer, you can use it to filter filenames, so you can for instance consume todays file using the File Language syntax: mydata-${date:now:yyyyMMdd}.txt.
flatten false Flatten is used to flatten the file name path to strip any leading paths, so its just the file name. This allows you to consume recursive into sub directories but when you eg write the files to another directory they will be written in a single directory. Setting this to true on the producer enforces that any file name recived in CamelFileName header will be stripped for any leading paths.

Consumer only

Name Default Value Description
initialDelay 1000 milliseconds before polling the file/directory starts
delay 500 milliseconds before the next poll of the file/directory
useFixedDelay false true to use fixed delay between pools, otherwise fixed rate is used. See ScheduledExecutorService in JDK for details.
recursive false if a directory, will look for files in all the sub directories as well.
delete false If delete is true then the file will be deleted after it is processed
noop false If true then the file is not moved or deleted in any way. This option is good for read only data, or for ETL type requirements. If noop=true then Camel will set idempotent=true as well, avoiding consuming the same files over and over again.
preMove null Use Expression such as File Language to dynamically set the filename when moving it before processing. For example to move in progress file into the order directory set this value to order
move .camel Use Expression such as File Language to dynamically set the filename when moving it after processing. To move files into a .done subdirectory just enter .done.
include null Is used to include files if filename matches the regex pattern.
exclude null Is used to exclude files if filename matches the regex pattern.
idempotent false Option to use the Idempotent Consumer EIP pattern to let Camel skip already processed files. Will default use a memory based LRUCache that holds 1000 entries. If noop=true then idempotent will be enabled as well to avoid consuming the same files over and over again.
idempotentRepository null Pluggable repository as a org.apache.camel.processor.idempotent.MessageIdRepository class. Will default use MemoryMessageIdRepository if none is specified and idempotent is true.
filter null Pluggable filter as a org.apache.camel.component.file.GenericFileFilter class. Will skip files if filter returns false in its accept method. Camel also ships with an ANT path matcher filter in the camel-spring component. More details in section below.
sorter null Pluggable sorter as a java.util.Comparator<org.apache.camel.component.file.GenericFile> class.
sortBy null Build in sort by using the File Language. Supports nested sorts so you can have a sort by file name and as a 2nd group sort by modified date. See sorting section below for details.
readLock markerFile Used by consumer, to only poll the files if it has exclusive read lock to the file (= the file is not in progress of being written). Camel will wait until the file lock is granted. This option provides the build in strategies: fileLock, rename, markerFile and none. fileLock is for using java.nio.channels.FileLock. rename is for using a try to rename the file as a test if we can get exclusive read lock. markerFile is the behaviour from Camel 1.x, where Camel will create a marker file and hold lock on the marker file. none is for no read locks at all.
readLockTimeout 0 Optional timeout in millis for the read lock, if supported by the read lock. If the read lock could not be granted and the timeout triggered then Camel will skip the file. At next poll Camel will try the file again, and this time maybe the read lock could be granted.
exclusiveReadLockStrategy null Pluggable read lock as a org.apache.camel.component.file.GenericFileExclusiveReadLockStrategy implementation.

Producer only

Name Default Value Description
autoCreate true If set to true Camel will create the directory to the file if the file path does not exists.
append true When writing do we append to the end of the file, or replace it?
tempPrefix null This option is used to write the file using a temporary name, and then after the write is complete rename it to the real name. Can be used to identify files being written and also avoid consumers (not using exclusive read locks) reading in progress files. Is often used by FTP when uploading big files.

Default behavior for file consumer

  • By default the file is locked for the duration of the processing.
  • After the route has completed they are moved into the .camel subdirectory; so that they appear to be deleted.
  • The File Consumer will always skip any file which name starts with a dot, such as ".", ".camel", ".m2" or ".groovy".
  • Only files (not directories) is matched for valid filename if options such as: includeNamePrefix, includeNamePostfix, excludeNamePrefix, excludeNamePostfix, regexPattern is used.

Move and Delete operations

Any move or delete operations is executed after (post command) the routing has completed; so during processing of the Exchange the file is still located in the inbox folder.

Lets illustrate this with an example:

from("file://inobox?move=.done").to("bean:handleOrder");

When a file is dropped in the inbox folder the file consumer notices this and creates a new FileExchange that is routed to the handleOrder bean. The bean then processes the File. At this point in time the File is still located in the inbox folder. After the bean completes and thus the route is completed the file consumer will perform the move operation and move the file to the .done sub folder.

The move and preMove option should be a directory name. It can be either relative or absolute. If relative the directory is created as a sub folder from within the folder where the file was consumed.

By default Camel will move consumed files to the sub folder .camel relative where the file was consumed.

If you want to delete the file after processing, then the route should be:

from("file://inobox?delete=true").to("bean:handleOrder");

We have introduced a pre move operation to move files before they are processed. This allows you to mark which files has been scanned as they are moved to this sub folder before being processed.

from("file://inobox?preMove=inprogress").to("bean:handleOrder");

You can combine the pre move and the regular move:

from("file://inobox?preMove=inprogress&move=.done").to("bean:handleOrder");

So in this situation the file is in the inprogress folder when being processed, and after it's processed it's moved to the .done folder.

Fine grained control over Move and PreMove option

The move and preMove option is Expression based so we have the full power of the File Language to do advanced configuration of the directory and name pattern.
Camel will in fact internally convert the directory name you enter into a File Language expression. So when we enter move=.done Camel will convert this into: ${file:parent}/.done/${file:onlyname}. This is only done if Camel detects that you have not provided a ${ } in the option value yourself. So when you enter a ${ } Camel will not convert it and thus you have the full power.

So if we want to move the file into a backup folder with todays date as the pattern we can do:

move=backup/${date:now:yyyyMMdd}/${file:name}

See more examples at File Language

Message Headers

The following headers is supported by this component.

File producer only

Header Description
CamelFileName Specifies the name of the file to write (relative to the endpoint directory). The name can be a String, a String with a File Language or Simple expression. Or an Expression object. If its null then Camel will auto generate a filename based on the message unique id.

File consumer only

Header Description
CamelFileName Name of the consumed file as a relative file path with offset from the starting directory configured on the endpoint.
CamelFileNameOnly Only the file name, is just the name with no leading paths.
CamelFileNameProduced The actual absolute filepath (path + name) for the output file that was written. This header is set by Camel and its purpose is providing end-users the name of the file that was written.
CamelFileAbsolute A boolean whether the consumed file denotes a absolute path or not. Should normally be false for relative paths. Absolute path should normally not be used but we added to the move option to allow moving files to absolute paths. But can be used elsewhere as well.
CamelFileAbsolutePath The absolute path to the file. For relative files this path holds the relative path instead.
CamelFilePath The file path. For relative files this is the starting directory + the relative filename. For absolute files this is the absolute path.
CamelFileRelativePath The relative path.
CamelFileParent The parent path.
CamelFileLength A long containing the file size
CamelFileLastModified A Date containing the last modified timestamp of the file.
CamelFileBatchSize Total number of files being consumed in this batch.
CamelFileBatchIndex Current index out of total number of files being consumed in this batch.

Exchange Properties

As the file consumer is BatchConsumer it supports batching the files it polls. By batching it means that Camel will add some properties to the Exchange so you know the number of files polled the current index in that order.

Property Description
CamelBatchSize The total number of files that was polled in this batch.
CamelBatchIndex The current index of the batch. Starts from 0.
CamelBatchComplete A boolean indicating the last Exchange in the batch. Is only try for the last entry.

This allows you for instance to know how many files exists in this batch and for instance let the Aggregator aggregate this number of files.

Common gotchas with folder and filenames

When Camel is producing files (writing files) there are a few gotchas how to set a filename of your choice. By default Camel will use the message id as the filename, and since the message id is normally a unique generated id you will end up with filenames such as: ID-MACHINENAME-2443-1211718892437-1-0. If such a filename is not desired, then a filename must be provided in the message header "CamelFileName". The constant Exchange.FILE_NAME can also be used.

The sample code below produces files using the message id as the filename:

from("direct:report").to("file:target/reports");

To use report.txt as the filename you have to do:

from("direct:report").setHeader(Exchange.FILE_NAME, constant("report.txt")).to( "file:target/reports");

... the same as above, but with "CamelFileName":

from("direct:report").setHeader("CamelFileName", constant("report.txt")).to( "file:target/reports");

And a syntax where we set the filename on the endpoint with the fileName URI option.

from("direct:report").to("file:target/reports/?fileName=report.txt");

Filename Expression

Filename can be set either using the expression option or as a string based File Language expression in the CamelFileName header. See the File Language for syntax and samples.

Samples

Read from a directory and write to another directory

from("file://inputdir/?delete=true").to("file://outputdir")

Listen on a directory and create a message for each file dropped there. Copy the contents to the outputdir and delete the file in the inputdir.

Reading recursive from a directory and write the another

from("file://inputdir/?recursive=true&delete=true").to("file://outputdir")

Listen on a directory and create a message for each file dropped there. Copy the contents to the outputdir and delete the file in the inputdir. Will scan recursive into sub directories. Will layout the files in the same directory structure in the outputdir as the inputdir, incl. any sub directory.

inputdir/foo.txt
inputdir/sub/bar.txt

Will result to a layout as:

outputdir/foo.txt
outputdir/sub/bar.txt
Using flatten

If you want to store the files in the outputdir directory in the same directory, disregard the source directory layout (eg to flatten out the path), then you just add the flatten=true option on the file producer side:

from("file://inputdir/?recursive=true&delete=true").to("file://outputdir?flatten=true")

Will result to a layout as:

outputdir/foo.txt
outputdir/bar.txt

Reading from a directory and the default move operation

Camel will by default move any processed file into a .camel subdirectory in the dir the file was consumed from.

from("file://inputdir/?recursive=true&delete=true").to("file://outputdir")

Will result in a layout as:
before

inputdir/foo.txt
inputdir/sub/bar.txt

after

inputdir/.camel/foo.txt
inputdir/sub/.camel/bar.txt
outputdir/foo.txt
outputdir/sub/bar.txt

Read from a directory and process the message in java

from("file://inputdir/").process(new Processor() {
  public void process(Exchange exchange) throws Exception {
    Object body = exchange.getIn().getBody();
    // do some business logic with the input body
  }
});

Body will be File object pointing to the file that was just dropped to the inputdir directory.

Read files from a directory and send the content to a jms queue

from("file://inputdir/").convertBodyTo(String.class).to("jms:test.queue")

By default the file endpoint sends a FileMessage which contains a File as body. If you send this directly to the jms component the jms message will only contain the File object but not the content. By converting the File to a String the message will contain the file contents what is probably what you want to do.

The route above using Spring DSL:

<route>
      <from uri="file://inputdir/"/>
      <convertBodyTo type="java.lang.String"/>
      <to uri="jms:test.queue"/>
   </route>

Writing to files

Camel is of course also able to write files, eg. producing files. In the sample below we receive some reports on the SEDA queue that we processes before they are written to a directory.

public void testToFile() throws Exception {
    MockEndpoint mock = getMockEndpoint("mock:result");
    mock.expectedMessageCount(1);
    mock.expectedFileExists("target/test-reports/report.txt");

    template.sendBody("seda:reports", "This is a great report");

    assertMockEndpointsSatisfied();
}

protected JndiRegistry createRegistry() throws Exception {
    // bind our processor in the registry with the given id
    JndiRegistry reg = super.createRegistry();
    reg.bind("processReport", new ProcessReport());
    return reg;
}

protected RouteBuilder createRouteBuilder() throws Exception {
    return new RouteBuilder() {
        public void configure() throws Exception {
            // the reports from the seda queue is processed by our processor
            // before they are written to files in the target/reports directory
            from("seda:reports").processRef("processReport").to("file://target/test-reports", "mock:result");
        }
    };
}

private class ProcessReport implements Processor {

    public void process(Exchange exchange) throws Exception {
        String body = exchange.getIn().getBody(String.class);
        // do some business logic here

        // set the output to the file
        exchange.getOut().setBody(body);

        // set the output filename using java code logic, notice that this is done by setting
        // a special header property of the out exchange
        exchange.getOut().setHeader(Exchange.FILE_NAME, "report.txt");
    }

}

Write to subdirectory using Exchange.FILE_NAME

Using a single route, it is possible to write a file to any number of subdirectories. If you have a route setup as such:

<route>
    <from uri="bean:myBean"/>
    <to uri="file:/rootDirectory"/>
  </route>

You can have myBean set the header Exchange.FILE_NAME to values such as:

Exchange.FILE_NAME = hello.txt => /rootDirectory/hello.txt
Exchange.FILE_NAME = foo/bye.txt => /rootDirectory/foo/bye.txt

This allows you to have a single route to write files to multiple destinations.

Using expression for filenames

In this sample we want to move consumed files to a backup folder using todays date as a sub foldername:

from("file://inbox?move=backup/${date:now:yyyyMMdd}/${file:name}").to("...");

See File Language for more samples.

Avoiding reading the same file more than once (idempotent consumer)

Camel supports Idempotent Consumer directly within the component so it will skip already processed files. This feature can be enabled by setting the idempotent=true option.

from("file://inbox?idempotent=true").to("...");

By default Camel uses a in memory based store for keeping track of consumed files, it uses a least recently used cache storing holding up to 1000 entries. You can plugin your own implementation of this store by using the idempotentRepository option using the # sign in the value to indicate it's a referring to a bean in the Registry with this id.

<!-- define our store as a plain spring bean -->
   <bean id="myStore" class="com.mycompany.MyIdempotentStore"/>

  <route>
    <from uri="file://inbox?idempotent=true&amp;idempotentRepository=#myStore"/>
    <to uri="bean:processInbox"/>
  </route>

Camel will log at DEBUG level if it skips a file because it has been consumed before:

DEBUG FileConsumer is idempotent and the file has been consumed before. Will skip this file: target\idempotent\report.txt

Using a File based idempotent repository

In this section we will use the file based idempotent repository org.apache.camel.processor.idempotent.FileIdempotentRepository instead of the in memory based that is used as default.
This repository uses a 1st level cache to avoid reading the file repository. It will only use the file repository to store the content of the 1st level cache. Thereby the repository can survive server restarts. It will load the content of the file into the 1st level cache upon startup. The file structure is very simple as it store the key in separate lines in the file. By default the file store has a size limit of 1mb when the file grew larger Camel will truncate the file store be rebuilding the content by flushing the 1st level cache in a fresh empty file.

We configure our repository using Spring XML creating our file idempotent repository and define our file consumer to use our repository with the idempotentRepository using # sign to indicate Registry lookup:

<!-- this is our file based idempotent store configured to use the .filestore.dat as file -->
<bean id="fileStore" class="org.apache.camel.processor.idempotent.FileIdempotentRepository">
    <!-- the filename for the store -->
    <property name="fileStore" value="target/fileidempotent/.filestore.dat"/>
    <!-- the max filesize in bytes for the file. Camel will trunk and flush the cache
         if the file gets bigger -->
    <property name="maxFileStoreSize" value="512000"/>
    <!-- the number of elements in our store -->
    <property name="cacheSize" value="250"/>
</bean>

<camelContext id="camel" xmlns="http://camel.apache.org/schema/spring">
    <route>
        <from uri="file://target/fileidempotent/?idempotent=true&amp;idempotentRepository=#fileStore&amp;move=done/${file:name}"/>
        <to uri="mock:result"/>
    </route>
</camelContext>

Using a JPA based idempotent repository

In this section we will use the JPA based idempotent repository instead of the in memory based that is used as default.

First we need a persistence-unit in META-INF/persistence.xml where we need to use the class org.apache.camel.processor.idempotent.jpa.MessageProcessed as model.

<persistence-unit name="idempotentDb" transaction-type="RESOURCE_LOCAL">
  <class>org.apache.camel.processor.idempotent.jpa.MessageProcessed</class>

  <properties>
    <property name="openjpa.ConnectionURL" value="jdbc:derby:target/idempotentTest;create=true"/>
    <property name="openjpa.ConnectionDriverName" value="org.apache.derby.jdbc.EmbeddedDriver"/>
    <property name="openjpa.jdbc.SynchronizeMappings" value="buildSchema"/>
    <property name="openjpa.Log" value="DefaultLevel=WARN, Tool=INFO"/>
  </properties>
</persistence-unit>

Then we need to setup a Spring jpaTemplate in the spring XML file:

<!-- this is standard spring JPA configuration -->
<bean id="jpaTemplate" class="org.springframework.orm.jpa.JpaTemplate">
    <property name="entityManagerFactory" ref="entityManagerFactory"/>
</bean>

<bean id="entityManagerFactory" class="org.springframework.orm.jpa.LocalEntityManagerFactoryBean">
    <!-- we use idempotentDB as the persitence unit name defined in the persistence.xml file -->
    <property name="persistenceUnitName" value="idempotentDb"/>
</bean>

And finally we can create our JPA idempotent repository in the spring XML file as well:

<!-- we define our jpa based idempotent repository we want to use in the file consumer -->
<bean id="jpaStore" class="org.apache.camel.processor.idempotent.jpa.JpaMessageIdRepository">
    <!-- Here we refer to the spring jpaTemplate -->
    <constructor-arg index="0" ref="jpaTemplate"/>
    <!-- This 2nd parameter is the name  (= a cateogry name).
         You can have different repositories with different names -->
    <constructor-arg index="1" value="FileConsumer"/>
</bean>

And yes then we just need to refer to the jpaStore bean in the file consumer endpoint using the [[idempotentRepository}} using the # syntax option:

<route>
    <from uri="file://inbox?idempotent=true&amp;idempotentRepository=#jpaStore"/>
    <to uri="bean:processInbox"/>
  </route>

Filter using org.apache.camel.component.file.GenericFileFilter

Camel supports pluggable filtering strategies. You can then configure the endpoint with such a filter to skip certain files being processed.

In the sample we have build our own filter that skips files starting with skip in the filename:

public class MyFileFilter implements GenericFileFilter {
    public boolean accept(GenericFile pathname) {
        // we dont accept any files starting with skip in the name
        return !pathname.getFileName().startsWith("skip");
    }
}

And then we can configure our route using the filter attribute to reference our filter (using # notation) that we have defines in the spring XML file:

<!-- define our sorter as a plain spring bean -->
   <bean id="myFilter" class="com.mycompany.MyFileSorter"/>

  <route>
    <from uri="file://inbox?filter=#myFilter"/>
    <to uri="bean:processInbox"/>
  </route>

Filtering using ANT path matcher

The ANT path matcher is shipped out-of-the-box in the camel-spring jar. So you need to depend on camel-spring if you are using Maven.
The reasons is that we leverage Spring's AntPathMatcher to do the actual matching.

The file paths is matched with the following rules:

  • ? matches one character
  • * matches zero or more characters
  • ** matches zero or more directories in a path

The sample below demonstrates how to use it:

<camelContext xmlns="http://camel.apache.org/schema/spring">
    <template id="camelTemplate"/>

    <!-- use myFilter as filter to allow setting ANT paths for which files to scan for -->
    <endpoint id="myFileEndpoint" uri="file://target/antpathmatcher?recursive=true&amp;filter=#myAntFilter"/>

    <route>
        <from ref="myFileEndpoint"/>
        <to uri="mock:result"/>
    </route>
</camelContext>

<!-- we use the antpath file filter to use ant paths for includes and exlucde -->
<bean id="myAntFilter" class="org.apache.camel.component.file.AntPathMatcherGenericFileFilter">
    <!-- include and file in the subfolder that has day in the name -->
    <property name="includes" value="**/subfolder/**/*day*"/>
    <!-- exclude all files with bad in name or .xml files. Use comma to seperate multiple excludes -->
    <property name="excludes" value="**/*bad*,**/*.xml"/>
</bean>

Sorting using Comparator

Camel supports pluggable sorting strategies. This strategy it to use the build in java.util.Comparator in Java. You can then configure the endpoint with such a comparator and have Camel sort the files before being processed.

In the sample we have build our own comparator that just sorts by file name:

public class MyFileSorter implements Comparator<GenericFile> {
    public int compare(GenericFile o1, GenericFile o2) {
        return o1.getFileName().compareTo(o2.getFileName());
    }
}

And then we can configure our route using the sorter option to reference to our sorter (mySorter) we have defined in the spring XML file:

<!-- define our sorter as a plain spring bean -->
   <bean id="mySorter" class="com.mycompany.MyFileSorter"/>

  <route>
    <from uri="file://inbox?sorter=#mySorter"/>
    <to uri="bean:processInbox"/>
  </route>
URI options can reference beans using the # syntax

In the Spring DSL route about notice that we can refer to beans in the Registry by prefixing the id with #. So writing sorter=#mySorter, will instruct Camel to go look in the Registry for a bean with the id mySorter.

Sorting using sortBy

Camel supports pluggable sorting strategies. This strategy it to use the File Language to configure the sorting. The sortBy is configured as:

sortBy=group 1;group 2;group 3;...

Where each group is separated with semi colon. In the simple situations you just use one group, so a simple example could be:

sortBy=file:name

This will sort by file name, you can reverse the order by prefixing reverse: to the group, so the sorting is now Z..A:

sortBy=reverse:file:name

As we have the full power of File Language we can use some of the other parameters, so if we want to sort by file size we do:

sortBy=file:size

You can configure to ignore the case, using ignoreCase: for string comparison, so if you want to use file name sorting but to ignore the case then we do:

sortBy=ignoreCase:file:name

You can combine ignore case and reverse, however reverse must be specified first:

sortBy=reverse:ignoreCase:file:name

In the sample below we want to sort by last modified file, so we do:

sortBy=file:modifed

And then we want to group by name as a 2nd option so files with same modifcation is sorted by name:

sortBy=file:modifed;file:name

Now there is an issue here, can you spot it? Well the modified timestamp of the file is too fine as it will be in millis, but what if we want to sort by date only and then sub group by name?
Well as we have the true power of File Language we can use the its date command that supports patterns. So this can be solved as:

sortBy=date:file:yyyyMMdd;file:name

Yeah that is pretty powerful, oh by the way you can also use reverse per group so we could reverse the file names:

sortBy=date:file:yyyyMMdd;reverse:file:name

Debug logging

This component has log level TRACE that can be helpful if you have problems.

See Also

Powered by Atlassian Confluence (Version: 2.2.9 Build:#527 Sep 07, 2006) - Bug/feature request

Unsubscribe or edit your notifications preferences