Mailing-List: contact users-help@camel.apache.org; run by ezmlm
Precedence: bulk
Reply-To: users@camel.apache.org
Received-SPF: pass (nike.apache.org: domain of 0x6e6562@gmail.com designates
 209.85.212.42 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <3194D3D7-62E9-446C-A633-2ED200432B6B@gmail.com>
References: <etPan.5278f4fc.6fd4a113.1267b@backd8>
	<CAGB5yNm0xGeky55CZSd0T1x=TVnbSSSAG2f85wiXaqr_7Ac7wQ@mail.gmail.com>
	<C3E681A4-A977-4D9F-8CFE-E52B325ECB43@gmail.com>
	<3194D3D7-62E9-446C-A633-2ED200432B6B@gmail.com>
Date: Tue, 5 Nov 2013 22:43:15 +0000
Message-ID: 
 <CAB-G1jjoJXwiVbTKd=HbKUQp_k4_raPxgfxUwEStjwzS8mM7SA@mail.gmail.com>
Subject: Re: Forcing file write to complete without stream caching
From: Ben Hood <0x6e6562@gmail.com>
To: users@camel.apache.org
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: quoted-printable

Hey guys,

Thanks a lot for your help. So by reading the file2 documentation
instead of the file documentation, I was able to solve my problem by
adding

doneFileName=3D${file:name.noext}.done

to the uri for the file being written and also to the uri of the route
that was attempting the subsequent read.

This allows me to get rid of the hack I put into the processor
component to wait for the file to receive data.

For anybody reading this thread, the relevant documentation on this
feature is under the headings "Using done files" and "Writing done
files".

Thanks for all of your help,

Cheers,

Ben

On Tue, Nov 5, 2013 at 7:03 PM, Taariq Levack <taariql@gmail.com> wrote:
> That's the old file component, have a look at file2.
> http://camel.apache.org/file2.html
>
>> On 05 Nov 2013, at 20:36, Ben Hood <0x6e6562@gmail.com> wrote:
>>
>> Hey Claus,
>>
>> Having to acquire a lock on the file sounds like a good way to implement=
 the "don't start attempting to read an empty file" semantics I'm looking f=
or.
>>
>> Having said that, the documentation on read locks is somewhat misleading=
. It notes a boolean URI parameter called consumer.exclusiveReadLock. The c=
onfiguration processor doesn't seem to consider this to be an acceptable op=
tion - maybe I'm doing something wrong.
>>
>> So turning to the source, the FileProcessStrategyFactory appears to acce=
pt a flag called "readLock", which can be either none, markerFile, fileLock=
, rename or changed. So I went for fileLock. However, it seems that this st=
rategy is not scoped on an individual endpoint, rather it appears to be set=
 globally for the entire camel context (I gained this impression by debuggi=
ng the 2.12.1 release). So it seems that which ever file endpoint is proces=
sed first sets the strategy for the entire context.
>>
>> Or am I missing the point?
>>
>> Cheers,
>>
>> Ben
>>
>>> On Nov 5, 2013, at 15:19, Claus Ibsen <claus.ibsen@gmail.com> wrote:
>>>
>>> If you are talking about how to not pickup new files in a Camel from
>>> route, then take a look at the various read lock documentation on the
>>> file component.
>>>
>>>> On Tue, Nov 5, 2013 at 2:39 PM, Ben Hood <0x6e6562@gmail.com> wrote:
>>>> Hi,
>>>>
>>>> In my first attempt to use Camel I=92ve run into a intra-route timing =
issue that I=92ve only solved with a hack, so I was wondering whether there=
 are any best practices of dealing with timing issues when dealing with mul=
tiple processing steps in a batch file pipeline.
>>>>
>>>> Basically I am trying to avoid doing an HTTP POST with an empty payloa=
d, since the route performing the HTTP POST is triggered before the file th=
at it is wired to upload has been written.
>>>>
>>>> If I turn stream caching on, this problem goes away. However, since so=
me files can be quite big, I=92d prefer not to have do stream caching.
>>>>
>>>> So to solve the issue, I=92ve written a workaround bean that just does=
 a Thread.sleep() in order to wait for the upload file to actually get some=
 data in it before firing off the HTTP POST.
>>>>
>>>> I=92ve got a two step pipeline that:
>>>>
>>>> 1. Transcodes a batch input file into an intermediate format (using ms=
gpack serialization);
>>>> 2. Performs an HTTP POST of the intermediate format to a remote server=
;
>>>>
>>>> I=92d like to keep the intermediate format around on disk for debuggin=
g and manual replay tasks.
>>>>
>>>> My camel context has two routes:
>>>>
>>>> <route id=3D=93transcode-to-msgpack">
>>>>     <from uri=3D"file:/tmp/d"/>
>>>>     <log message=3D"Transcoding ${file:name} to msgpack" />
>>>>     <to uri=3D"bean:transcoder"/>
>>>>     <to uri=3D"file:/tmp/b?fileName=3D${file:name.noext}.msgpack"/>
>>>> </route>
>>>>
>>>> <route id=3D=93post-msgpack-payload">
>>>>     <from uri=3D"file:/tmp/b"/>
>>>>     <from uri=3D"file:/tmp/e"/>
>>>>     <log message=3D"POSTing ${file:name} to the rating API" />
>>>>     <setProperty propertyName=3D"url.template">
>>>>               <constant>http://localhost:9999/calls/:source/:sequence<=
/constant>
>>>>     </setProperty>
>>>>     <process ref=3D=93httpDataPump"/>
>>>> </route>
>>>>
>>>> I have two custom beans doing the work:
>>>>
>>>> 1. transcoder - This takes an InputStream, and returns an InputStream =
that wraps and transcodes the InputStream from the file;
>>>> 2. httpDataPump - This contains an HTTP client that uploads the FileIn=
putStream the the InMessage from the Exchange refers.
>>>>
>>>> Doing a Thread.sleep() seems like a real hack to me, so I was wonderin=
g if there is a more idiomatic way to solve the issue. I=92ve looked into t=
he preMoveNamePrefix options, but they appear to apply only to input files.=
 Ideally I=92m looking for something that can move the output file after it=
 has been written.
>>>>
>>>> Any pointers are appreciated.
>>>>
>>>> Cheers,
>>>>
>>>> Ben
>>>
>>>
>>>
>>> --
>>> Claus Ibsen
>>> -----------------
>>> Red Hat, Inc.
>>> Email: cibsen@redhat.com
>>> Twitter: davsclaus
>>> Blog: http://davsclaus.com
>>> Author of Camel in Action: http://www.manning.com/ibsen