camel-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chirag Dewan <chirag.dewa...@yahoo.in>
Subject Re: A basic query with using Direct Component
Date Fri, 20 Jun 2014 02:46:09 GMT
Got the problem.


The issue was actually with HDFSConsumer-FTPProducer route. HDFS consumer creates a .opened
file before reading and FTP distribution was failing,due to which .opened files remains in
the HDFS directory. And these files are not picked once the distribution resumes.

Now I can try and use an empty string suffix for the consumer route so that it does not ignore
files.

Thanks for the assistance.

Chirag



________________________________
 From: Claus Ibsen <claus.ibsen@gmail.com>
To: "users@camel.apache.org" <users@camel.apache.org> 
Sent: Thursday, 19 June 2014 12:16 PM
Subject: Re: A basic query with using Direct Component
 

I am not sure, maybe try setting the openedSuffix to an empty string
to see if that works?

On Thu, Jun 19, 2014 at 8:39 AM, Chirag Dewan <chirag.dewan22@yahoo.in> wrote:
> Hi Claus,
>
> Thanks for response.
>
> I am using Camel 2.12.1 with camel-hdfs 2.
>
> Yes,that is the problem. Write operation fails and .opened file is still there on HDFS.
Is there any option through which I can skip creating .opened file?
>
>
> Chirag
>
>
>
>
>
>
>
>
>
>
> ________________________________
>  From: Claus Ibsen <claus.ibsen@gmail.com>
> To: "users@camel.apache.org" <users@camel.apache.org>
> Sent: Thursday, 19 June 2014 11:37 AM
> Subject: Re: A basic query with using Direct Component
>
>
> On Thu, Jun 19, 2014 at 7:11 AM, Chirag Dewan <chirag.dewan22@yahoo.in> wrote:
>> Hi All,
>>
>> I have a route with direct as following :
>>
>> from("direct:abc").to("hdfs2:xxxxx?splitStrategy=MESSAGE");
>>
>> I use the route write files in HDFS. I use the producer template to send a OutputStream
to direct:
>>
>> producerTemplate,sendBody("direct:abc",OutputStream);
>>
>> Now HDFS producer creates an .opened file first and renames it upon writing the file.
>>
>>
>> So,I will only get the callback from direct,once the file is written and renamed.
Is this correct?
>>
>
> Yes
>
>>
>> Somehow my route has finished but .opene files is still in the HDFS directory,which
in turn is polled by a HDFS consumer who fail to poll the .opened file.
>>
>
> I suspect either the hdfs write failed or something, leaving the
> .opened file orphaned.
>
> And the consumer ought not to poll the .opened file. I would have
> assumed the consumer skips these kind of files.
>
> What version of Camel do you use?
>
>
>
>
>
>>
>> BR,
>>
>> Chirag
>
>
>
> --
> Claus Ibsen
> -----------------
> Red Hat, Inc.
> Email: cibsen@redhat.com
> Twitter: davsclaus
> Blog: http://davsclaus.com
> Author of Camel in Action: http://www.manning.com/ibsen
> hawtio: http://hawt.io/
> fabric8: http://fabric8.io/






-- 
Claus Ibsen
-----------------
Red Hat, Inc.
Email: cibsen@redhat.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
hawtio: http://hawt.io/
fabric8: http://fabric8.io/
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message