camel-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Claus Ibsen (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CAMEL-5971) HdfsOutputStream is not closing
Date Tue, 22 Jan 2013 08:22:13 GMT

    [ https://issues.apache.org/jira/browse/CAMEL-5971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13559476#comment-13559476
] 

Claus Ibsen commented on CAMEL-5971:
------------------------------------

Why is closing a stream not efficient? 

Is the idea to keep the stream open just in case there is a lot of writes to HDFS?
But even if the stream is open, the data must be flushed at some point? Or does HDFS only
let "other parties" see the file when the stream is closed.
So you can write to the stream 1000 times and only when you close its visibile for other parties?

I dont see a point of keeping the stream open, when the end user has no way of closing it
after usage.
Either we should
- always close the stream after writing in the producer
- use some background task to close the stream if idle for more than X
- allow people to explicit close the stream in some way
- add new option: closeStream=true|false which people can turn off|on to have old behavior
etc


                
> HdfsOutputStream is not closing
> -------------------------------
>
>                 Key: CAMEL-5971
>                 URL: https://issues.apache.org/jira/browse/CAMEL-5971
>             Project: Camel
>          Issue Type: Bug
>          Components: camel-hdfs
>    Affects Versions: 2.10.3
>            Reporter: Joe Luo
>            Assignee: Willem Jiang
>             Fix For: 2.10.4, 2.11.0
>
>
> I have a simple camel route that takes file from a camel-file consumer endpoint and sends
to a camel-hdfs producer endpoint:
> <from uri="file:/local/workspace/inbox?delete=true"/>
> <to uri="hdfs://localhost:9000/local/workspace/outbox/file1"/>
> However, my Hadoop server only creates a zero length file "file1.opened" unless I stop
camel route or a splitting condition is met with a "splitStratedy" option added to URI. In
above cases, a file called "file1" is created with proper contents and the "file1.opened"
is disappeared.  
> It looks like that close() function of HdfsOutputStream is never called unless the camel
route/context is stopping or we are splitting the file by looking at source code.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message