Return-Path: 
 <issues-return-8138-apmail-camel-issues-archive=camel.apache.org@camel.apache.org>
X-Original-To: apmail-camel-issues-archive@minotaur.apache.org
Delivered-To: apmail-camel-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 9C6E9E534
	for <apmail-camel-issues-archive@minotaur.apache.org>;
 Tue, 22 Jan 2013 08:22:16 +0000 (UTC)
Received: (qmail 66718 invoked by uid 500); 22 Jan 2013 08:22:16 -0000
Delivered-To: apmail-camel-issues-archive@camel.apache.org
Received: (qmail 66287 invoked by uid 500); 22 Jan 2013 08:22:14 -0000
Mailing-List: contact issues-help@camel.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:issues-help@camel.apache.org>
List-Unsubscribe: <mailto:issues-unsubscribe@camel.apache.org>
List-Post: <mailto:issues@camel.apache.org>
List-Id: <issues.camel.apache.org>
Reply-To: dev@camel.apache.org
Delivered-To: mailing list issues@camel.apache.org
Received: (qmail 66226 invoked by uid 99); 22 Jan 2013 08:22:13 -0000
Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Jan 2013 08:22:13 +0000
Date: Tue, 22 Jan 2013 08:22:13 +0000 (UTC)
From: "Claus Ibsen (JIRA)" <jira@apache.org>
To: issues@camel.apache.org
Message-ID: <JIRA.12627641.1358255631638.172324.1358842933144@arcas>
In-Reply-To: <JIRA.12627641.1358255631638@arcas>
References: <JIRA.12627641.1358255631638@arcas>
Subject: [jira] [Commented] (CAMEL-5971) HdfsOutputStream is not closing
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/CAMEL-5971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13559476#comment-13559476 ] 

Claus Ibsen commented on CAMEL-5971:
------------------------------------

Why is closing a stream not efficient? 

Is the idea to keep the stream open just in case there is a lot of writes to HDFS?
But even if the stream is open, the data must be flushed at some point? Or does HDFS only let "other parties" see the file when the stream is closed.
So you can write to the stream 1000 times and only when you close its visibile for other parties?

I dont see a point of keeping the stream open, when the end user has no way of closing it after usage.
Either we should
- always close the stream after writing in the producer
- use some background task to close the stream if idle for more than X
- allow people to explicit close the stream in some way
- add new option: closeStream=true|false which people can turn off|on to have old behavior etc


                
> HdfsOutputStream is not closing
> -------------------------------
>
>                 Key: CAMEL-5971
>                 URL: https://issues.apache.org/jira/browse/CAMEL-5971
>             Project: Camel
>          Issue Type: Bug
>          Components: camel-hdfs
>    Affects Versions: 2.10.3
>            Reporter: Joe Luo
>            Assignee: Willem Jiang
>             Fix For: 2.10.4, 2.11.0
>
>
> I have a simple camel route that takes file from a camel-file consumer endpoint and sends to a camel-hdfs producer endpoint:
> <from uri="file:/local/workspace/inbox?delete=true"/>
> <to uri="hdfs://localhost:9000/local/workspace/outbox/file1"/>
> However, my Hadoop server only creates a zero length file "file1.opened" unless I stop camel route or a splitting condition is met with a "splitStratedy" option added to URI. In above cases, a file called "file1" is created with proper contents and the "file1.opened" is disappeared.  
> It looks like that close() function of HdfsOutputStream is never called unless the camel route/context is stopping or we are splitting the file by looking at source code.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira