Mailing-List: contact dev-help@camel.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@camel.apache.org
Date: Mon, 17 Oct 2011 18:51:12 +0000 (UTC)
From: "Ben Hoyt (Updated) (JIRA)" <jira@apache.org>
To: dev@camel.apache.org
Message-ID: 
 <1400739994.1513.1318877472299.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: 
 <1272984865.1509.1318877472147.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (CAMEL-4555) Support nested directories with
 multiple segment files in the HDFS endpoint consumer
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


     [ https://issues.apache.org/jira/browse/CAMEL-4555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ben Hoyt updated CAMEL-4555:
----------------------------

    Attachment: hdfs.patch
    
> Support nested directories with multiple segment files in the HDFS endpoint consumer
> ------------------------------------------------------------------------------------
>
>                 Key: CAMEL-4555
>                 URL: https://issues.apache.org/jira/browse/CAMEL-4555
>             Project: Camel
>          Issue Type: Improvement
>          Components: camel-hdfs
>    Affects Versions: 2.8.1
>            Reporter: Ben Hoyt
>              Labels: hdfs
>         Attachments: hdfs.patch
>
>   Original Estimate: 72h
>  Remaining Estimate: 72h
>
> A common pattern in HDFS is for multiple segment files underneath a given directory, representing the fragments of data.  Lots of tools understand to automatically merge these segment files (ie hadoop fs -getmerge, pig script loaders).  This patch does the same for the HDFS consumer, using a temporary local directory for the merging.
> Additionally, tools like pig and oozie understand to look for a _SUCCESS file in one of these directories containing segments.  This file indicates that the segments have been completely written.  This patch additionally skips the directory if a _SUCCESS file is not present.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira