Mailing-List: contact mapreduce-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: mapreduce-user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of harsh@cloudera.com designates
 209.85.216.176 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CACvfG=rnt=g0+_KW0SxjyMFY0JgQBRcbFYtHxyEc9o4rsoaVgg@mail.gmail.com>
References: 
 <CACvfG=rnt=g0+_KW0SxjyMFY0JgQBRcbFYtHxyEc9o4rsoaVgg@mail.gmail.com>
From: Harsh J <harsh@cloudera.com>
Date: Fri, 13 Jul 2012 11:44:15 +0530
Message-ID: 
 <CAOcnVr2G21BMgA5KiBFHjjQPsWErg9mqM7NNO9FsZrTOvh0upw@mail.gmail.com>
Subject: Re: suggest Best way to upload xml files to HDFS
To: mapreduce-user@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

If you're looking at automated file/record/event collection, take a
look at Apache Flume: http://incubator.apache.org/flume/. It does well
for distributed collections as well and is very configurable.

Otherwise, write a scheduled script to do the uploads every X period
(your choice). Consider using
https://github.com/edwardcapriolo/filecrush or similar tools too, if
your files are much small and getting in the way of MR processing.

On Fri, Jul 13, 2012 at 8:59 AM, Manoj Babu <manoj444@gmail.com> wrote:
> Hi,
>
> I need to upload large xml files files daily. Right now am having a small
> program to read all the files from local folder and writing it to HDFS as a
> single file. Is this a right way?
> If there any best practices or optimized way to achieve this Kindly let me
> know.
>
> Thanks in advance!
>
> Cheers!
> Manoj.
>


-- 
Harsh J