Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: general@hadoop.apache.org
Received-SPF: neutral (athena.apache.org: local policy)
MIME-Version: 1.0
In-Reply-To: 
 <7CA0D5FE7FA83048893E9230C1E9C0280B8C0AF0@DNVREMSA01.jsq.bsg.ad.adp.com>
References: <590321.36008.qm@web31813.mail.mud.yahoo.com>
 <536061.77419.qm@web130101.mail.mud.yahoo.com>
 <A632B3F93D861843A3C966F3962C3F84010822D5000C@EMAIL05.pnl.gov>
 <4D18AF52.9080800@mozilla.com>
 <A632B3F93D861843A3C966F3962C3F84010822D50026@EMAIL05.pnl.gov>
 <AANLkTimxVqwnJDbq9pi7weR2KTAijWpUX1+_xqvAemTw@mail.gmail.com>
 <1293579543.5455.150.camel@sledge.emsl.pnl.gov>
 <A632B3F93D861843A3C966F3962C3F84010822D50029@EMAIL05.pnl.gov>
 <7CA0D5FE7FA83048893E9230C1E9C0280B8C0AF0@DNVREMSA01.jsq.bsg.ad.adp.com>
From: Ted Dunning <tdunning@maprtech.com>
Date: Wed, 29 Dec 2010 13:19:00 -0800
Message-ID: <AANLkTinkSt8Kf1QD97oSoLLe=A0cFTzC6Oi4FKzOcr1Z@mail.gmail.com>
Subject: Re: What is the fastest way to get a large amount of data into the
 Hadoop HDFS file system (or Hbase)?
To: user@hbase.apache.org
Cc: general@hadoop.apache.org, "Fox, Kevin M" <kevin.fox@pnl.gov>,
	Patrick Angeles <patrick@cloudera.com>, "Brown,
 David M JR" <david.brown@pnl.gov>
Content-Type: multipart/alternative; boundary=20cf30050ec05687570498931fcd

--20cf30050ec05687570498931fcd
Content-Type: text/plain; charset=ISO-8859-1

The problem there is that HDFS isn't a first class file system.  That means
that the nice and easy ways of mounting will lead to problems (notably NFS
which maintains no state will require random write capabilities).

On Wed, Dec 29, 2010 at 1:16 PM, Hiller, Dean (Contractor) <
dean.hiller@broadridge.com> wrote:

> I wonder if having linux mount hdfs would help here so as people put the
> file on your linux /hdfs directory, it was actually writing to hdfs and
> not linux ;) (yeah, you still have that one machine bottle neck as the
> files come in unless that can be clustered too somehow).  Just google
> mounting hdfs from linux....something that sounds pretty cool that we
> may be using later.
>
> Later,
> Dean
>
> -----Original Message-----
> From: Taylor, Ronald C [mailto:ronald.taylor@pnl.gov]
> Sent: Tuesday, December 28, 2010 5:05 PM
> To: Fox, Kevin M; Patrick Angeles
> Cc: general@hadoop.apache.org; user@hbase.apache.org; Brown, David M JR;
> Taylor, Ronald C
> Subject: RE: What is the fastest way to get a large amount of data into
> the Hadoop HDFS file system (or Hbase)?
>
>
> Hi Kevin,
>
> So - from what Patrick and Ted are saying it sounds like we want the
> best way to parallelize a source-based push, rather than doing a
> parallelized pull through a MapReduce program. And I see that what you
> ask about below is on parallelizing a push, so we are on the same page.
>  Ron
>
> -----Original Message-----
> From: Fox, Kevin M
> Sent: Tuesday, December 28, 2010 3:39 PM
> To: Patrick Angeles
> Cc: general@hadoop.apache.org; user@hbase.apache.org; Taylor, Ronald C;
> Brown, David M JR
> Subject: Re: What is the fastest way to get a large amount of data into
> the Hadoop HDFS file system (or Hbase)?
>
> On Tue, 2010-12-28 at 14:26 -0800, Patrick Angeles wrote:
> > Ron,
> >
> >
> > While MapReduce can help to parallelize the load effort, your likely
> > bottleneck is the source system (where the files come from). If the
> > files are coming from a single server, then parallelizing the load
> > won't gain you much past a certain point. You have to figure in how
> > fast you can read the file(s) off disk(s) and push the bits through
> > your network and finally onto HDFS.
> >
> >
> > The best scenario is if you can parallelize the reads and have a fat
> > network pipe (10GbE or more) going into your Hadoop cluster.
>
>
> We have a way to parallelize a push from the archive storage cluster to
> the hadoop storage cluster.
>
> Is there a way to target a particular storage node with a push into the
> hadoop file system? The hadoop cluster nodes are 1gig attached to its
> core switch and we have a 10 gig uplink to the core from the storage
> archive. Say, we have 4 nodes in each storage cluster (we have more,
> just a simplified example):
>
> a0 --\                                /-- h0
> a1 --+                                +-- h1
> a2 --+ (A switch) -10gige- (h switch) +-- h2
> a3 --/                                \-- h3
>
> I want to be able to have a0 talk to h0 and not have h0 decide the data
> belongs on h3, slowing down a3's ability to write data into h3, greatly
> reducing bandwidth.
>
> Thanks,
> Kevin
>
> >
> >
> > Regards,
> >
> >
> > - Patrick
> >
> > On Tue, Dec 28, 2010 at 5:04 PM, Taylor, Ronald C
> > <ronald.taylor@pnl.gov> wrote:
> >
> >         Folks,
> >
> >         We plan on uploading large amounts of data on a regular basis
> >         onto a Hadoop cluster, with Hbase operating on top of Hadoop.
> >         Figure eventually on the order of multiple terabytes per week.
> >         So - we are concerned about doing the uploads themselves as
> >         fast as possible from our native Linux file system into HDFS.
> >         Figure files will be in, roughly, the 1 to 300 GB range.
> >
> >         Off the top of my head, I'm thinking that doing this in
> >         parallel using a Java MapReduce program would work fastest. So
> >         my idea would be to have a file listing all the data files
> >         (full paths) to be uploaded, one per line, and then use that
> >         listing file as input to a MapReduce program.
> >
> >         Each Mapper would then upload one of the data files (using
> >         "hadoop fs -copyFromLocal <source> <dest>") in parallel with
> >         all the other Mappers, with the Mappers operating on all the
> >         nodes of the cluster, spreading out the file upload across the
> >         nodes.
> >
> >         Does that sound like a wise way to approach this? Are there
> >         better methods? Anything else out there for doing automated
> >         upload in parallel? We would very much appreciate advice in
> >         this area, since we believe upload speed might become a
> >         bottleneck.
> >
> >          - Ron Taylor
> >
> >         ___________________________________________
> >         Ronald Taylor, Ph.D.
> >         Computational Biology & Bioinformatics Group
> >
> >         Pacific Northwest National Laboratory
> >         902 Battelle Boulevard
> >         P.O. Box 999, Mail Stop J4-33
> >         Richland, WA  99352 USA
> >         Office:  509-372-6568
> >         Email: ronald.taylor@pnl.gov
> >
> >
> >
> >
>
>
> This message and any attachments are intended only for the use of the
> addressee and
> may contain information that is privileged and confidential. If the reader
> of the
> message is not the intended recipient or an authorized representative of
> the
> intended recipient, you are hereby notified that any dissemination of this
> communication is strictly prohibited. If you have received this
> communication in
> error, please notify us immediately by e-mail and delete the message and
> any
> attachments from your system.
>
>

--20cf30050ec05687570498931fcd--