hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Payne <er...@yahoo-inc.com>
Subject RE: HDFS on trunk is now quite slow
Date Wed, 06 Jul 2011 17:16:10 GMT
No, it's not that easy to set up the environment. I have 10 nodes, 1 is the NN and 9 are running
MiniDFSCluster to simulate about 100 datanodes each. There are a lot of specific configurations
that need to be set, as well as the HDFS-1875 patch to MiniDFSCluster.

Once the environment is set up, the org.apache.hadoop.fs.loadGenerator.LoadGenerator test
program (hadoop-hdfs-test.jar) is run on the client with 6 threads that randomly create, write,
read, and delete 'simulated' files.

Thanks,
-Eric

> -----Original Message-----
> From: Todd Lipcon [mailto:todd@cloudera.com]
> Sent: Wednesday, July 06, 2011 11:12 AM
> To: hdfs-dev@hadoop.apache.org
> Subject: Re: HDFS on trunk is now quite slow
> 
> On Wed, Jul 6, 2011 at 9:00 AM, Eric Payne <ericp@yahoo-inc.com> wrote:
> 
> > I will attempt to recreate the tests on 20.203.
> >
> > Currently, I'm comparing trunk against branches/MR-279/, and the
> slowdown
> > is many times slower. I have run several tests (45 or 50) with different
> > variables, and they all seem to be slower on trunk.
> >
> > Just for example, in one test here are my findings:
> >
> > Operation                       Trunk     branches/MR-279/
> > -------------------             -----     ----------------
> > Average operations per second:   24       200
> > Average open execution time:     41ms       5ms
> > Average deletion time:           43ms       5ms
> > Average creation time:           47ms       9ms
> > Average write close time:       658ms      100ms
> >
> >
> Seems pretty bad. Which test case is it that you're running? Something
> that's easy for others to reproduce? How many concurrent threads access
> the
> NN? If you jstack the NN do you see some particular lock causing lots of
> contention?
> 
> -Todd
> 
> 
> > > -----Original Message-----
> > > From: Todd Lipcon [mailto:todd@cloudera.com]
> > > Sent: Wednesday, July 06, 2011 10:26 AM
> > > To: hdfs-dev@hadoop.apache.org
> > > Subject: Re: HDFS on trunk is now quite slow
> > >
> > > On Wed, Jul 6, 2011 at 6:54 AM, Eric Payne <ericp@yahoo-inc.com>
> wrote:
> > >
> > > > Thanks Todd.
> > > >
> > > > Yes, the stress test is NN-only. The simulated datanodes (using
> > > > MiniDFSCluster) don't read or write actual data, only log the
> metadata.
> > > >
> > > > So, it sounds like the slowdown on the NN is to be expected,
> correct?
> > > The
> > > > race condition I was experiencing before is no longer happening, so
> the
> > > > benefit of correct locking has resulted in an acceptable slowdown on
> > the
> > > > namenode. Is that correct?
> > > >
> > >
> > > How does the slowdown compare to 0.20.203 for example? We may have
> made
> > > the
> > > locking _too_ coarse -- ie overcompensated for the bug.
> > >
> > > -Todd
> > >
> > >
> > > >
> > > > Thanks,
> > > > -Eric
> > > >
> > > > > -----Original Message-----
> > > > > From: Todd Lipcon [mailto:todd@cloudera.com]
> > > > > Sent: Friday, July 01, 2011 7:49 PM
> > > > > To: hdfs-dev@hadoop.apache.org
> > > > > Subject: Re: HDFS on trunk is now quite slow
> > > > >
> > > > > My guess is HDFS-988 caused the slowdown by coarsening some
> locking
> > > that
> > > > > was
> > > > > previously incorrect. Your stress test is NN-only (metadata ops),
> not
> > > an
> > > > > I/O
> > > > > benchmark, right? I/O should be faster in trunk than ever before.
> > > > >
> > > > > -Todd
> > > > >
> > > > > On Fri, Jul 1, 2011 at 8:23 AM, Eric Payne
> <eric.payne1000@yahoo.com
> > >
> > > > > wrote:
> > > > >
> > > > > > Hi gang,
> > > > > >
> > > > > > I ran some stress tests on the latest HDFS trunk yesterday,
and
> the
> > > > > > performance
> > > > > > is a lot slower (sometimes 10 times slower) when compared with
> the
> > > HDFS
> > > > > in
> > > > > > MR-279. The HDFS in MR-279 is slightly behind trunk. The
> stability
> > > of
> > > > > > HDFS trunk
> > > > > > seems to be better than HDFS MR-279, but I'm not sure if the
> > > slowness
> > > > is
> > > > > > just
> > > > > > avoiding the race contitions or if they are actually fixed in
> > trunk.
> > > > > >
> > > > > > At this point, I'm not sure what is causing this performance
> > > disparity.
> > > > > I
> > > > > > notice
> > > > > > that Block management has recently undergone significant changes
> in
> > > > > trunk.
> > > > > > It
> > > > > > has some new locking and it is now in its own package. Could
> this
> > be
> > > > > part
> > > > > > of the
> > > > > > cause?
> > > > > >
> > > > > > Thanks,
> > > > > > -Eric
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Todd Lipcon
> > > > > Software Engineer, Cloudera
> > > >
> > >
> > >
> > >
> > > --
> > > Todd Lipcon
> > > Software Engineer, Cloudera
> >
> 
> 
> 
> --
> Todd Lipcon
> Software Engineer, Cloudera

Mime
View raw message