hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Todd Lipcon <t...@cloudera.com>
Subject Re: HDFS on trunk is now quite slow
Date Wed, 06 Jul 2011 15:26:19 GMT
On Wed, Jul 6, 2011 at 6:54 AM, Eric Payne <ericp@yahoo-inc.com> wrote:

> Thanks Todd.
>
> Yes, the stress test is NN-only. The simulated datanodes (using
> MiniDFSCluster) don't read or write actual data, only log the metadata.
>
> So, it sounds like the slowdown on the NN is to be expected, correct? The
> race condition I was experiencing before is no longer happening, so the
> benefit of correct locking has resulted in an acceptable slowdown on the
> namenode. Is that correct?
>

How does the slowdown compare to 0.20.203 for example? We may have made the
locking _too_ coarse -- ie overcompensated for the bug.

-Todd


>
> Thanks,
> -Eric
>
> > -----Original Message-----
> > From: Todd Lipcon [mailto:todd@cloudera.com]
> > Sent: Friday, July 01, 2011 7:49 PM
> > To: hdfs-dev@hadoop.apache.org
> > Subject: Re: HDFS on trunk is now quite slow
> >
> > My guess is HDFS-988 caused the slowdown by coarsening some locking that
> > was
> > previously incorrect. Your stress test is NN-only (metadata ops), not an
> > I/O
> > benchmark, right? I/O should be faster in trunk than ever before.
> >
> > -Todd
> >
> > On Fri, Jul 1, 2011 at 8:23 AM, Eric Payne <eric.payne1000@yahoo.com>
> > wrote:
> >
> > > Hi gang,
> > >
> > > I ran some stress tests on the latest HDFS trunk yesterday, and the
> > > performance
> > > is a lot slower (sometimes 10 times slower) when compared with the HDFS
> > in
> > > MR-279. The HDFS in MR-279 is slightly behind trunk. The stability of
> > > HDFS trunk
> > > seems to be better than HDFS MR-279, but I'm not sure if the slowness
> is
> > > just
> > > avoiding the race contitions or if they are actually fixed in trunk.
> > >
> > > At this point, I'm not sure what is causing this performance disparity.
> > I
> > > notice
> > > that Block management has recently undergone significant changes in
> > trunk.
> > > It
> > > has some new locking and it is now in its own package. Could this be
> > part
> > > of the
> > > cause?
> > >
> > > Thanks,
> > > -Eric
> >
> >
> >
> >
> > --
> > Todd Lipcon
> > Software Engineer, Cloudera
>



-- 
Todd Lipcon
Software Engineer, Cloudera

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message