hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jean-Marc Spaggiari <jean-m...@spaggiari.org>
Subject Re: Optimizing bulk load performance
Date Thu, 24 Oct 2013 21:28:25 GMT
Your nodes are almost 50% idle... Might be something else. Sound it's not
your disks nor your CPU... Maybe to many RCPs?

Have you investigate on your network side? netperf might be a good help for
you.

JM


2013/10/24 Harry Waye <hwaye@arachnys.com>

> p.s. I guess this is more turning into a general hadoop issue, but I'll
> keep the discussion here seeing that I have an audience, unless there are
> objections.
>
>
> On 24 October 2013 22:02, Harry Waye <hwaye@arachnys.com> wrote:
>
> > So just a short update, I'll read into it a little more tomorrow.  This
> is
> > from three of the nodes:
> > https://gist.github.com/hazzadous/1264af7c674e1b3cf867
> >
> > The first is the grey guy.  Just glancing at it, it looks to fluctuate
> > more than the others.  I guess that could suggest that there are some
> > issues with reading from the disks.  Interestingly, it's the only one
> that
> > doesn't have smartd installed, which alerts us on changes for the other
> > nodes.  I suspect there's probably some mileage in checking its smart
> > attributes.  Will do that tomorrow though.
> >
> > Out of curiosity, how do people normally monitor disk issues?  I'm going
> > to set up collectd to push various things from smartctl tomorrow, at the
> > moment all we do is receive emails, which is mostly noise about problem
> > sector counts increasing +1.
> >
> >
> > On 24 October 2013 19:40, Jean-Marc Spaggiari <jean-marc@spaggiari.org
> >wrote:
> >
> >> Can you try vmstat 2? 2 is the interval in seconds it will display the
> >> disk
> >> usage. On the extract here, nothing is running. only 8% is used. (1%
> disk
> >> IO, 6% User, 1% sys)
> >>
> >> Run it on 2 or 3 different nodes while you are putting the load on the
> >> cluster. And take a look at the 4 last numbers and see what the value of
> >> the last one?
> >>
> >> On the usercpu0 graph, who is the gray guy showing hight?
> >>
> >> JM
> >>
> >> 2013/10/24 Harry Waye <hwaye@arachnys.com>
> >>
> >> > Ok I'm running a load job atm, I've add some possibly incomprehensible
> >> > coloured lines to the graph: http://goo.gl/cUGCGG
> >> >
> >> > This is actually with one fewer nodes due to decommissioning to
> replace
> >> a
> >> > disk, hence I guess the reason for one squiggly line showing no disk
> >> > activity.  I've included only the cpu stats for CPU0 from each node.
> >>  The
> >> > last graph should read "Memory Used".  vmstat from one of the nodes:
> >> >
> >> > procs -----------memory---------- ---swap-- -----io---- -system--
> >> > ----cpu----
> >> >  r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us
> sy
> >> id
> >> > wa
> >> >  6  0      0 392448 524668 43823900    0    0   501  1044    0    0  6
> >>  1
> >> > 91  1
> >> >
> >> > To me the wait doesn't seem that high.  Job stats are
> >> > http://goo.gl/ZYdUKp,  the job setup is
> >> > https://gist.github.com/hazzadous/ac57a384f2ab685f07f6
> >> >
> >> > Does anything jump out at you?
> >> >
> >> > Cheers
> >> > H
> >> >
> >> >
> >> > On 24 October 2013 16:16, Harry Waye <hwaye@arachnys.com> wrote:
> >> >
> >> > > Hi JM
> >> > >
> >> > > I took a snapshot on the initial run, before the changes:
> >> > >
> >> >
> >>
> https://www.evernote.com/shard/s95/sh/b8e1516d-7c49-43f0-8b5f-d16bbdd3fe13/00d7c6cd6dd9fba92d6f00f90fb54fc1/res/4f0e20a2-1ecb-4085-8bc8-b3263c23afb5/screenshot.png
> >> > >
> >> > > Good timing, disks appear to be exploding (ATA errors) atm thus I'm
> >> > > decommissioning and reprovisioning with new disks.  I'll be
> >> > reprovisioning
> >> > > as without RAID (it's software RAID just to compound the issue)
> >> although
> >> > > not sure how I'll go about migrating all nodes.  I guess I'd need
to
> >> put
> >> > > more correctly speced nodes in the rack and decommission the
> existing.
> >> > >  Makes diff. to
> >> > >
> >> > > We're using hetzner at the moment which may not have been a good
> >> choice.
> >> > >  Has anyone had any experience with them wrt. Hadoop?  They offer
7
> >> and
> >> > 15
> >> > > disk options, but are low on the cpu front (quad core).  Our
> workload
> >> > will
> >> > > be I assume on the high side.  There's also a 8 disk Dell PowerEdge
> >> what
> >> > is
> >> > > a little more powerful.  What hosting providers would people
> >> recommended?
> >> > >  (And what would be the strategy for migrating?)
> >> > >
> >> > > Anyhow, when I have things more stable I'll have a look at checking
> >> out
> >> > > what's using the cpu.  In the mean time, can anything be gleamed
> from
> >> the
> >> > > above snap?
> >> > >
> >> > > Cheers
> >> > > H
> >> > >
> >> > >
> >> > > On 24 October 2013 15:14, Jean-Marc Spaggiari <
> >> jean-marc@spaggiari.org
> >> > >wrote:
> >> > >
> >> > >> Hi Harry,
> >> > >>
> >> > >> Do you have more details on the exact load? Can you run vmstats
and
> >> see
> >> > >> what kind of load it is? Is it user? cpu? wio?
> >> > >>
> >> > >> I suspect your disks to be the issue. There is 2 things here.
> >> > >>
> >> > >> First, we don't recommend RAID for the HDFS/HBase disk. The best
is
> >> to
> >> > >> simply mount the disks on 2 mounting points and give them to HDFS.
> >> > >> Second, 2 disks per not is very low. On a dev cluster is not even
> >> > >> recommended. In production, you should go with 12 or more.
> >> > >>
> >> > >> So with only 2 disks in RAID, I suspect your WIO to be high which
> is
> >> > what
> >> > >> might slow your process.
> >> > >>
> >> > >> Can you take a look on that direction? If it's not that, we will
> >> > continue
> >> > >> to investigate ;)
> >> > >>
> >> > >> Thanks,
> >> > >>
> >> > >> JM
> >> > >>
> >> > >>
> >> > >> 2013/10/23 Harry Waye <hwaye@arachnys.com>
> >> > >>
> >> > >> > I'm trying to load data into hbase using HFileOutputFormat
and
> >> > >> incremental
> >> > >> > bulk load but am getting rather lackluster performance, 10h
for
> >> ~0.5TB
> >> > >> > data, ~50000 blocks.  This is being loaded into a table that
has
> 2
> >> > >> > families, 9 columns, 2500 regions and is ~10TB in size. 
Keys are
> >> md5
> >> > >> > hashes and regions are pretty evenly spread.  The majority
of
> time
> >> > >> appears
> >> > >> > to be spend in the reduce phase, with the map phase completing
> very
> >> > >> > quickly.  The network doesn't appear to be saturated, but
the
> load
> >> is
> >> > >> > consistently at 6 which is the number or reduce tasks per
node.
> >> > >> >
> >> > >> > 12 hosts (6 cores, 2 disk as RAID0, 1GB eth, no one else
on the
> >> rack).
> >> > >> >
> >> > >> > MR conf: 6 mappers, 6 reducers per node.
> >> > >> >
> >> > >> > I spoke to someone on IRC and they recommended reducing job
> output
> >> > >> > replication to 1, and reducing the number of mappers which
I
> >> reduced
> >> > to
> >> > >> 2.
> >> > >> >  Reducing replication appeared not to make any difference,
> reducing
> >> > >> > reducers appeared just to slow the job down.  I'm going to
have a
> >> look
> >> > >> at
> >> > >> > running the benchmarks mentioned on Michael Noll's blog and
see
> >> what
> >> > >> that
> >> > >> > turns up.  I guess some questions I have are:
> >> > >> >
> >> > >> > How does the global number/size of blocks affect perf.? 
(I have
> a
> >> lot
> >> > >> of
> >> > >> > 10mb files, which are the input files)
> >> > >> >
> >> > >> > How does the job local number/size of input blocks affect
perf.?
> >> > >> >
> >> > >> > What is actually happening in the reduce phase that requires
so
> >> much
> >> > >> CPU?
> >> > >> >  I assume the actual construction of HFiles isn't intensive.
> >> > >> >
> >> > >> > Ultimately, how can I improve performance?
> >> > >> > Thanks
> >> > >> >
> >> > >>
> >> > >
> >> > >
> >> > >
> >> > > --
> >> > > Harry Waye, Co-founder/CTO
> >> > > harry@arachnys.com
> >> > > +44 7890 734289
> >> > >
> >> > > Follow us on Twitter: @arachnys <https://twitter.com/#!/arachnys>
> >> > >
> >> > > ---
> >> > > Arachnys Information Services Limited is a company registered in
> >> England
> >> > &
> >> > > Wales. Company number: 7269723. Registered office: 40 Clarendon St,
> >> > > Cambridge, CB1 1JX.
> >> > >
> >> >
> >> >
> >> >
> >> > --
> >> > Harry Waye, Co-founder/CTO
> >> > harry@arachnys.com
> >> > +44 7890 734289
> >> >
> >> > Follow us on Twitter: @arachnys <https://twitter.com/#!/arachnys>
> >> >
> >> > ---
> >> > Arachnys Information Services Limited is a company registered in
> >> England &
> >> > Wales. Company number: 7269723. Registered office: 40 Clarendon St,
> >> > Cambridge, CB1 1JX.
> >> >
> >>
> >
> >
> >
> > --
> > Harry Waye, Co-founder/CTO
> > harry@arachnys.com
> > +44 7890 734289
> >
> > Follow us on Twitter: @arachnys <https://twitter.com/#!/arachnys>
> >
> > ---
> > Arachnys Information Services Limited is a company registered in England
> &
> > Wales. Company number: 7269723. Registered office: 40 Clarendon St,
> > Cambridge, CB1 1JX.
> >
>
>
>
> --
> Harry Waye, Co-founder/CTO
> harry@arachnys.com
> +44 7890 734289
>
> Follow us on Twitter: @arachnys <https://twitter.com/#!/arachnys>
>
> ---
> Arachnys Information Services Limited is a company registered in England &
> Wales. Company number: 7269723. Registered office: 40 Clarendon St,
> Cambridge, CB1 1JX.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message