hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harry Waye <hw...@arachnys.com>
Subject Re: Optimizing bulk load performance
Date Thu, 24 Oct 2013 21:02:55 GMT
So just a short update, I'll read into it a little more tomorrow.  This is
from three of the nodes:
https://gist.github.com/hazzadous/1264af7c674e1b3cf867

The first is the grey guy.  Just glancing at it, it looks to fluctuate more
than the others.  I guess that could suggest that there are some issues
with reading from the disks.  Interestingly, it's the only one that doesn't
have smartd installed, which alerts us on changes for the other nodes.  I
suspect there's probably some mileage in checking its smart attributes.
 Will do that tomorrow though.

Out of curiosity, how do people normally monitor disk issues?  I'm going to
set up collectd to push various things from smartctl tomorrow, at the
moment all we do is receive emails, which is mostly noise about problem
sector counts increasing +1.


On 24 October 2013 19:40, Jean-Marc Spaggiari <jean-marc@spaggiari.org>wrote:

> Can you try vmstat 2? 2 is the interval in seconds it will display the disk
> usage. On the extract here, nothing is running. only 8% is used. (1% disk
> IO, 6% User, 1% sys)
>
> Run it on 2 or 3 different nodes while you are putting the load on the
> cluster. And take a look at the 4 last numbers and see what the value of
> the last one?
>
> On the usercpu0 graph, who is the gray guy showing hight?
>
> JM
>
> 2013/10/24 Harry Waye <hwaye@arachnys.com>
>
> > Ok I'm running a load job atm, I've add some possibly incomprehensible
> > coloured lines to the graph: http://goo.gl/cUGCGG
> >
> > This is actually with one fewer nodes due to decommissioning to replace a
> > disk, hence I guess the reason for one squiggly line showing no disk
> > activity.  I've included only the cpu stats for CPU0 from each node.  The
> > last graph should read "Memory Used".  vmstat from one of the nodes:
> >
> > procs -----------memory---------- ---swap-- -----io---- -system--
> > ----cpu----
> >  r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy
> id
> > wa
> >  6  0      0 392448 524668 43823900    0    0   501  1044    0    0  6  1
> > 91  1
> >
> > To me the wait doesn't seem that high.  Job stats are
> > http://goo.gl/ZYdUKp,  the job setup is
> > https://gist.github.com/hazzadous/ac57a384f2ab685f07f6
> >
> > Does anything jump out at you?
> >
> > Cheers
> > H
> >
> >
> > On 24 October 2013 16:16, Harry Waye <hwaye@arachnys.com> wrote:
> >
> > > Hi JM
> > >
> > > I took a snapshot on the initial run, before the changes:
> > >
> >
> https://www.evernote.com/shard/s95/sh/b8e1516d-7c49-43f0-8b5f-d16bbdd3fe13/00d7c6cd6dd9fba92d6f00f90fb54fc1/res/4f0e20a2-1ecb-4085-8bc8-b3263c23afb5/screenshot.png
> > >
> > > Good timing, disks appear to be exploding (ATA errors) atm thus I'm
> > > decommissioning and reprovisioning with new disks.  I'll be
> > reprovisioning
> > > as without RAID (it's software RAID just to compound the issue)
> although
> > > not sure how I'll go about migrating all nodes.  I guess I'd need to
> put
> > > more correctly speced nodes in the rack and decommission the existing.
> > >  Makes diff. to
> > >
> > > We're using hetzner at the moment which may not have been a good
> choice.
> > >  Has anyone had any experience with them wrt. Hadoop?  They offer 7 and
> > 15
> > > disk options, but are low on the cpu front (quad core).  Our workload
> > will
> > > be I assume on the high side.  There's also a 8 disk Dell PowerEdge
> what
> > is
> > > a little more powerful.  What hosting providers would people
> recommended?
> > >  (And what would be the strategy for migrating?)
> > >
> > > Anyhow, when I have things more stable I'll have a look at checking out
> > > what's using the cpu.  In the mean time, can anything be gleamed from
> the
> > > above snap?
> > >
> > > Cheers
> > > H
> > >
> > >
> > > On 24 October 2013 15:14, Jean-Marc Spaggiari <jean-marc@spaggiari.org
> > >wrote:
> > >
> > >> Hi Harry,
> > >>
> > >> Do you have more details on the exact load? Can you run vmstats and
> see
> > >> what kind of load it is? Is it user? cpu? wio?
> > >>
> > >> I suspect your disks to be the issue. There is 2 things here.
> > >>
> > >> First, we don't recommend RAID for the HDFS/HBase disk. The best is to
> > >> simply mount the disks on 2 mounting points and give them to HDFS.
> > >> Second, 2 disks per not is very low. On a dev cluster is not even
> > >> recommended. In production, you should go with 12 or more.
> > >>
> > >> So with only 2 disks in RAID, I suspect your WIO to be high which is
> > what
> > >> might slow your process.
> > >>
> > >> Can you take a look on that direction? If it's not that, we will
> > continue
> > >> to investigate ;)
> > >>
> > >> Thanks,
> > >>
> > >> JM
> > >>
> > >>
> > >> 2013/10/23 Harry Waye <hwaye@arachnys.com>
> > >>
> > >> > I'm trying to load data into hbase using HFileOutputFormat and
> > >> incremental
> > >> > bulk load but am getting rather lackluster performance, 10h for
> ~0.5TB
> > >> > data, ~50000 blocks.  This is being loaded into a table that has 2
> > >> > families, 9 columns, 2500 regions and is ~10TB in size.  Keys are
> md5
> > >> > hashes and regions are pretty evenly spread.  The majority of time
> > >> appears
> > >> > to be spend in the reduce phase, with the map phase completing very
> > >> > quickly.  The network doesn't appear to be saturated, but the load
> is
> > >> > consistently at 6 which is the number or reduce tasks per node.
> > >> >
> > >> > 12 hosts (6 cores, 2 disk as RAID0, 1GB eth, no one else on the
> rack).
> > >> >
> > >> > MR conf: 6 mappers, 6 reducers per node.
> > >> >
> > >> > I spoke to someone on IRC and they recommended reducing job output
> > >> > replication to 1, and reducing the number of mappers which I reduced
> > to
> > >> 2.
> > >> >  Reducing replication appeared not to make any difference, reducing
> > >> > reducers appeared just to slow the job down.  I'm going to have a
> look
> > >> at
> > >> > running the benchmarks mentioned on Michael Noll's blog and see what
> > >> that
> > >> > turns up.  I guess some questions I have are:
> > >> >
> > >> > How does the global number/size of blocks affect perf.?  (I have a
> lot
> > >> of
> > >> > 10mb files, which are the input files)
> > >> >
> > >> > How does the job local number/size of input blocks affect perf.?
> > >> >
> > >> > What is actually happening in the reduce phase that requires so much
> > >> CPU?
> > >> >  I assume the actual construction of HFiles isn't intensive.
> > >> >
> > >> > Ultimately, how can I improve performance?
> > >> > Thanks
> > >> >
> > >>
> > >
> > >
> > >
> > > --
> > > Harry Waye, Co-founder/CTO
> > > harry@arachnys.com
> > > +44 7890 734289
> > >
> > > Follow us on Twitter: @arachnys <https://twitter.com/#!/arachnys>
> > >
> > > ---
> > > Arachnys Information Services Limited is a company registered in
> England
> > &
> > > Wales. Company number: 7269723. Registered office: 40 Clarendon St,
> > > Cambridge, CB1 1JX.
> > >
> >
> >
> >
> > --
> > Harry Waye, Co-founder/CTO
> > harry@arachnys.com
> > +44 7890 734289
> >
> > Follow us on Twitter: @arachnys <https://twitter.com/#!/arachnys>
> >
> > ---
> > Arachnys Information Services Limited is a company registered in England
> &
> > Wales. Company number: 7269723. Registered office: 40 Clarendon St,
> > Cambridge, CB1 1JX.
> >
>



-- 
Harry Waye, Co-founder/CTO
harry@arachnys.com
+44 7890 734289

Follow us on Twitter: @arachnys <https://twitter.com/#!/arachnys>

---
Arachnys Information Services Limited is a company registered in England &
Wales. Company number: 7269723. Registered office: 40 Clarendon St,
Cambridge, CB1 1JX.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message