hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jean-Marc Spaggiari <jean-m...@spaggiari.org>
Subject Re: Acceptable CPU_WIO % ?
Date Fri, 08 Feb 2013 15:43:19 GMT
Hi Kevin,

I think it will take time before I get a chance to have 5 drives in
the same server, so I will see at that time to test RAID5.

I'm going to add one drive per server today or tomorrow to try to
improve that. What IOPs should I try to have? 100? Less? It will all
be SATA3 drives and I will configure all in RAID0.

It doesn't seems to me to be an issue to lose one node, since data
will be replicated everywhere else. I will "simply" have to replace
the failing disk and restart the node, no?

JM

2013/2/8, Kevin O'dell <kevin.odell@cloudera.com>:
> Azuryy,
>
>   The main reason to recommend against RAID is that it is slow and it adds
> redundancy that we already have in Hadoop.  RAID0 is another story as long
> as all of the drives are healthy and you don't mind losing the whole volume
> if you lose one drive.
>
> JM,
>
>   I would not even waste my time testing RAID5 or RAID6(unless it is just
> for educational purposes :) ).  200+ IOPs consistently on one SATA drive is
> pretty high, that would explain your high I/O wait time.  If your use case
> allows for you to lose the whole node, there is not a good reason for you
> to shy away from RAID0.  Please let us know how this plays out with your
> environment.
>
> On Thu, Feb 7, 2013 at 10:23 PM, Azuryy Yu <azuryyyu@gmail.com> wrote:
>
>> JM,
>>
>> I don't have the context, but if you are using Hadoop/Hbase, so don't do
>> RAID on your disk.
>>
>>
>> On Fri, Feb 8, 2013 at 11:15 AM, Jean-Marc Spaggiari <
>> jean-marc@spaggiari.org> wrote:
>>
>> > Ok. I see. For my usecase I prefer to loose the data and have faster
>> > process. So I will go for RAID0 and keep the replication factor to
>> > 3... If at some point I have 5 disks in the node, I will most probably
>> > give a try to RAID5 and see the performances compared to the other
>> > RAID/JBOD options.
>> >
>> > Is there a "rule", like, 1 HD per core? Or we can't really simplify
>> > that
>> > much?
>> >
>> > So far I have that in the sar output:
>> > 21:35:03          tps      rtps      wtps   bread/s   bwrtn/s
>> > 21:45:03       218,85    215,97      2,88  45441,95    308,04
>> > 21:55:02       209,73    206,67      3,06  43985,28    378,32
>> > 22:05:04       215,03    211,71      3,33  44831,00    312,95
>> > Average :      214,54    211,45      3,09  44753,09    333,07
>> >
>> > But I'm not sure what it means. I will wait for tomorrow to get more
>> > results, but my job will be done over night, so I'm not sure the
>> > average will be accurate...
>> >
>> > JM
>> >
>> >
>> > 2013/2/7, Kevin O'dell <kevin.odell@cloudera.com>:
>> > > JM,
>> > >
>> > >   I think you misunderstood me.  I am not advocating any form of RAID
>> for
>> > > Hadoop.  It is true that we already have redundancy built in with
>> > > HDFS.
>> >  So
>> > > unless you were going to do something silly like sacrifice speed to
>> > > run
>> > > RAID1 or RAID5 and lower your replication to 2...just don't do it :)
>> > >  Anyway, yes you probably should have 3 - 4 drives per node if not
>> more.
>> > >  At that point then the you will really see the benefit of JBOD over
>> > RAID0
>> > >
>> > > Do you want to be able to lose a drive and keep the node up?  If yes,
>> > then
>> > > JBOD is for you.  Do you not care if you lose that node due to drive
>> > > failure? You just need speed, then RAID0 may be the correct choice.
>>  Sar
>> > > will take some time to populate.  Give it about 24 hours and you
>> > > should
>> > be
>> > > able to glean some interesting information.
>> > >
>> > > On Thu, Feb 7, 2013 at 9:50 PM, Jean-Marc Spaggiari
>> > > <jean-marc@spaggiari.org
>> > >> wrote:
>> > >
>> > >> Ok. I see with RAID0 might be better for me compare to JBOD. Also,
>> > >> why
>> > >> do we want to use RAID1 or RAID5? We already have the redundancy
>> > >> done
>> > >> by hadoop, is it not going to add another non-required level of
>> > >> redundancy?
>> > >>
>> > >> Should I already think to have 3 or even 4 drives in each node?
>> > >>
>> > >> I tried sar -A and it's only giving me 2 lines.
>> > >> root@node7:/home/hbase# sar -A
>> > >> Linux 3.2.0-4-amd64 (node7)     2013-02-07      _x86_64_        (4
>> CPU)
>> > >>
>> > >> 21:29:54          LINUX RESTART
>> > >>
>> > >> It was not enabled, so I just enabled it and restart sysstat, but
>> > >> seems that it's still not populated.
>> > >>
>> > >> I have the diskstats plugin installed on ganglia, so I have a LOT of
>> > >> disks information, but not this specific one.
>> > >>
>> > >> My write_bytes_per_sec is pretty low. Average is 232K for the last
2
>> > >> hours. But my erad_bytes_per_sec is avera 22.83M for the same
>> > >> period.
>> > >> The graph is looking like a comb.
>> > >>
>> > >> I just retried sar and some data is coming.. I will need to let it
>> > >> run
>> > >> for few more minutes to get some more data ...
>> > >>
>> > >> JM
>> > >>
>> > >>
>> > >> 2013/2/7, Kevin O'dell <kevin.odell@cloudera.com>:
>> > >> > JM,
>> > >> >
>> > >> >   Okay, I think I see what was happening.  You currently only
have
>> one
>> > >> > drive in the system that is showing High I/O wait correct?  You
>> > >> > are
>> > >> looking
>> > >> > at bringing in a second drive to help distribute the load?  In
>> > >> > your
>> > >> testing
>> > >> > with two drives you saw that RAID0 offerred superior performance
>> > >> > vs
>> > >> > JBOD.
>> > >> >  Typically when we see RAID vs JBOD we are dealing with about
6 -
>> > >> > 12
>> > >> > drives.  Here are some of the pluses and minuses:
>> > >> >
>> > >> > RAID0 - faster performance since the data is striped, but you
are
>> > >> > as
>> > >> > fast
>> > >> > as your slowest drive and one drive failure you lose the whole
>> volume.
>> > >> >
>> > >> > JBOD - Better redundancy and faster than a RAID1, or a RAID5
>> > >> > configuration(unsure about a RAID4), but you are slower than RAID0
>> > >> >
>> > >> > It sounds like since you only have 1 drive in the node right now,
>> you
>> > >> > wouldn't be gaining or losing any redundancy by going with RAID0.
>>  For
>> > >> what
>> > >> > it is worth, I would agree that you are I/O bound.  If you run
a
>> > >> > sar
>> > -A
>> > >> > >
>> > >> > /tmp/sar.out and you take a look at the drive utilization what
is
>> your
>> > >> > TPS(IOPs) count that you are seeing?
>> > >> >
>> > >> > On Thu, Feb 7, 2013 at 9:00 PM, Jean-Marc Spaggiari
>> > >> > <jean-marc@spaggiari.org
>> > >> >> wrote:
>> > >> >
>> > >> >> Hi Kevin,
>> > >> >>
>> > >> >> I'm facing some issues on one of my nodes and I'm trying to
find
>> > >> >> a
>> > way
>> > >> >> to fix that. CPU is used about 10% by user, and 80% for WIO.
So
>> > >> >> I'm
>> > >> >> looking for a way to improve that. The mother board can do
RAIDx
>> and
>> > >> >> JBOD too. It's the server I used few weeks ago to run some
disks
>> > >> >> benchs.
>> > >> >>
>> > >> >> http://www.spaggiari.org/index.php/hbase/hard-drives-performances
>> > >> >>
>> > >> >> The conclusion was that RAID0 was 70% faster than JBOD. But
JBOD
>> was
>> > >> >> faster than RAID1.
>> > >> >>
>> > >> >> I have a 2TB drive in this server and was thinking about just
>> adding
>> > >> >> another 2TB drive.
>> > >> >>
>> > >> >> What are the advantages of JBOD compared to RAID0? From the
last
>> > tests
>> > >> >> I did, it was slower.
>> > >> >>
>> > >> >> Since I will have to re-format the disks anyway, I can re-run
the
>> > >> >> tests just in case I did not configured something properly....
>> > >> >>
>> > >> >> JM
>> > >> >>
>> > >> >> 2013/2/7, Kevin O'dell <kevin.odell@cloudera.com>:
>> > >> >> > Hey JM,
>> > >> >> >
>> > >> >> >   Why RAID0?  That has a lot of disadvantages to using
a JBOD
>> > >> >> > configuration?  Wait I/O is a symptom, not a problem.
 Are you
>> > >> actually
>> > >> >> > experiencing a problem or are you treating for something
you
>> think
>> > >> >> > should
>> > >> >> > be lower?
>> > >> >> >
>> > >> >> > On Thu, Feb 7, 2013 at 8:19 PM, Jean-Marc Spaggiari
>> > >> >> > <jean-marc@spaggiari.org
>> > >> >> >> wrote:
>> > >> >> >
>> > >> >> >> Hi,
>> > >> >> >>
>> > >> >> >> What is an acceptable CPU_WIO % while running an
heavy MR job?
>> > >> >> >> Should
>> > >> >> >> we also try to keep that under 10%? Or it's not realistic
and
>> > >> >> >> we
>> > >> >> >> will
>> > >> >> >> see more about 50%?
>> > >> >> >>
>> > >> >> >> One of my nodes is showing 70% :( It's WAY to much.
I will add
>> > >> another
>> > >> >> >> disk tomorrow and put them in RAID0, but I'm wondering
how low
>> > >> >> >> shoud
>> > >> I
>> > >> >> >> go?
>> > >> >> >>
>> > >> >> >> JM
>> > >> >> >>
>> > >> >> >
>> > >> >> >
>> > >> >> >
>> > >> >> > --
>> > >> >> > Kevin O'Dell
>> > >> >> > Customer Operations Engineer, Cloudera
>> > >> >> >
>> > >> >>
>> > >> >
>> > >> >
>> > >> >
>> > >> > --
>> > >> > Kevin O'Dell
>> > >> > Customer Operations Engineer, Cloudera
>> > >> >
>> > >>
>> > >
>> > >
>> > >
>> > > --
>> > > Kevin O'Dell
>> > > Customer Operations Engineer, Cloudera
>> > >
>> >
>>
>
>
>
> --
> Kevin O'Dell
> Customer Operations Engineer, Cloudera
>

Mime
View raw message