Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (athena.apache.org: domain of xietao.mailbox@gmail.com
 designates 209.85.160.41 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=YjztWU1w1JGIwNa0t+616jshnlH44UYSmRiI3BR/1BO1lihCZuLfh7tfYHpP7UgyR8
         nbRyMalc/PNyo2lxCbZT/hW8Njzz2zvucV+gyVmIsXA9UMJIaNhK2wof+SLgWHEfARD2
         mKqa4mSDz39WUrSSerEV7Bu/d//hBK9u+jeag=
MIME-Version: 1.0
In-Reply-To: <AANLkTimxqLn9yOfgjHZN3xQ4OH35K0Dq-FdVb2DAr7T7@mail.gmail.com>
References: <AANLkTikuN+zCn6=hAPd6QtedSOZDWw+M_obKQO3vvgBi@mail.gmail.com>
	<AANLkTimxqLn9yOfgjHZN3xQ4OH35K0Dq-FdVb2DAr7T7@mail.gmail.com>
Date: Fri, 10 Sep 2010 12:47:59 +0800
Message-ID: <AANLkTimZB6QTPbcHvQ31V78tAjV5HpQk-iCEHKDX=Pn1@mail.gmail.com>
Subject: Re: ycsb test on hbase
From: Tao Xie <xietao.mailbox@gmail.com>
To: user@hbase.apache.org
Content-Type: multipart/alternative; boundary=000e0cd17ce27332ff048fe073fd

--000e0cd17ce27332ff048fe073fd
Content-Type: text/plain; charset=ISO-8859-1

You mean "hbase.hregion.max.filesize", I set it to 2G. But I see more
waiting time and lower ops. I have no real use case for intensive write. I
just use YCSB to do a performance test.

2010-09-10 11:20:42,796   332 sec: 2437024 operations; 7509.85 current
ops/sec; [INSERT AverageLatency(ms)=35.09]
2010-09-10 11:20:52,801   342 sec: 2437024 operations; 0 current ops/sec;
2010-09-10 11:21:02,806   352 sec: 2441720 operations; 469.37 current
ops/sec; [INSERT AverageLatency(ms)=21.37]
2010-09-10 11:21:12,807   362 sec: 2451112 operations; 939.11 current
ops/sec; [INSERT AverageLatency(ms)=40.12]
2010-09-10 11:21:22,809   372 sec: 2569586 operations; 11845.03 current
ops/sec; [INSERT AverageLatency(ms)=31.16]
2010-09-10 11:21:32,811   382 sec: 2714088 operations; 14447.31 current
ops/sec; [INSERT AverageLatency(ms)=33.44]
2010-09-10 11:21:42,814   392 sec: 2718784 operations; 469.46 current
ops/sec; [INSERT AverageLatency(ms)=34.64]
2010-09-10 11:21:52,815   402 sec: 2779832 operations; 6104.19 current
ops/sec; [INSERT AverageLatency(ms)=36.46]
2010-09-10 11:22:02,817   412 sec: 2930104 operations; 15024.2 current
ops/sec; [INSERT AverageLatency(ms)=38.77]
2010-09-10 11:22:12,819   422 sec: 3009936 operations; 7981.6 current
ops/sec; [INSERT AverageLatency(ms)=43.41]
2010-09-10 11:22:22,821   432 sec: 3009936 operations; 0 current ops/sec;
2010-09-10 11:22:32,823   442 sec: 3009936 operations; 0 current ops/sec;
2010-09-10 11:22:42,825   452 sec: 3144094 operations; 13413.12 current
ops/sec; [INSERT AverageLatency(ms)=56.82]
2010-09-10 11:22:52,827   462 sec: 3310480 operations; 16635.27 current
ops/sec; [INSERT AverageLatency(ms)=34.46]
2010-09-10 11:23:02,829   472 sec: 3338656 operations; 2817.04 current
ops/sec; [INSERT AverageLatency(ms)=20.91]
2010-09-10 11:23:12,831   482 sec: 3338656 operations; 0 current ops/sec;
2010-09-10 11:23:22,832   492 sec: 3438535 operations; 9986.9 current
ops/sec; [INSERT AverageLatency(ms)=26.74]
2010-09-10 11:23:35,600   505 sec: 3566729 operations; 10040.26 current
ops/sec; [INSERT AverageLatency(ms)=27.53]
2010-09-10 11:23:45,601   515 sec: 3620416 operations; 5368.16 current
ops/sec; [INSERT AverageLatency(ms)=48.66]
2010-09-10 11:23:55,603   525 sec: 3620416 operations; 0 current ops/sec;
2010-09-10 11:24:05,605   535 sec: 3620416 operations; 0 current ops/sec;
2010-09-10 11:24:15,607   545 sec: 3648592 operations; 2817.04 current
ops/sec; [INSERT AverageLatency(ms)=52.15]

2010/9/10 Jean-Daniel Cryans <jdcryans@apache.org>

> If you have a very heavy write load (like YCSB when only inserting),
> then you really have to tune HBase for that kind of workload since
> it's not the "normal" use case. Setting MAX_FILESIZE really high
> (1-2GB) and even pre-splitting the table when creating it (available
> in 0.89) will help.
>
> Most of the time spent waiting is due to splitting and blocking due to
> either MemStores growing over their max size and the global MemStore
> size limit being reached. It's kinda rough and could probably be
> "smoother", but do you really have a use case that requires it or just
> poking?
>
> J-D
>
> On Thu, Sep 9, 2010 at 7:32 PM, Tao Xie <xietao.mailbox@gmail.com> wrote:
> > hi, all
> > I use YCSB to measure the insert/read latency of hbase.
> > I found there will be 0 records inserted in up to 10 seconds during the
> > insertion procedure.
> > See the following result at 1514 second. I want to know why this occurs.
> Is
> > this due to compaction?
> > And I also want to know why the ops/sec varies all the time. Seems no a
> > stable time.
> > Thanks.
> >
> > 2010-09-10 00:07:29,608   1484 sec: 28786280 operations; 23475.3 current
> > ops/sec; [INSERT AverageLatency(ms)=8.81]
> > 2010-09-10 00:07:39,610   1494 sec: 28842632 operations; 5634.07 current
> > ops/sec; [INSERT AverageLatency(ms)=6.68]
> > 2010-09-10 00:07:49,612   1504 sec: 28964728 operations; 12207.16 current
> > ops/sec; [INSERT AverageLatency(ms)=7.68]
> > 2010-09-10 00:07:59,614   1514 sec: 28964728 operations; 0 current
> ops/sec;
> > 2010-09-10 00:08:10,778   1525 sec: 29130475 operations; 14846.56 current
> > ops/sec; [INSERT AverageLatency(ms)=24.45]
> > 2010-09-10 00:08:20,782   1535 sec: 29606967 operations; 47630.15 current
> > ops/sec; [INSERT AverageLatency(ms)=12.64]
> > 2010-09-10 00:08:30,784   1545 sec: 29908624 operations; 30159.67 current
> > ops/sec; [INSERT AverageLatency(ms)=0.12]
> > 2010-09-10 00:08:40,786   1555 sec: 30016632 operations; 10798.64 current
> > ops/sec; [INSERT AverageLatency(ms)=5.66]
> >
>

--000e0cd17ce27332ff048fe073fd--