Mailing-List: contact dev-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hbase.apache.org
Received-SPF: pass (athena.apache.org: domain of todd@cloudera.com designates
 209.85.215.50 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CADcMMgHpZ0r=xextLBcdyZzdKXqWbANGu8nrOYgNvjP1B2QyGA@mail.gmail.com>
References: <DA8340397F7BAE41B8102757834CEB124FA5D7C8@ex-mbox1>
 <CADcMMgHpZ0r=xextLBcdyZzdKXqWbANGu8nrOYgNvjP1B2QyGA@mail.gmail.com>
From: Todd Lipcon <todd@cloudera.com>
Date: Tue, 29 Jul 2014 10:09:15 -0700
Message-ID: 
 <CADY20s5nF7Dg9MMeO+jrt5NFjKOW9gjpmABcag41iH29_s=37w@mail.gmail.com>
Subject: Re: About "dfs.client-write-packet-size" setting
To: dev <dev@hbase.apache.org>
Content-Type: multipart/alternative; boundary=001a11c3edf0995b9504ff581ac1

--001a11c3edf0995b9504ff581ac1
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

What about a patch to HDFS to reuse these buffers in a pool?


On Tue, Jul 29, 2014 at 10:08 AM, Stack <stack@duboce.net> wrote:

> You the man @liang xie.  Let me try your suggestion here on my little tes=
t
> bench.  Lets get the below into refguide also....
> St.Ack
>
>
> On Mon, Jul 28, 2014 at 8:03 PM, =E8=B0=A2=E8=89=AF <xieliang@xiaomi.com>=
 wrote:
>
> > The default dfs.client-write-packet-size value is 64k, at least it's in
> my
> > Hadoop2 env.
> > I did a benchmark about i via ycsb loading 2 million records(3*200
> bytes):
> > 1) dfs.client-write-packet-size=3D64k ygc count:399, ygct:4.208s
> > 2) dfs.client-write-packet-size=3D8k ygc count:163, ygct:2.644s
> > you see, it's about 40% benefit on gct:)
> > It's because: in DFSOutputStream.Packet class, each "Create a new packe=
t"
> > operation,
> > will call "buf =3D new byte[PacketHeader.PKT_MAX_HEADER_LEN + pktSize];=
",
> > here "pktSize" comes from dfs.client-write-packet-size setting, and in
> > HBase write scenario,
> > we sync WAL asap, so all the new packets are very small
> > (in my ycsb testing, most of them were only hundreds of bytes, or a few
> > kilo bytes),
> > rarely reached to 64k, so always allocating 64k array is just a waste.
> > It would be better that if we add it to refguide note:)
> >
> > ps; 8k just a test setting, we should set it according the real kv size
> > pattern.
> >
> > Thanks,
> >
>


--=20
Todd Lipcon
Software Engineer, Cloudera

--001a11c3edf0995b9504ff581ac1--