Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (nike.apache.org: domain of yuzhihong@gmail.com designates
 209.85.212.172 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <51BE77A7231E9E488E58380AFBD7727225EF1983@mail1.impetus.co.in>
References: <51BE77A7231E9E488E58380AFBD7727225EF1879@mail1.impetus.co.in>
	<CALte62yG4-+KAdGXEA8q999PnERvQDMBUH33_cR4R4ckBDBLoQ@mail.gmail.com>
	<51BE77A7231E9E488E58380AFBD7727225EF18C6@mail1.impetus.co.in>
	<CALte62xETOYuFo=WzaRFGXLTUCLC5BbiyRtGaNQcdkpXu8HKwg@mail.gmail.com>
	<51BE77A7231E9E488E58380AFBD7727225EF194C@mail1.impetus.co.in>
	<CAPQV63UZXEwxgBuGnttwinmysrV7RA5bZKuhd=GOw9OuyEoiDQ@mail.gmail.com>
	<51BE77A7231E9E488E58380AFBD7727225EF1983@mail1.impetus.co.in>
Date: Mon, 25 Mar 2013 12:39:13 -0700
Message-ID: 
 <CALte62ykVzPh_=h5KG9fG-SL55ELbiwf1B43FDPMCEz=Mkbb6A@mail.gmail.com>
Subject: Re: HBase Writes With Large Number of Columns
From: Ted Yu <yuzhihong@gmail.com>
To: user@hbase.apache.org
Content-Type: multipart/alternative; boundary=047d7bfd0c2ea6cc0d04d8c4f5ab

--047d7bfd0c2ea6cc0d04d8c4f5ab
Content-Type: text/plain; charset=ISO-8859-1

bq. These records are being written using batch mutation with thrift API
This is an important information, I think.

Batch mutation through Java API would incur lower overhead.

On Mon, Mar 25, 2013 at 11:40 AM, Pankaj Misra
<pankaj.misra@impetus.co.in>wrote:

> Firstly, Thanks a lot Jean and Ted for your extended help, very much
> appreciate it.
>
> Yes Ted I am writing to all the 40 columns and 1.5 Kb of record data is
> distributed across these columns.
>
> Jean, some columns are storing as small as a single byte value, while few
> of the columns are storing as much as 80-125 bytes of data. The overall
> record size is 1.5 KB. These records are being written using batch mutation
> with thrift API, where in we are writing 100 records per batch mutation.
>
> Thanks and Regards
> Pankaj Misra
>
>
> ________________________________________
> From: Jean-Marc Spaggiari [jean-marc@spaggiari.org]
> Sent: Monday, March 25, 2013 11:57 PM
> To: user@hbase.apache.org
> Subject: Re: HBase Writes With Large Number of Columns
>
> I just ran some LoadTest to see if I can reproduce that.
>
> bin/hbase org.apache.hadoop.hbase.util.LoadTestTool -write 4:512:100
> -num_keys 1000000
> 13/03/25 14:18:25 INFO util.MultiThreadedAction: [W:100] Keys=997172,
> cols=3,8m, time=00:03:55 Overall: [keys/s= 4242, latency=23 ms]
> Current: [keys/s=4413, latency=22 ms], insertedUpTo=-1
>
> bin/hbase org.apache.hadoop.hbase.util.LoadTestTool -write 100:512:100
> -num_keys 1000000
>
> This one crashed because I don't have enought disk space, so I'm
> re-running it, but just before it crashed it was showing about 24.5
> slower. which is coherent since it's writing 25 more columns.
>
> What size of data do you have? Big cells? Small cells? I will retry
> the test above with more lines and keep you posted.
>
> 2013/3/25 Pankaj Misra <pankaj.misra@impetus.co.in>:
> > Yes Ted, you are right, we are having table regions pre-split, and we
> see that both regions are almost evenly filled in both the tests.
> >
> > This does not seem to be a regression though, since we were getting good
> write rates when we had lesser number of columns.
> >
> > Thanks and Regards
> > Pankaj Misra
> >
> >
> > ________________________________________
> > From: Ted Yu [yuzhihong@gmail.com]
> > Sent: Monday, March 25, 2013 11:15 PM
> > To: user@hbase.apache.org
> > Cc: ankitjaincs06@gmail.com
> > Subject: Re: HBase Writes With Large Number of Columns
> >
> > Copying Ankit who raised the same question soon after Pankaj's initial
> > question.
> >
> > On one hand I wonder if this was a regression in 0.94.5 (though
> unlikely).
> >
> > Did the region servers receive (relatively) same write load for the
> second
> > test case ? I assume you have pre-split your tables in both cases.
> >
> > Cheers
> >
> > On Mon, Mar 25, 2013 at 10:18 AM, Pankaj Misra
> > <pankaj.misra@impetus.co.in>wrote:
> >
> >> Hi Ted,
> >>
> >> Sorry for missing that detail, we are using HBase version 0.94.5
> >>
> >> Regards
> >> Pankaj Misra
> >>
> >>
> >> ________________________________________
> >> From: Ted Yu [yuzhihong@gmail.com]
> >> Sent: Monday, March 25, 2013 10:29 PM
> >> To: user@hbase.apache.org
> >> Subject: Re: HBase Writes With Large Number of Columns
> >>
> >> If you give us the version of HBase you're using, that would give us
> some
> >> more information to help you.
> >>
> >> Cheers
> >>
> >> On Mon, Mar 25, 2013 at 9:55 AM, Pankaj Misra <
> pankaj.misra@impetus.co.in
> >> >wrote:
> >>
> >> > Hi,
> >> >
> >> > The issue that I am facing is around the performance drop of Hbase,
> when
> >> I
> >> > was having 20 columns in a column family Vs now when I am having 40
> >> columns
> >> > in a column family. The number of columns have doubled and the
> >> > ingestion/write speed has also dropped by half. I am writing 1.5 KB of
> >> data
> >> > per row across 40 columns.
> >> >
> >> > Are there any settings that I should look into for tweaking Hbase to
> >> write
> >> > higher number of columns faster?
> >> >
> >> > I would request community's help to let me know how can I write to a
> >> > column family with large number of columns efficiently.
> >> >
> >> > Would greatly appreciate any help /clues around this issue.
> >> >
> >> > Thanks and Regards
> >> > Pankaj Misra
> >> >
> >> > ________________________________
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > NOTE: This message may contain information that is confidential,
> >> > proprietary, privileged or otherwise protected by law. The message is
> >> > intended solely for the named addressee. If received in error, please
> >> > destroy and notify the sender. Any use of this email is prohibited
> when
> >> > received in error. Impetus does not represent, warrant and/or
> guarantee,
> >> > that the integrity of this communication has been maintained nor that
> the
> >> > communication is free of errors, virus, interception or interference.
> >> >
> >>
> >> ________________________________
> >>
> >>
> >>
> >>
> >>
> >>
> >> NOTE: This message may contain information that is confidential,
> >> proprietary, privileged or otherwise protected by law. The message is
> >> intended solely for the named addressee. If received in error, please
> >> destroy and notify the sender. Any use of this email is prohibited when
> >> received in error. Impetus does not represent, warrant and/or guarantee,
> >> that the integrity of this communication has been maintained nor that
> the
> >> communication is free of errors, virus, interception or interference.
> >>
> >
> > ________________________________
> >
> >
> >
> >
> >
> >
> > NOTE: This message may contain information that is confidential,
> proprietary, privileged or otherwise protected by law. The message is
> intended solely for the named addressee. If received in error, please
> destroy and notify the sender. Any use of this email is prohibited when
> received in error. Impetus does not represent, warrant and/or guarantee,
> that the integrity of this communication has been maintained nor that the
> communication is free of errors, virus, interception or interference.
>
> ________________________________
>
>
>
>
>
>
> NOTE: This message may contain information that is confidential,
> proprietary, privileged or otherwise protected by law. The message is
> intended solely for the named addressee. If received in error, please
> destroy and notify the sender. Any use of this email is prohibited when
> received in error. Impetus does not represent, warrant and/or guarantee,
> that the integrity of this communication has been maintained nor that the
> communication is free of errors, virus, interception or interference.
>

--047d7bfd0c2ea6cc0d04d8c4f5ab--