Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (nike.apache.org: domain of kevin.odell@cloudera.com
 designates 209.85.215.50 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <BLU0-SMTP44DA61EAFC30AA9C1997D18F390@phx.gbl>
References: 
 <CAMPmjhkSfze3vUHLcXzedRwqC4Zom+-0+cSNsmwcZ_2p9yguSA@mail.gmail.com>
	<CAPQV63XtzB20pbJnDOgaygO_AJQ_qU0V1=pjNWkwZ4EarMURKQ@mail.gmail.com>
	<CAMPmjhnrpRHNyJ2ionikfFg5JXRhsTHU-F5vv92fAgZJBZY7aw@mail.gmail.com>
	<EEA57224-3509-469E-AA8B-5E4331BA040A@gmail.com>
	<CAMPmjhk6y5HB1MBW+u=yf+NeSX4x+mfjjX7QBSZdRytEwDg4CQ@mail.gmail.com>
	<CAMPNBf5e4=T441tU+f-2AGgkDHg_5jLgZpazB09NERDwZ6+Vag@mail.gmail.com>
	<CAMPmjh=0rikzv+hj-DsJ0dkh0N6ZEh7oqaMWAdc_X7yVg1ys3w@mail.gmail.com>
	<CAMPmjhkPEf2AT1eJjCqaKNApemea4z8W9m-cCM2BuUWC0oBgmg@mail.gmail.com>
	<BLU0-SMTP44DA61EAFC30AA9C1997D18F390@phx.gbl>
Date: Wed, 11 Sep 2013 11:02:02 -0400
Message-ID: 
 <CAGngS9cAd58WJrOfGe62Jf691vcgXq8RYtydNpwK6s2Sb_sUYQ@mail.gmail.com>
Subject: Re: HBase Region Server crash if column size become to big
From: "Kevin O'dell" <kevin.odell@cloudera.com>
To: "user@hbase.apache.org" <user@hbase.apache.org>
Content-Type: multipart/alternative; boundary=001a11c3327a62b20c04e61ce7f5

--001a11c3327a62b20c04e61ce7f5
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

I have not see the exact error, but if I recall correctly jobs will fail if
the column is larger than 10MB and we have not raised the default
setting(which I don't have in front of me) ?


On Wed, Sep 11, 2013 at 10:53 AM, Michael Segel
<michael_segel@hotmail.com>wrote:

> Just out of curiosity...
>
> How wide are the columns?
>
> What's the region size?
>
> Does anyone know the error message you'll get if your row is wider than a
> region?
>
>
> On Sep 11, 2013, at 9:47 AM, John <johnnyenglish739@gmail.com> wrote:
>
> > sry, I mean 570000 columns, not rows
> >
> >
> > 2013/9/11 John <johnnyenglish739@gmail.com>
> >
> >> thanks for all the answers! The only entry I got in the
> >> "hbase-cmf-hbase1-REGIONSERVER-mydomain.org.log.out" log file after I
> >> executing the get command in the hbase shell is this:
> >>
> >> 2013-09-11 16:38:56,175 WARN org.apache.hadoop.ipc.HBaseServer:
> >> (operationTooLarge): {"processingtimems":3196,"client":"
> 192.168.0.1:50629
> >>
> ","timeRange":[0,9223372036854775807],"starttimems":1378910332920,"respon=
sesize":108211303,"class":"HRegionServer","table":"P_SO","cacheBlocks":true=
,"families":{"myCf":["ALL"]},"row":"myRow","queuetimems":0,"method":"get","=
totalColumns":1,"maxVersions":1}
> >>
> >> After this the RegionServer is down, nothing more. BTW I found out tha=
t
> >> the row should have ~570000 rows. The size should be arround ~70mb
> >>
> >> Thanks
> >>
> >>
> >>
> >> 2013/9/11 Bing Jiang <jiangbinglover@gmail.com>
> >>
> >>> hi john.
> >>> I think it is a fresh question. Could you print the log from the
> >>> regionserver crashed ?
> >>> On Sep 11, 2013 8:38 PM, "John" <johnnyenglish739@gmail.com> wrote:
> >>>
> >>>> Okay, I will take a look at the ColumnPaginationFilter.
> >>>>
> >>>> I tried to reproduce the error. I created a new table and add one ne=
w
> >>> row
> >>>> with 250 000 columns, but everything works fine if I execute a get t=
o
> >>> the
> >>>> table. The only difference to my original programm was that I have
> added
> >>>> the data directly throught the hbase java api and not with the map
> >>> reduce
> >>>> bulk load. Maybe that can be the reason?
> >>>>
> >>>> I wonder a little bit about the hdfs structure if I compare both
> methods
> >>>> (hbase api/bulk load). If I add the data through the hbase api there
> is
> >>> no
> >>>> file in
> >>> /hbase/MyTable/5faaf42997925e2f637d8d38c420862f/MyColumnFamily/*,
> >>>> but if I use the bulk load method there is a file for every time I
> >>> executed
> >>>> a new bulk load:
> >>>>
> >>>> root@pc11:~/hadoop# hadoop fs -ls
> >>>> /hbase/mytestTable/5faaf42997925e2f637d8d38c420862f/mycf
> >>>> root@pc11:~/hadoop# hadoop fs -ls
> >>>> /hbase/bulkLoadTable/f95294bd3c8651a7bbdf9fac27f8961a/mycf2/
> >>>> Found 2 items
> >>>> -rw-r--r--   1 root supergroup  118824462 2013-09-11 11:46
> >>>>
> >>>>
> >>>
> /hbase/bulkLoadTable/f95294bd3c8651a7bbdf9fac27f8961a/mycf2/28e919a0cc8a4=
592b7f2c09defaaea3a
> >>>> -rw-r--r--   1 root supergroup  158576842 2013-09-11 11:35
> >>>>
> >>>>
> >>>
> /hbase/bulkLoadTable/f95294bd3c8651a7bbdf9fac27f8961a/mycf2/35c5e6df64c04=
d0a880ffe82593258b8
> >>>>
> >>>> If I ececute a get operation in the hbase shell to my the "MyTable"
> >>> table
> >>>> if got the result:
> >>>>
> >>>> hbase(main):004:0> get 'mytestTable', 'sampleRowKey'
> >>>> ... <-- all results
> >>>> 250000 row(s) in 38.4440 seconds
> >>>>
> >>>> but if I try to get the results for my "bulkLoadTable" I got this (+
> the
> >>>> region server crash):
> >>>>
> >>>> hbase(main):003:0> get 'bulkLoadTable', 'oneSpecificRowKey'
> >>>> COLUMN                          CELL
> >>>>
> >>>> ERROR: org.apache.hadoop.hbase.client.RetriesExhaustedException:
> Failed
> >>>> after attempts=3D7, exceptions:
> >>>> Wed Sep 11 14:21:05 CEST 2013,
> >>>> org.apache.hadoop.hbase.client.HTable$3@adc4d8f, java.io.IOException=
:
> >>> Call
> >>>> to pc17.pool.ifis.uni-luebeck.de/141.83.150.97:60020 failed on local
> >>>> exception: java.io.EOFException
> >>>> Wed Sep 11 14:21:06 CEST 2013,
> >>>> org.apache.hadoop.hbase.client.HTable$3@adc4d8f,
> >>>> java.net.ConnectException:
> >>>> Connection refused
> >>>> Wed Sep 11 14:21:07 CEST 2013,
> >>>> org.apache.hadoop.hbase.client.HTable$3@adc4d8f,
> >>>> org.apache.hadoop.hbase.ipc.HBaseClient$FailedServerException: This
> >>> server
> >>>> is in the failed servers list:
> >>>> pc17.pool.ifis.uni-luebeck.de/141.83.150.97:60020
> >>>> Wed Sep 11 14:21:08 CEST 2013,
> >>>> org.apache.hadoop.hbase.client.HTable$3@adc4d8f,
> >>>> java.net.ConnectException:
> >>>> Connection refused
> >>>> Wed Sep 11 14:21:10 CEST 2013,
> >>>> org.apache.hadoop.hbase.client.HTable$3@adc4d8f,
> >>>> java.net.ConnectException:
> >>>> Connection refused
> >>>> Wed Sep 11 14:21:12 CEST 2013,
> >>>> org.apache.hadoop.hbase.client.HTable$3@adc4d8f,
> >>>> java.net.ConnectException:
> >>>> Connection refused
> >>>> Wed Sep 11 14:21:16 CEST 2013,
> >>>> org.apache.hadoop.hbase.client.HTable$3@adc4d8f,
> >>>> java.net.ConnectException:
> >>>> Connection refused
> >>>>
> >>>>
> >>>>
> >>>> 2013/9/11 Ted Yu <yuzhihong@gmail.com>
> >>>>
> >>>>> Take a look at
> >>>>>
> >>>>
> >>>
> http://hbase.apache.org/0.94/apidocs/org/apache/hadoop/hbase/filter/Colum=
nPaginationFilter.html
> >>>>>
> >>>>> Cheers
> >>>>>
> >>>>> On Sep 11, 2013, at 4:42 AM, John <johnnyenglish739@gmail.com>
> wrote:
> >>>>>
> >>>>>> Hi,
> >>>>>>
> >>>>>> thanks for your fast answer! with size becoming too big I mean I
> >>> have
> >>>> one
> >>>>>> row with thousands of columns. For example:
> >>>>>>
> >>>>>> myrowkey1 -> column1, column2, column3 ... columnN
> >>>>>>
> >>>>>> What do you mean with "change the batch size"? I try to create a
> >>> little
> >>>>>> java test code to reproduce the problem. It will take a moment
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> 2013/9/11 Jean-Marc Spaggiari <jean-marc@spaggiari.org>
> >>>>>>
> >>>>>>> Hi John,
> >>>>>>>
> >>>>>>> Just to be sure. What is " the size become too big"? The size of =
a
> >>>>> single
> >>>>>>> column within this row? Or the number of columns?
> >>>>>>>
> >>>>>>> If it's the number of columns, you can change the batch size to g=
et
> >>>> less
> >>>>>>> columns in a single call? Can you share the relevant piece of cod=
e
> >>>> doing
> >>>>>>> the call?
> >>>>>>>
> >>>>>>> JM
> >>>>>>>
> >>>>>>>
> >>>>>>> 2013/9/11 John <johnnyenglish739@gmail.com>
> >>>>>>>
> >>>>>>>> Hi,
> >>>>>>>>
> >>>>>>>> I store a lot of columns for one row key and if the size become =
to
> >>>> big
> >>>>>>> the
> >>>>>>>> relevant Region Server crashs if I try to get or scan the row. F=
or
> >>>>>>> example
> >>>>>>>> if I try to get the relevant row I got this error:
> >>>>>>>>
> >>>>>>>> 2013-09-11 12:46:43,696 WARN org.apache.hadoop.ipc.HBaseServer:
> >>>>>>>> (operationTooLarge): {"processingtimems":3091,"client":"
> >>>>>>> 192.168.0.34:52488
> >>>>>>>> ","ti$
> >>>>>>>>
> >>>>>>>> If I try to load the relevant row via Apache Pig and the
> >>> HBaseStorage
> >>>>>>>> Loader (use the scan operation) I got this message and after tha=
t
> >>> the
> >>>>>>>> Region Servers crashs:
> >>>>>>>>
> >>>>>>>> 2013-09-11 10:30:23,542 WARN org.apache.hadoop.ipc.HBaseServer:
> >>>>>>>> (responseTooLarge):
> >>>>>>>> {"processingtimems":1851,"call":"next(-588368116791418695,
> >>>>>>>> 1), rpc version=3D1, client version=3D29,$
> >>>>>>>>
> >>>>>>>> I'm using Cloudera 4.4.0 with 0.94.6-cdh4.4.0
> >>>>>>>>
> >>>>>>>> Any clues?
> >>>>>>>>
> >>>>>>>> regards
> >>>>>>>
> >>>>>
> >>>>
> >>>
> >>
> >>
>
> The opinions expressed here are mine, while they may reflect a cognitive
> thought, that is purely accidental.
> Use at your own risk.
> Michael Segel
> michael_segel (AT) hotmail.com
>
>
>
>
>
>


--=20
Kevin O'Dell
Systems Engineer, Cloudera

--001a11c3327a62b20c04e61ce7f5--