hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From NNever <nnever...@gmail.com>
Subject Re: Region autoSplit when not reach 'hbase.hregion.max.filesize' ?
Date Thu, 07 Jun 2012 02:33:33 GMT
We use hbase 0.94.0, running on a single mechine.
The hbase-site.xml:

<configuration>
> <property>
> <name>hbase.rootdir</name>
> <value>hdfs://xxxxx/hbase</value>
>  </property>
> <property>
> <name>hbase.cluster.distributed</name>
>  <value>true</value>
> </property>
> <property>
>  <name>hbase.zookeeper.quorum</name>
> <value>xxxxx</value>
> </property>
>  <property>
> <name>hbase.zookeeper.property.dataDir</name>
> <value>/mybk/zookeeper</value>
>  </property>
> <property>
> <name>hbase.hregion.max.filesize</name>
>  <value>107374182400</value>
> </property>
> <property>
>  <name>zookeeper.session.timeout</name>
> <value>60000</value>
> </property>
>  <property>
> <name>hbase.regionserver.handler.count</name>
> <value>4000</value>
>  </property>
> <property>
> <name>hbase.client.write.buffer</name>
>  <value>1048576</value>
> </property>
> <property>
>  <name>hbase.client.scanner.caching</name>
> <value>10</value>
> </property>
> </configuration>




2012/6/7 NNever <nneverwei@gmail.com>

> The rowkey is just like an UUID. no order.  And there is an coprocessor
> trigger datas to another 2 index-tables when doing put...
>
> Thanks. yours
> NN
>
>
> 2012/6/7 Michael Segel <michael_segel@hotmail.com>
>
>> Just out of curiosity, describe the data?
>> Sorted?
>> The more we know, the easier it is to help... Also, can you recheck your
>> math ?
>>
>>
>> Sent from my iPhone
>>
>> On Jun 6, 2012, at 6:17 PM, "NNever" <nneverwei@gmail.com> wrote:
>>
>> > It comes again. I truncate the table, and put about 10million datas
>> into it
>> > last night.
>> > The table auto-split to 4, each has about 3Gb storefileUncompressedSize.
>> >
>> > I grep the log and out but nothing about the split.
>> >
>> > the logs are as below:
>> > 2012-06-06 19:31:15,402 WARN org.apache.hadoop.ipc.HBaseServer:
>> > (responseTooSlow):
>> > {"processingtimems":10296,"call":"next(1511657428305700194, 1), rpc
>> > version=1, client version=29, methodsFingerPrint=-1508511443","client":"
>> > 192.168.1.145:46456
>> >
>> ","starttimems":1338982265104,"queuetimems":0,"class":"HRegionServer","responsesize":6,"method":"next"}
>> > 2012-06-06 19:31:15,606 WARN org.apache.hadoop.ipc.HBaseServer:
>> > (responseTooSlow):
>> > {"processingtimems":10842,"call":"next(-2954106234340837837, 1), rpc
>> > version=1, client version=29, methodsFingerPrint=-1508511443","client":"
>> > 192.168.1.145:46456
>> >
>> ","starttimems":1338982264763,"queuetimems":1,"class":"HRegionServer","responsesize":6,"method":"next"}
>> > 2012-06-06 19:31:29,795 WARN org.apache.hadoop.ipc.HBaseServer:
>> > (responseTooSlow):
>> > {"processingtimems":10668,"call":"next(2455689470981850756, 1), rpc
>> > version=1, client version=29, methodsFingerPrint=-1508511443","client":"
>> > 192.168.1.145:46456
>> >
>> ","starttimems":1338982279126,"queuetimems":0,"class":"HRegionServer","responsesize":6,"method":"next"}
>> > 2012-06-06 20:24:54,157 WARN org.apache.hadoop.ipc.HBaseServer:
>> > (responseTooSlow):
>> >
>> {"processingtimems":2920400,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@6b39de40
>> ),
>> > rpc version=1, client version=29,
>> methodsFingerPrint=-1508511443","client":"
>> > 192.168.1.145:46456
>> >
>> ","starttimems":1338982573756,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"}
>> > 2012-06-06 20:24:54,251 WARN org.apache.hadoop.ipc.HBaseServer: IPC
>> Server
>> > Responder, call
>> multi(org.apache.hadoop.hbase.client.MultiAction@6b39de40),
>> > rpc version=1, client version=29, methodsFingerPrint=-1508511443 from
>> > 192.168.1.145:46456: output error
>> > 2012-06-06 20:24:54,294 WARN org.apache.hadoop.ipc.HBaseServer: IPC
>> Server
>> > handler 2159 on 60020 caught a ClosedChannelException, this means that
>> the
>> > server was processing a request but the client went away. The error
>> message
>> > was: null
>> > 2012-06-06 20:25:00,868 WARN org.apache.hadoop.ipc.HBaseServer:
>> > (responseTooSlow):
>> {"processingtimems":2927114,"call":"multi(org.apache.hado
>> >
>> > You can see on 19:31:29 the log stop for about 1 hour, It may doing
>> split
>> > there.(this is the regionserver.log)
>> > And in the regionServer.out I cannot find any information about split,
>> Only
>> > lots of 'org.apache.hadoop.hbase.NotServingRegionException' when split.
>> No
>> > log about start doing split and why do split.
>> >
>> > logs are too large to upload somewhere.
>> >
>> > I'll dig into it....It really confuse me...
>> >
>> > Thanks, yours
>> > NN
>> >
>> >
>> > 2012/6/6 NNever <nneverwei@gmail.com>
>> >
>> >> I'll. I changed the log level.
>> >> Putting datas and waiting for the strange split now   :).....
>> >>
>> >> Yours,
>> >> NN
>> >>
>> >> 2012/6/6 dong.yajun <dongtalk@gmail.com>
>> >>
>> >> Hi NNever
>> >>>
>> >>> If you find any issues, please let us known, thanks.
>> >>>
>> >>>
>> >>> On Wed, Jun 6, 2012 at 5:09 PM, NNever <nneverwei@gmail.com> wrote:
>> >>>
>> >>>> I'm sorry, the log4j now is WARN, not INFO
>> >>>>
>> >>>> 2012/6/6 NNever <nneverwei@gmail.com>
>> >>>>
>> >>>>> We currently run in INFO mode.
>> >>>>> It actully did the split, but I cannot find any logs about this
>> split.
>> >>>>> I will change the log4j to DEBUG, if got any log valuable, I
will
>> >>> paste
>> >>>>> here...
>> >>>>>
>> >>>>> Thanks Ram,
>> >>>>> NN
>> >>>>>
>> >>>>> 2012/6/6 Ramkrishna.S.Vasudevan <ramkrishna.vasudevan@huawei.com>
>> >>>>>
>> >>>>> You have any logs corresponding to this?
>> >>>>>>
>> >>>>>> Regards
>> >>>>>> Ram
>> >>>>>>
>> >>>>>>> -----Original Message-----
>> >>>>>>> From: NNever [mailto:nneverwei@gmail.com]
>> >>>>>>> Sent: Wednesday, June 06, 2012 2:12 PM
>> >>>>>>> To: user@hbase.apache.org
>> >>>>>>> Subject: Region autoSplit when not reach
>> >>> 'hbase.hregion.max.filesize'
>> >>>> ?
>> >>>>>>>
>> >>>>>>> The 'hbase.hregion.max.filesize' are set to 100G (The
recommed
>> >>> value
>> >>>> to
>> >>>>>>> act
>> >>>>>>> as auto-split turn off). And there is a table, we keep
put datas
>> >>> into
>> >>>>>>> it.
>> >>>>>>> When the storefileUncompressedSizeMB reached about 1Gb,
the region
>> >>>> auto
>> >>>>>>> splite to 2.
>> >>>>>>> I don't know how it happened? 1G is far more less than
>> >>> max.filesize-
>> >>>>>>> 100G.
>> >>>>>>> So if there is any possible scenery that will
>> >>>>>>> ignore hbase.hregion.max.filesize and do split?
>> >>>>>>>
>> >>>>>>> How can I totally shutdown the autoSplit?
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> -----------------
>> >>>>>>> Best regards,
>> >>>>>>> NN
>> >>>>>>
>> >>>>>>
>> >>>>>
>> >>>>
>> >>>
>> >>>
>> >>>
>> >>> --
>> >>> *Ric Dong *
>> >>> Newegg Ecommerce, MIS department
>> >>>
>> >>
>> >>
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message