hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 吴国泉wgq <wgq...@qunar.com>
Subject Re: java.lang.OutOfMemoryError when count hbase table
Date Wed, 19 Oct 2016 12:16:58 GMT
hi biodata:

        you can try “ scan.setbatch()”  or other filter to limit the number of column

       This is because: There is a very large row in your table,when you try to retrieve it,
OOM will happen.

        As I can see, There is no other method to solve this problem. Default,every read operation,
hbase will retrieve an entire row if you don’t limit  the col。

        Don’t insert very large row.  you can delete it .  But don’t forget  trigger the
major compaction after you delete the row.

吴国泉   wgq.wu
Post: DBA  Hbase
Email: wgq.wu@qunar.com<http://qunar.com>
Tel: 13051697997
Adr: 中国电子大厦17层

在 2016年10月19日,下午7:43,big data <bigdatabase@outlook.com<mailto:bigdatabase@outlook.com>>

I've adjusted the jvm xmx in hbase-env.xml, now in hbase shell, count
runs well.

But java client still crashes because :

Caused by: com.google.protobuf.InvalidProtocolBufferException: Protocol
message was too large.  May be malicious.  Use
CodedInputStream.setSizeLimit() to increase the size limit.

I've browsed apache's jira, but still don't know how to call
CodedInputStream.setSizeLimit()  in client side.

在 16/10/19 下午6:38, Jean-Marc Spaggiari 写道:
Interesting. Can you bump the client heap size? How much do you have for
the client?


2016-10-19 3:50 GMT-04:00 big data <bigdatabase@outlook.com<mailto:bigdatabase@outlook.com>>:

Dear all,

I've a hbase table, one row has a huge keyvalue, about 100M size.

When I execute count table in hbase shell, hbase crash to bash, and
display error like this:

hbase(main):005:0> count 'table', CACHE=>10000
java.lang.OutOfMemoryError: Java heap space
Dumping heap to java_pid17735.hprof ...
Unable to create java_pid17735.hprof: 权限不够
# java.lang.OutOfMemoryError: Java heap space
# -XX:OnOutOfMemoryError="kill -9 %p"
#   Executing /bin/sh -c "kill -9 17735"...

Meanwhile, my java client execute get operation from this table, it
hangs for a long time.

How can I adjust some parameters to support huge keyvalue?


  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message