hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Doug Meil <doug.m...@explorysmedical.com>
Subject Re: Memory Requirements
Date Sat, 10 Mar 2012 13:42:24 GMT
Hi there-

Here are the recommendations from the HBase RefGuide:

http://hbase.apache.org/book.html#perf.os

... and they are consistently with what the book says (recommends 64-bit
OS and more memory).

Also, keep this in mind...

http://hbase.apache.org/book.html#arch.overview


... the standalone deployment should be considered a development config
only.



On 3/9/12 10:20 PM, "Bing Li" <lblabs@gmail.com> wrote:

>Dear all,
>
>Today I ran HBase in the pseudo-distributed mode. Since I need to save the
>meta information about Web pages crawled from more than 10,000 Web sites
>simultaneously and periodically. I believe the data size is large. After
>running for about 30 minutes, it got the following exceptions.
>
>     [java] ...............2012-3-10 10:09:04
>org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementatio
>n
>processBatchCallback
>     [java] Warning: Failed all from
>region=PlayTable,,1331343367325.2bf6bf580051da504c347c19eb08f9d5.,
>hostname=greatfreeinpain, port=60020
>     [java] java.util.concurrent.ExecutionException:
>java.lang.RuntimeException: java.lang.OutOfMemoryError: Java heap space
>     [java] at
>java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
>     [java] at java.util.concurrent.FutureTask.get(FutureTask.java:83)
>     [java] at
>org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementatio
>n.processBatchCallback(HConnectionManager.java:1557)
>     [java] at
>org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementatio
>n.processBatch(HConnectionManager.java:1409)
>     [java] at
>org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:900)
>     [java] at 
>org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:777)
>     [java] at org.apache.hadoop.hbase.client.HTable.put(HTable.java:760)
>     [java] at
>org.apache.hadoop.hbase.client.HTablePool$PooledHTable.put(HTablePool.java
>:402)
>     [java] at
>com.greatfree.hbase.AuthorityPersister.InsertInitialPlayRecord(AuthorityPe
>rsister.java:128)
>     [java] at
>com.greatfree.hbase.RandomPersistDispatcher.run(RandomPersistDispatcher.ja
>va:92)
>     [java] at
>java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.
>java:886)
>     [java] at
>java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java
>:908)
>     [java] at java.lang.Thread.run(Thread.java:662)
>     [java] Caused by: java.lang.RuntimeException:
>java.lang.OutOfMemoryError: Java heap space
>     [java] at
>org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementatio
>n.getRegionServerWithoutRetries(HConnectionManager.java:1371)
>     [java] at
>org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementatio
>n$3.call(HConnectionManager.java:1383)
>     [java] at
>org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementatio
>n$3.call(HConnectionManager.java:1381)
>     [java] at
>java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>     [java] at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>     [java] ... 3 more
>     [java] Caused by: java.lang.OutOfMemoryError: Java heap space
>     [java] at java.util.Arrays.copyOf(Arrays.java:2786)
>     [java] at
>java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94)
>     [java] at java.io.DataOutputStream.write(DataOutputStream.java:90)
>     [java] at org.apache.hadoop.hbase.client.Put.write(Put.java:422)
>     [java] at
>org.apache.hadoop.hbase.io.HbaseObjectWritable.writeObject(HbaseObjectWrit
>able.java:464)
>     [java] at org.apache.hadoop.hbase.client.Action.write(Action.java:94)
>     [java] at
>org.apache.hadoop.hbase.io.HbaseObjectWritable.writeObject(HbaseObjectWrit
>able.java:464)
>     [java] at
>org.apache.hadoop.hbase.client.MultiAction.write(MultiAction.java:102)
>     [java] at
>org.apache.hadoop.hbase.io.HbaseObjectWritable.writeObject(HbaseObjectWrit
>able.java:464)
>     [java] at
>org.apache.hadoop.hbase.ipc.Invocation.write(Invocation.java:138)
>     [java] at
>org.apache.hadoop.hbase.ipc.HBaseClient$Connection.sendParam(HBaseClient.j
>ava:537)
>     [java] at
>org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:879)
>     [java] at
>org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEn
>gine.java:150)
>     [java] at $Proxy6.multi(Unknown Source)
>     [java] at
>org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementatio
>n$3$1.call(HConnectionManager.java:1386)
>     [java] at
>org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementatio
>n$3$1.call(HConnectionManager.java:1384)
>     [java] at
>org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementatio
>n.getRegionServerWithoutRetries(HConnectionManager.java:1365)
>     [java] ... 7 more
>
>The machine running the HBase has 1G memory only over Ubuntu 10. According
>to the book, HBase the Definitive Guide, 64-bit OS and 4GB are required. I
>need to upgrade my system?
>
>I change the parameter, HBASE_HEAPSIZE, from 1000 to 2048. The exceptions
>still occurred. When the value was changed to 3072, it got the following
>errors when I ran start-hbase.sh.
>
>    Could not create the Java virtual machine.
>    Could not create the Java virtual machine.
>    starting master, logging to
>/opt/hbase-0.92.0/bin/../logs/hbase-libing-master-greatfreeinpain.out
>    Error occurred during initialization of VM
>    Could not reserve enough space for object heap
>    Could not create the Java virtual machine.
>    localhost: starting regionserver, logging to
>/opt/hbase-0.92.0/bin/../logs/hbase-libing-regionserver-greatfreeinpain.ou
>t
>    localhost: Error occurred during initialization of VM
>    localhost: Could not reserve enough space for object heap
>    localhost: Could not create the Java virtual machine.
>
>It seems that I need to increase the memory size? Do I need to install a
>64-bit Ubuntu Linux?
>
>Thanks so much for your help!
>
>Best regards,
>Bing



Mime
View raw message