hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jean-Daniel Cryans <jdcry...@apache.org>
Subject Re: Java Client hangs after around 2 min
Date Wed, 07 Jul 2010 18:01:43 GMT
After doing some google researches, it seems that your machine is
running out of memory when the RawLocalFilesystem forks in order to
create and chmod a new file (since that creates a new process with as
much mem as the original process).

It was discussed a lot in
https://issues.apache.org/jira/browse/HADOOP-5059 since this is in
Hadoop's land. Get more RAM, or more swap.

J-D

On Wed, Jul 7, 2010 at 10:53 AM, manua <agarwal.manu@gmail.com> wrote:
>
> Hi,
>
> I am using Hbase version 0.20.5.
>
> Also, I am running both the server on client on the same box. Please find
> below the other log snippets,
>
> Log File : hbase-manu-regionserver-domU-12-31-39-06-62-43.log
>
> Wed Jul  7 13:42:17 EDT 2010 Starting regionserver on domU-12-31-39-06-62-43
> ulimit -n 1024
> 2010-07-07 13:42:18,222 WARN
> org.apache.hadoop.hbase.regionserver.HRegionServer: Not starting a distinct
> region server because hbase.cluster.distributed is false
>
> Files hbase-manu-zookeeper-domU-12-31-39-06-62-43.out and
> hbase-manu-regionserver-domU-12-31-39-06-62-43.out are blank.
>
> File:hbase-manu-zookeeper-domU-12-31-39-06-62-43.log
>
> 2010-07-07 13:42:15,799 INFO org.apache.zookeeper.server.ZooKeeperServer:
> Server
> environment:java.library.path=/home/manu/hbase/hbase-0.20.5/bin/../lib/native/Linux-i386-32
> 2010-07-07 13:42:15,799 INFO org.apache.zookeeper.server.ZooKeeperServer:
> Server environment:java.io.tmpdir=/tmp
> 2010-07-07 13:42:15,800 INFO org.apache.zookeeper.server.ZooKeeperServer:
> Server environment:java.compiler=<NA>
> 2010-07-07 13:42:15,800 INFO org.apache.zookeeper.server.ZooKeeperServer:
> Server environment:os.name=Linux
> 2010-07-07 13:42:15,800 INFO org.apache.zookeeper.server.ZooKeeperServer:
> Server environment:os.arch=i386
> 2010-07-07 13:42:15,800 INFO org.apache.zookeeper.server.ZooKeeperServer:
> Server environment:os.version=2.6.21.7-2.ec2.v1.2.fc8xen
> 2010-07-07 13:42:15,800 INFO org.apache.zookeeper.server.ZooKeeperServer:
> Server environment:user.name=manu
> 2010-07-07 13:42:15,800 INFO org.apache.zookeeper.server.ZooKeeperServer:
> Server environment:user.home=/home/manu
> 2010-07-07 13:42:15,800 INFO org.apache.zookeeper.server.ZooKeeperServer:
> Server environment:user.dir=/home/manu/hbase/hbase-0.20.5
> 2010-07-07 13:42:15,843 INFO
> org.apache.zookeeper.server.persistence.FileSnap: Reading snapshot
> /home/_hbase_/hbase-manu/zookeeper/version-2/snapshot.170
> 2010-07-07 13:42:15,879 INFO
> org.apache.zookeeper.server.persistence.FileTxnSnapLog: Snapshotting: 17f
> 2010-07-07 13:42:17,090 INFO org.apache.zookeeper.server.NIOServerCnxn:
> Connected to /127.0.0.1:36319 lastZxid 0
> 2010-07-07 13:42:17,091 INFO org.apache.zookeeper.server.NIOServerCnxn:
> Creating new session 0x129ae00344b0000
> 2010-07-07 13:42:17,169 INFO org.apache.zookeeper.server.NIOServerCnxn:
> Finished init of 0x129ae00344b0000 valid:true
> 2010-07-07 13:42:17,258 INFO org.apache.zookeeper.server.NIOServerCnxn:
> Connected to /127.0.0.1:36320 lastZxid 0
> 2010-07-07 13:42:17,258 INFO org.apache.zookeeper.server.NIOServerCnxn:
> Creating new session 0x129ae00344b0001
> 2010-07-07 13:42:17,263 INFO org.apache.zookeeper.server.NIOServerCnxn:
> Finished init of 0x129ae00344b0001 valid:true
> 2010-07-07 13:42:18,437 WARN
> org.apache.zookeeper.server.PrepRequestProcessor: Got exception when
> processing sessionid:0x129ae00344b0000 type:create cxid:0xd
> zxid:0xfffffffffffffffe txntype:unknown n/a
> org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode =
> NodeExists
>        at
> org.apache.zookeeper.server.PrepRequestProcessor.pRequest(PrepRequestProcessor.java:245)
>        at
> org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:114)
> 2010-07-07 13:44:02,841 INFO org.apache.zookeeper.server.NIOServerCnxn:
> Connected to /127.0.0.1:36325 lastZxid 0
> 2010-07-07 13:44:02,841 INFO org.apache.zookeeper.server.NIOServerCnxn:
> Creating new session 0x129ae00344b0002
> 2010-07-07 13:44:02,850 INFO org.apache.zookeeper.server.NIOServerCnxn:
> Finished init of 0x129ae00344b0002 valid:true
> 2010-07-07 13:46:05,328 INFO
> org.apache.zookeeper.server.PrepRequestProcessor: Processed session
> termination request for id: 0x129ae00344b0001
> 2010-07-07 13:46:05,344 INFO org.apache.zookeeper.server.NIOServerCnxn:
> closing session:0x129ae00344b0001 NIOServerCnxn:
> java.nio.channels.SocketChannel[connected local=/127.0.0.1:2181
> remote=/127.0.0.1:36320]
>
> File : hbase-manu-master-domU-12-31-39-06-62-43.out
>
> java.io.IOException: Cannot run program "chmod": java.io.IOException:
> error=12, Cannot allocate memory
>        at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)
>        at org.apache.hadoop.util.Shell.runCommand(Shell.java:149)
>        at org.apache.hadoop.util.Shell.run(Shell.java:134)
>        at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:286)
>        at org.apache.hadoop.util.Shell.execCommand(Shell.java:354)
>        at org.apache.hadoop.util.Shell.execCommand(Shell.java:337)
>        at
> org.apache.hadoop.fs.RawLocalFileSystem.execCommand(RawLocalFileSystem.java:481)
>        at
> org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:473)
>        at
> org.apache.hadoop.fs.FilterFileSystem.setPermission(FilterFileSystem.java:280)
>        at
> org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:372)
>        at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:484)
>        at
> org.apache.hadoop.io.SequenceFile$Writer.<init>(SequenceFile.java:843)
>        at
> org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:393)
>        at
> org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:284)
>        at org.apache.hadoop.hbase.regionserver.HLog$1.run(HLog.java:1055)
>        at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>        at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>        at java.lang.Thread.run(Thread.java:619)
> Caused by: java.io.IOException: java.io.IOException: error=12, Cannot
> allocate memory
>        at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
>        at java.lang.ProcessImpl.start(ProcessImpl.java:65)
>        at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)
>        ... 17 more
>
> Please let me know your inputs.
>
> Thanks,
> Manu
>
> --
> View this message in context: http://old.nabble.com/Java-Client-hangs-after-around-2-min-tp29097955p29099055.html
> Sent from the HBase User mailing list archive at Nabble.com.
>
>

Mime
View raw message