hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: Never ending "Doing distributed log split" task.,
Date Thu, 29 Aug 2013 16:43:53 GMT
What is your HBase / Hadoop version ?

Can you check namenode log looking for lines related to
hdfs://node3:9000/hbase/.logs/node1,60020,1377789460683-
splitting/node1%2C60020%2C1377789460683.1377789462024 ?

Thanks


On Thu, Aug 29, 2013 at 9:03 AM, Jean-Marc Spaggiari <
jean-marc@spaggiari.org> wrote:

> I have restart my cluster and I'm now waiting for this task to end:
>
> Doing distributed log split in
> [hdfs://node3:9000/hbase/.logs/node1,60020,1377789460683-splitting]
>
> It's running fir now 30 minutes. There was nothing running on the cluster.
> No reads, no writes, nothing, for days...
>
> I got that on the logs:
>
> 2013-08-29 11:36:10,862 WARN
> org.apache.hadoop.hbase.regionserver.SplitLogWorker: log splitting of
>
> hdfs://node3:9000/hbase/.logs/node1,60020,1377789460683-splitting/node1%2C60020%2C1377789460683.1377789462024
> interrupted, resigning
> java.io.InterruptedIOException
>     at
>
> org.apache.hadoop.hbase.util.FSHDFSUtils.recoverDFSFileLease(FSHDFSUtils.java:136)
>     at
>
> org.apache.hadoop.hbase.util.FSHDFSUtils.recoverFileLease(FSHDFSUtils.java:54)
>     at
>
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.getReader(HLogSplitter.java:780)
>     at
>
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLogFile(HLogSplitter.java:414)
>     at
>
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLogFile(HLogSplitter.java:381)
>     at
>
> org.apache.hadoop.hbase.regionserver.SplitLogWorker$1.exec(SplitLogWorker.java:112)
>     at
>
> org.apache.hadoop.hbase.regionserver.SplitLogWorker.grabTask(SplitLogWorker.java:280)
>     at
>
> org.apache.hadoop.hbase.regionserver.SplitLogWorker.taskLoop(SplitLogWorker.java:211)
>     at
>
> org.apache.hadoop.hbase.regionserver.SplitLogWorker.run(SplitLogWorker.java:179)
>     at java.lang.Thread.run(Thread.java:722)
> Caused by: java.lang.InterruptedException: sleep interrupted
>     at java.lang.Thread.sleep(Native Method)
>     at
>
> org.apache.hadoop.hbase.util.FSHDFSUtils.recoverDFSFileLease(FSHDFSUtils.java:118)
>     ... 9 more
> 2013-08-29 11:36:10,950 WARN
> org.apache.hadoop.hbase.regionserver.SplitLogWorker: Interrupted while
> trying to assert ownership of
>
> /hbase/splitlog/hdfs%3A%2F%2Fnode3%3A9000%2Fhbase%2F.logs%2Fnode1%2C60020%2C1377789460683-splitting%2Fnode1%252C60020%252C1377789460683.1377789462024
> java.lang.InterruptedException
>     at java.lang.Object.wait(Native Method)
>     at java.lang.Object.wait(Object.java:503)
>     at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1253)
>     at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1129)
>     at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1160)
>     at
>
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.setData(RecoverableZooKeeper.java:361)
>     at
>
> org.apache.hadoop.hbase.regionserver.SplitLogWorker.attemptToOwnTask(SplitLogWorker.java:346)
>     at
>
> org.apache.hadoop.hbase.regionserver.SplitLogWorker.grabTask(SplitLogWorker.java:264)
>     at
>
> org.apache.hadoop.hbase.regionserver.SplitLogWorker.taskLoop(SplitLogWorker.java:211)
>     at
>
> org.apache.hadoop.hbase.regionserver.SplitLogWorker.run(SplitLogWorker.java:179)
>     at java.lang.Thread.run(Thread.java:722)
>
>
> I'm not 100% what is causing that. I have restarted it and still getting
> the same result.
>
> Any hint?
>
> Thanks,
>
> JM
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message