hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tom Nichols <tmnich...@gmail.com>
Subject MapReduce error
Date Fri, 08 May 2009 18:54:21 GMT
(Apologies if this is more appropriate for the Hadoop user list, but
to be fair, my input and output are HBase tables...)

I'm trying to run a MR on a 0.19.2 hbase cluster and I'm getting the
following error:
org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to
contact region server 172.16.10.140:60020 for region
Platts_Megawatt,,1241729216060, row '', but failed after 10 attempts.
Exceptions:
java.lang.NullPointerException
java.lang.NullPointerException
java.lang.NullPointerException
java.lang.NullPointerException
java.lang.NullPointerException
java.lang.NullPointerException
java.lang.NullPointerException
java.lang.NullPointerException
java.lang.NullPointerException
java.lang.NullPointerException

        at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:858)
        at org.apache.hadoop.hbase.client.HTable$ClientScanner.nextScanner(HTable.java:1594)
...

RegionServer log: -----------------------------------------

2009-05-08 14:05:36,464 ERROR
org.apache.hadoop.hbase.regionserver.HRegionServer:
org.apache.hadoop.hbase.UnknownScannerException: Name: -1
2009-05-08 14:05:36,466 INFO org.apache.hadoop.ipc.HBaseServer: IPC
Server handler 9 on 60020, call next(-1, 30) from 172.16.10.95:60969:
error: org.apache.hadoop.hbase.UnknownScannerException: Name: -1
org.apache.hadoop.hbase.UnknownScannerException: Name: -1
        at org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1574)
        at sun.reflect.GeneratedMethodAccessor26.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
        at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:912)

-----------------------------------------

It happens that for this particular table, the datanode is on the same
machine as the hbase master/namenode server.  Now, I don't see any
datanode errors for that exact same timeframe, but there are a couple
other errors around that time:

DataNode.log: -----------------------------------------------

2009-05-08 14:30:52,605 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src:
/172.16.10.140:50010, dest: /172.16.10.140:60214, bytes: 1915392, op:
HDFS_READ, cliID: DFSClient_-221953257, srvID:
DS-526908905-172.16.10.140-50010-1229537939612, blockid:
blk_-8408339440397934538_221228
2009-05-08 14:30:52,605 WARN
org.apache.hadoop.hdfs.server.datanode.DataNode:
DatanodeRegistration(172.16.10.140:50010,
storageID=DS-526908905-172.16.10.140-50010-1229537939612,
infoPort=50075, ipcPort=50020):Got exception while serving
blk_-8408339440397934538_221228 to /172.16.10.140:
java.net.SocketTimeoutException: 480000 millis timeout while waiting
for channel to be ready for write. ch :
java.nio.channels.SocketChannel[connected local=/172.16.10.140:50010
remote=/172.16.10.140:60214]
        at org.apache.hadoop.net.SocketIOWithTimeout.waitForIO(SocketIOWithTimeout.java:185)
        at org.apache.hadoop.net.SocketOutputStream.waitForWritable(SocketOutputStream.java:159)
        at org.apache.hadoop.net.SocketOutputStream.transferToFully(SocketOutputStream.java:198)
        at org.apache.hadoop.hdfs.server.datanode.BlockSender.sendChunks(BlockSender.java:293)
        at org.apache.hadoop.hdfs.server.datanode.BlockSender.sendBlock(BlockSender.java:387)
        at org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.java:179)
        at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:94)
        at java.lang.Thread.run(Thread.java:619)

2009-05-08 14:30:52,605 ERROR
org.apache.hadoop.hdfs.server.datanode.DataNode:
DatanodeRegistration(172.16.10.140:50010,
storageID=DS-526908905-172.16.10.140-50010-1229537939612,
infoPort=50075, ipcPort=50020):DataXceiver
java.net.SocketTimeoutException: 480000 millis timeout while waiting
for channel to be ready for write. ch :
java.nio.channels.SocketChannel[connected local=/172.16.10.140:50010
remote=/172.16.10.140:60214]
        at org.apache.hadoop.net.SocketIOWithTimeout.waitForIO(SocketIOWithTimeout.java:185)
        at org.apache.hadoop.net.SocketOutputStream.waitForWritable(SocketOutputStream.java:159)
        at org.apache.hadoop.net.SocketOutputStream.transferToFully(SocketOutputStream.java:198)
        at org.apache.hadoop.hdfs.server.datanode.BlockSender.sendChunks(BlockSender.java:293)
        at org.apache.hadoop.hdfs.server.datanode.BlockSender.sendBlock(BlockSender.java:387)
        at org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.java:179)
        at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:94)
        at java.lang.Thread.run(Thread.java:619)

-------------------------------------------------

Any ideas?

Thanks in advance.
-Tom

Mime
View raw message