hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Snow <marksnow...@yahoo.com>
Subject failure after importing 42million rows
Date Fri, 25 Jul 2008 18:02:06 GMT
I'm running a hbase data import on 0.1.3. After 42million rows, the import fails with an RPC
timeout exception. I've tried twice- once on a 2 node cluster and once on a 10 node cluster
(ec2 with the same configuration) and it failed both times in the same spot, somewhere between
42 and 43 million rows. Where should I look to debug this?

>From the hbase shell, I can query the table and see the rows have been inserted, but when
I do a 'hadoop dfs -ls' I don't see the /hbase dir I specified, so I'm suspicious it's not
storing the data into dfs, and unsure where it is storing this data.

hbase root last log entries
2008-07-25 13:46:10,196 INFO org.apache.hadoop.hbase.HMaster: HMaster.rootScanner scanning
meta region {regionname: -ROOT-,,0, startKey: <>, server: 10.254.171.22:60020}
2008-07-25 13:46:10,213 DEBUG org.apache.hadoop.hbase.HMaster: HMaster.rootScanner regioninfo:
{regionname: .META.,,1, startKey: <>, endKey: <>, encodedName: 1028785192, tableDesc:
{name: .META., families: {info:={name: info, max versions: 1, compression: NONE, in memory:
false, max length: 2147483647, bloom filter: none}}}}, server: 10.254.243.146:60020, startCode:
1216947114706
2008-07-25 13:46:10,214 INFO org.apache.hadoop.hbase.HMaster: HMaster.rootScanner scan of
meta region {regionname: -ROOT-,,0, startKey: <>, server: 10.254.171.22:60020} complete

last log entries from one of the region servers
2008-07-25 13:44:28,190 DEBUG org.apache.hadoop.hbase.HRegion: Started memcache flush for
region relations,,1216948402123. Current region memcache size 0.0
2008-07-25 13:44:28,190 DEBUG org.apache.hadoop.hbase.HRegion: Finished memcache flush for
region relations,,1216948402123 in 0ms, sequence id=32
2008-07-25 13:44:28,190 DEBUG org.apache.hadoop.hbase.HRegionServer: Compaction requested
for region: relations,,1216948402123
2008-07-25 13:44:28,190 INFO org.apache.hadoop.hbase.HRegion: checking compaction on region
relations,,1216948402123
2008-07-25 13:44:28,192 INFO org.apache.hadoop.hbase.HRegion: checking compaction completed
on region relations,,1216948402123; status: false; 0sec

last lines from one of the data nodes
2008-07-25 10:10:33,398 INFO org.apache.hadoop.dfs.DataNode: BlockReport of 28 blocks got
processed in 3 msecs
2008-07-25 11:08:15,040 INFO org.apache.hadoop.dfs.DataNode: BlockReport of 28 blocks got
processed in 2 msecs
2008-07-25 12:05:56,871 INFO org.apache.hadoop.dfs.DataNode: BlockReport of 28 blocks got
processed in 2 msecs
2008-07-25 13:03:38,503 INFO org.apache.hadoop.dfs.DataNode: BlockReport of 28 blocks got
processed in 2 msecs

The relvant portion of my hbase-site.xml
<property>
    <name>hbase.rootdir</name>
    <value>hdfs://domU-12-31-39-00-E9-23:50001/hbase</value>
    <description>The directory shared by region servers.
    </description>
  </property>


Any ideas on where I can look to find an error message to help make sense of this?


      
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message