hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 梁景明 <futur...@gmail.com>
Subject Error to input Large data into Hbase whith One regionserver
Date Wed, 11 Nov 2009 09:45:23 GMT
Hi , i have a large data sized 27G to insert into hbase  whith one
regionserver.
and i used mapreduce to insert the data.
Whatever how much maps i used, how maps' thread to sleep to control the
speed,
it just worked before 20% data inserted  ,then failed , and hbase can't
start again.
it's weird.

is there some pool to store the data for insert,if there is too much than
its size ,will caused error.
so , will  .flushCommits() clear that pool? thanks for any help.

*here is my code to insert in a map process*.

============================
        table = new HTable(conf, tablename.getBytes());
        table.setAutoFlush(true);
        Put p = new Put(Bytes.toBytes(obj.getKey()));
        HashMap cols = obj.getColumns();
        .......
        table.put(p);
        table.flushCommits();
================================

*here is my logs*

Hadoop insert  log
======================================================
org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact
region server 192.168.1.116:60020 for region
chapter,b74054c6fba7f1f072c6a3a4fc3d329a,1257926538767, row
'b7e49883d0380b4194025170f8f9cb7f', but failed after 10 attempts.
Exceptions:
java.net.ConnectException: Connection refused at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:1001)
at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers$2.doCall(HConnectionManager.java:1192)

	at org.apache.hadoop.hbase.client.HConnectionManager$TableServers$Batch.process(HConnectionManager.java:1114)
	at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.processBatchOfRows(HConnectionManager.java:1200)
	at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:605)
	at org.apache.hadoop.hbase.client.HTable.put(HTable.java:470)
	at com.soko.hbase.tool.HbaseUtil.insertData(HbaseUtil.java:118)
	at com.soko.mr.HbaseFictionMR$Map.map(HbaseFictionMR.java:50)
	at com.soko.mr.HbaseFictionMR$Map.map(HbaseFictionMR.java:1)
	at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
	at org.apache.hadoop.mapred.Child.main(Child.java:170)
=========================================================================================

hbase-futureha-regionserver-ubuntu5.log
=================================================================
2009-11-11 16:44:06,998 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed chapter,90c0011cb6287924c818d371a27e145f,1257924043800
2009-11-11 16:44:06,998 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: aborting server at:
192.168.1.116:60020
2009-11-11 16:44:09,091 INFO org.apache.hadoop.hbase.Leases:
regionserver/192.168.1.116:60020.leaseChecker closing leases
2009-11-11 16:44:09,091 INFO org.apache.hadoop.hbase.Leases:
regionserver/192.168.1.116:60020.leaseChecker closed leases
2009-11-11 16:44:10,931 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: worker thread exiting
2009-11-11 16:44:10,932 INFO org.apache.zookeeper.ZooKeeper: Closing
session: 0x124e11cbae00001
2009-11-11 16:44:10,932 INFO org.apache.zookeeper.ClientCnxn: Closing
ClientCnxn for session: 0x124e11cbae00001
2009-11-11 16:44:10,968 INFO org.apache.zookeeper.ClientCnxn: Exception
while closing send thread for session 0x124e11cbae00001 : Read error rc = -1
java.nio.DirectByteBuffer[pos=0 lim=4 cap=4]
================================================================

Master
----------------------------------------
2009-11-11 16:59:38,792 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 2 on 60000: exiting
2009-11-11 16:59:38,791 INFO org.apache.zookeeper.ZooKeeper: Closing
session: 0x124e11cbae00000
2009-11-11 16:59:38,793 INFO org.apache.zookeeper.ClientCnxn: Closing
ClientCnxn for session: 0x124e11cbae00000
2009-11-11 16:59:38,795 INFO org.apache.zookeeper.ClientCnxn: Exception
while closing send thread for session 0x124e11cbae00000 : Read error rc = -1
java.nio.DirectByteBuffer[pos=0 lim=4 cap=4]
-----------------------------------------

ZooKeeper
=============================
2009-11-11 16:59:38,219 INFO org.apache.zookeeper.server.NIOServerCnxn:
Creating new session 0x124e11cbae0007e
2009-11-11 16:59:38,229 INFO org.apache.zookeeper.server.NIOServerCnxn:
Finished init of 0x124e11cbae0007e valid:true
2009-11-11 16:59:38,494 WARN
org.apache.zookeeper.server.PrepRequestProcessor: Got exception when
processing sessionid:0x124e11cbae00000 type:create cxid:0x10
zxid:0xfffffffffffffffe txntype:unknown n/a
org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode =
NodeExists
    at
org.apache.zookeeper.server.PrepRequestProcessor.pRequest(PrepRequestProcessor.java:245)
    at
org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:114)
2009-11-11 16:59:38,793 INFO
org.apache.zookeeper.server.PrepRequestProcessor: Processed session
termination request for id: 0x124e11cbae00000
============================

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message