hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Adam Portley <a_port...@hotmail.com>
Subject a couple of HBase issues
Date Wed, 15 Dec 2010 20:04:22 GMT

I ran into a few problems when playing around with some tables yesterday (hbase version 0.89.20100924)
and was wondering if anyone has seen these before.  


1) Multiple listings for a single table 

I altered the schema of an existing table through the hbase shell (disabling the table, changing
the compression setting, and re-enabling it).  All seemed to work fine but the next day I
noticed that the list of user tables showed two entries for the table, one with the new schema
and one with the old.  Flushing and major_compacting META does not clear up the duplicate
entry.  One possible clue found in the logs is that it appears the region server hosting META
died in the interim: 

2010-12-14 18:41:49,107 WARN org.mortbay.log: /master.jsp: org.apache.hadoop.hbase.client.RetriesExhaustedException:
Trying to contact region server (nodename) for region .META.,,1, row '', but failed after
10 attempts.
java.io.IOException: Call to (nodename) failed on local exception: java.io.EOFException
2010-12-14 18:56:32,698 INFO org.apache.hadoop.hbase.master.ServerManager: (nodename),1291715592733
znode expired
2010-12-14 18:56:32,698 DEBUG org.apache.hadoop.hbase.master.ServerManager: Added=(nodename),1291715592733
to dead servers, added shutdown processing operation
2010-12-14 18:56:32,698 DEBUG org.apache.hadoop.hbase.master.RegionServerOperationQueue: Processing
todo: ProcessServerShutdown of (nodename),1291715592733
2010-12-14 18:56:32,699 INFO org.apache.hadoop.hbase.master.RegionServerOperation: Process
shutdown of server (nodename),1291715592733: logSplit: false, rootRescanned: false, numberOfMetaRegions:
1, onlineMetaRegions.size(): 0
2010-12-14 18:56:32,700 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: Splitting 6 hlog(s)
in (node2)/hbase/.logs/(nodename),1291715592733
...


2) Incremental bulk load failures.  

When trying to prepare HFiles for incremental bulk load (using HFileOutputFormat::configureIncrementalLoad)
for the table above, I'm running into the following exception: 

10/12/15 19:49:51 INFO mapreduce.HFileOutputFormat: Looking up current regions for table org.apache.hadoop.hbase.client.HTable@36d1c778
10/12/15 19:49:52 INFO mapreduce.HFileOutputFormat: Configuring 602 reduce partitions to match
current region count
10/12/15 19:49:52 INFO mapreduce.HFileOutputFormat: Writing partition information to (path)/partitions_1292442592073

10/12/15 19:38:29 INFO mapred.JobClient: Task Id : attempt_201011120210_5338_m_000022_8, Status
: FAILED
java.lang.IllegalArgumentException: Can't read partitions file
        at org.apache.hadoop.hbase.mapreduce.hadoopbackport.TotalOrderPartitioner.setConf(TotalOrderPartitioner.java:111)
        at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62)
        at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
        at org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:490)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:575)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
        at org.apache.hadoop.mapred.Child.main(Child.java:159)
Caused by: java.io.IOException: Wrong number of partitions in keyset:600
        at org.apache.hadoop.hbase.mapreduce.hadoopbackport.TotalOrderPartitioner.setConf(TotalOrderPartitioner.java:84)

The number of partitions in keyset (600) does not match the configured number of reduce tasks
(602).  
This incremental load procedure used to work for my table when it contained the original 500
regions but now that some regions have split it always fails. 


3) Regions stuck in unassigned limbo 

I created a new table yesterday (using loadtable.rb) - table creation and initial bulk load
succeeded - but the table is inaccessible because some of its regions are never assigned.
 I have tried disabling and re-enabling the table but a scan of META shows that the same regions
are still unassigned 24 hours later.  They have a info:regioninfo column but no info:server.
 Is there a way to force assignment of these regions?  

 		 	   		  
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message