hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Karl Kuntz <kku...@tradebotsystems.com>
Subject RE: online_merge.rb from HBASE-1621 on 0.92.1
Date Thu, 03 May 2012 17:24:30 GMT
Thanks for the response. 

It's working now.  Turns out that the issue is a race condition between the HBaseAdmin.unassign()
call and deleting the associated records in META. 
By inserting a "sleep 1" between the two calls all was cleared up.  

Just a hunch, but it seems that maybe the behavior of the 'force' parameter in the unassign()
call has changed a bit? I noticed that parameter disappeared from the corresponding assign()
method.  At any rate, without the sleep some of the rows in META don't get deleted and old
directories on disk don't get cleaned up, presumably b/c they're still locked by the master.

The diff of my changes follows in case anybody would find it useful. I did just use nil for
the server info when closing regions so I didn't have to deal with serverstartcodes which
works, but I'm not sure if looking up the server from META again at this point in the script
could open up issues. 

clust01:~> diff online_merge.rb online_merge_updated.rb 
<     server = HServerAddress.new(String.from_java_bytes(value))
>     serverInfo = String.from_java_bytes(value).split(':')
>     server = HServerAddress.new(serverInfo[0],Integer(serverInfo[1]))
<         admin.closeRegion(row , server.getHostname + ":" + server.getPort.to_s)
>         admin.closeRegion(row, nil)
<   newHRI = HRegionInfo.new(tableDesc, firstHRI.getStartKey, lastHRI.getEndKey)
>   newHRI = HRegionInfo.new(tableDesc.getName, firstHRI.getStartKey, lastHRI.getEndKey)
<     HRegion.makeColumnFamilyDirs(fs, tableDir, newHRI, family)
>     HRegion.makeColumnFamilyDirs(fs, tableDir, newHRI, family) if normalRun
<       admin.unassign(row, true)
>       admin.unassign(row,true)
>       sleep 1
<     admin.assign(newHRI.getRegionName, true)
>     admin.assign(newHRI.getRegionName)

-----Original Message-----
From: saint.ack@gmail.com [mailto:saint.ack@gmail.com] On Behalf Of Stack
Sent: Wednesday, May 02, 2012 11:51 PM
To: user@hbase.apache.org
Subject: Re: online_merge.rb from HBASE-1621 on 0.92.1

On Wed, May 2, 2012 at 1:31 PM, Karl Kuntz <kkuntz@tradebotsystems.com> wrote:
> After looking at this again, the data is intact (I can count/scan all rows), and the
new regions are loaded on the different region servers, but the web UI doesn't show any regions
for the table and warnings appear in the log:
> 2012-05-02 14:51:43,847 WARN org.apache.hadoop.hbase.master.CatalogJanitor: REGIONINFO_QUALIFIER
is empty in keyvalues={test,,1335982638210.66ebbe65667be38836cfb9ee809b6b48./info:server/1335982868378/Put/vlen=29,
> At this point I'm wondering what's keeping the web UI from showing the regions for the
table. Is just the web UI out of sync, or are there other, potentially bigger issues as well?

WebUI view of cluster is produced via a scan of the .META. table so
its odd that it shows the tables as w/o regions (but you can
scan/count the table anyways -- the client uses .META. table to figure
out whats where stuff is).

Pastebin more of the master log?


View raw message