hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: Merge and HMerge
Date Sat, 02 Jul 2011 19:06:04 GMT
Can you file a JIRA ?

Thanks



On Jul 2, 2011, at 10:36 AM, Lars George <lars.george@gmail.com> wrote:

> Hi,
> 
> These two seem both in a bit of a weird state: HMerge is scoped package local, therefore
no one but the package can call the merge() functions... and no one does that but the unit
test. But it would be good to have this on the CLI and shell as a command (and in the shell
maybe with a confirmation message?), but it is not available AFAIK.
> 
> HMerge can merge regions of tables that are disabled. It also merges all that qualify,
i.e. where the merged region is less than or equal of half the configured max file size.
> 
> Merge on the other hand does have a main(), so can be invoked:
> 
> $ hbase org.apache.hadoop.hbase.util.Merge
> Usage: bin/hbase merge <table-name> <region-1> <region-2>
> 
> Note how the help insinuates that you can use it as a tool, but that is not correct.
Also, it only merges two given regions, and the cluster must be shut down (only the HBase
daemons). So that is a step back.
> 
> What is worse is that I cannot get it to work. I tried in the shell:
> 
> hbase(main):001:0> create 'testtable', 'colfam1',  {SPLITS => ['row-10','row-20','row-30','row-40','row-50']}
> 0 row(s) in 0.2640 seconds
> 
> hbase(main):002:0> for i in '0'..'9' do for j in '0'..'9' do put 'testtable', "row-#{i}#{j}",
"colfam1:#{j}", "#{j}" end end
> 0 row(s) in 1.0450 seconds
> 
> hbase(main):003:0> flush 'testtable'
> 0 row(s) in 0.2000 seconds
> 
> hbase(main):004:0> scan '.META.', { COLUMNS => ['info:regioninfo']}
> ROW                                  COLUMN+CELL
> testtable,,1309614509037.612d1e0112 column=info:regioninfo, timestamp=130...
> 406e6c2bb482eeaec57322.             STARTKEY => '', ENDKEY => 'row-10'
> testtable,row-10,1309614509040.2fba column=info:regioninfo, timestamp=130...
> fcc9bc6afac94c465ce5dcabc5d1.       STARTKEY => 'row-10', ENDKEY => 'row-20'
> testtable,row-20,1309614509041.e7c1 column=info:regioninfo, timestamp=130...
> 6267eb30e147e5d988c63d40f982.       STARTKEY => 'row-20', ENDKEY => 'row-30'
> testtable,row-30,1309614509041.a9cd column=info:regioninfo, timestamp=130...
> e1cbc7d1a21b1aca2ac7fda30ad8.       STARTKEY => 'row-30', ENDKEY => 'row-40'
> testtable,row-40,1309614509041.d458 column=info:regioninfo, timestamp=130...
> 236feae097efcf33477e7acc51d4.       STARTKEY => 'row-40', ENDKEY => 'row-50'
> testtable,row-50,1309614509041.74a5 column=info:regioninfo, timestamp=130...
> 7dc7e3e9602d9229b15d4c0357d1.       STARTKEY => 'row-50', ENDKEY => ''
> 6 row(s) in 0.0440 seconds
> 
> hbase(main):005:0> exit
> 
> $ ./bin/stop-hbase.sh
> 
> $ hbase org.apache.hadoop.hbase.util.Merge testtable \
> testtable,row-20,1309614509041.e7c16267eb30e147e5d988c63d40f982. \
> testtable,row-30,1309614509041.a9cde1cbc7d1a21b1aca2ac7fda30ad8.
> 
> But I get consistently errors:
> 
> 11/07/02 07:20:49 INFO util.Merge: Merging regions testtable,row-20,1309613053987.23a35ac696bdf4a8023dcc4c5b8419e0.
and testtable,row-30,1309613053987.3664920956c30ac5ff2a7726e4e6 in table testtable
> 11/07/02 07:20:49 INFO wal.HLog: HLog configuration: blocksize=32 MB, rollsize=30.4 MB,
enabled=true, optionallogflushinternal=1000ms
> 11/07/02 07:20:49 INFO wal.HLog: New hlog /Volumes/Macintosh-HD/Users/larsgeorge/.logs_1309616449171/hlog.1309616449181
> 11/07/02 07:20:49 INFO wal.HLog: getNumCurrentReplicas--HDFS-826 not available; hdfs_out=org.apache.hadoop.fs.FSDataOutputStream@25961581,
exception=org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.getNumCurrentReplicas()
> 11/07/02 07:20:49 INFO regionserver.HRegion: Setting up tabledescriptor config now ...
> 11/07/02 07:20:49 INFO regionserver.HRegion: Onlined -ROOT-,,0.70236052; next sequenceid=1
> info: null
> region1: [B@48fd918a
> region2: [B@7f5e2075
> 11/07/02 07:20:49 FATAL util.Merge: Merge failed
> java.io.IOException: Could not find meta region for testtable,row-20,1309613053987.23a35ac696bdf4a8023dcc4c5b8419e0.
>        at org.apache.hadoop.hbase.util.Merge.mergeTwoRegions(Merge.java:211)
>        at org.apache.hadoop.hbase.util.Merge.run(Merge.java:111)
>        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>        at org.apache.hadoop.hbase.util.Merge.main(Merge.java:386)
> 11/07/02 07:20:49 INFO regionserver.HRegion: Setting up tabledescriptor config now ...
> 11/07/02 07:20:49 INFO regionserver.HRegion: Onlined .META.,,1.1028785192; next sequenceid=1
> 11/07/02 07:20:49 INFO regionserver.HRegion: Closed -ROOT-,,0.70236052
> 11/07/02 07:20:49 INFO wal.HLog: main.logSyncer exiting
> 11/07/02 07:20:49 ERROR util.Merge: exiting due to error
> java.lang.NullPointerException
>        at org.apache.hadoop.hbase.util.Merge$1.processRow(Merge.java:119)
>        at org.apache.hadoop.hbase.util.MetaUtils.scanMetaRegion(MetaUtils.java:229)
>        at org.apache.hadoop.hbase.util.MetaUtils.scanMetaRegion(MetaUtils.java:258)
>        at org.apache.hadoop.hbase.util.Merge.run(Merge.java:116)
>        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>        at org.apache.hadoop.hbase.util.Merge.main(Merge.java:386)
> 
> After which I most of the times have shot .META. with an error
> 
> 2011-07-02 06:42:10,763 WARN org.apache.hadoop.hbase.master.HMaster: Failed getting all
descriptors
> java.io.FileNotFoundException: No status for hdfs://localhost:8020/hbase/.corrupt
>        at org.apache.hadoop.hbase.util.FSUtils.getTableInfoModtime(FSUtils.java:888)
>        at org.apache.hadoop.hbase.util.FSTableDescriptors.get(FSTableDescriptors.java:122)
>        at org.apache.hadoop.hbase.util.FSTableDescriptors.getAll(FSTableDescriptors.java:149)
>        at org.apache.hadoop.hbase.master.HMaster.getHTableDescriptors(HMaster.java:1429)
>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>        at java.lang.reflect.Method.invoke(Method.java:597)
>        at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:312)
>        at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1065)
> 
> Lars

Mime
View raw message