Mailing-List: contact dev-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hbase.apache.org
Received-SPF: pass (nike.apache.org: domain of yuzhihong@gmail.com designates
 74.125.83.169 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <096289DE-8873-4FB7-8FA7-558657853BF6@gmail.com>
References: <096289DE-8873-4FB7-8FA7-558657853BF6@gmail.com>
Date: Sat, 2 Jul 2011 13:55:09 -0700
Message-ID: 
 <CALte62xgMo2W8kgqiFL4nr=JPW5aLXJVbY0F__Ao4p4N-m-iZA@mail.gmail.com>
Subject: Re: Merge and HMerge
From: Ted Yu <yuzhihong@gmail.com>
To: dev@hbase.apache.org
Content-Type: multipart/alternative; boundary=bcaec531469b8a804804a71c59f7

--bcaec531469b8a804804a71c59f7
Content-Type: text/plain; charset=ISO-8859-1

There is TestMergeTool which tests Merge.

>From the log you provided, I got a little confused as why
'testtable,row-20,1309613053987.23a35ac696bdf4a8023dcc4c5b8419e0.' didn't
appear in your command line or the output from .META. scanning.

On Sat, Jul 2, 2011 at 10:36 AM, Lars George <lars.george@gmail.com> wrote:

> Hi,
>
> These two seem both in a bit of a weird state: HMerge is scoped package
> local, therefore no one but the package can call the merge() functions...
> and no one does that but the unit test. But it would be good to have this on
> the CLI and shell as a command (and in the shell maybe with a confirmation
> message?), but it is not available AFAIK.
>
> HMerge can merge regions of tables that are disabled. It also merges all
> that qualify, i.e. where the merged region is less than or equal of half the
> configured max file size.
>
> Merge on the other hand does have a main(), so can be invoked:
>
> $ hbase org.apache.hadoop.hbase.util.Merge
> Usage: bin/hbase merge <table-name> <region-1> <region-2>
>
> Note how the help insinuates that you can use it as a tool, but that is not
> correct. Also, it only merges two given regions, and the cluster must be
> shut down (only the HBase daemons). So that is a step back.
>
> What is worse is that I cannot get it to work. I tried in the shell:
>
> hbase(main):001:0> create 'testtable', 'colfam1',  {SPLITS =>
> ['row-10','row-20','row-30','row-40','row-50']}
> 0 row(s) in 0.2640 seconds
>
> hbase(main):002:0> for i in '0'..'9' do for j in '0'..'9' do put
> 'testtable', "row-#{i}#{j}", "colfam1:#{j}", "#{j}" end end
> 0 row(s) in 1.0450 seconds
>
> hbase(main):003:0> flush 'testtable'
> 0 row(s) in 0.2000 seconds
>
> hbase(main):004:0> scan '.META.', { COLUMNS => ['info:regioninfo']}
> ROW                                  COLUMN+CELL
>  testtable,,1309614509037.612d1e0112 column=info:regioninfo,
> timestamp=130...
>  406e6c2bb482eeaec57322.             STARTKEY => '', ENDKEY => 'row-10'
>  testtable,row-10,1309614509040.2fba column=info:regioninfo,
> timestamp=130...
>  fcc9bc6afac94c465ce5dcabc5d1.       STARTKEY => 'row-10', ENDKEY =>
> 'row-20'
>  testtable,row-20,1309614509041.e7c1 column=info:regioninfo,
> timestamp=130...
>  6267eb30e147e5d988c63d40f982.       STARTKEY => 'row-20', ENDKEY =>
> 'row-30'
>  testtable,row-30,1309614509041.a9cd column=info:regioninfo,
> timestamp=130...
>  e1cbc7d1a21b1aca2ac7fda30ad8.       STARTKEY => 'row-30', ENDKEY =>
> 'row-40'
>  testtable,row-40,1309614509041.d458 column=info:regioninfo,
> timestamp=130...
>  236feae097efcf33477e7acc51d4.       STARTKEY => 'row-40', ENDKEY =>
> 'row-50'
>  testtable,row-50,1309614509041.74a5 column=info:regioninfo,
> timestamp=130...
>  7dc7e3e9602d9229b15d4c0357d1.       STARTKEY => 'row-50', ENDKEY => ''
> 6 row(s) in 0.0440 seconds
>
> hbase(main):005:0> exit
>
> $ ./bin/stop-hbase.sh
>
> $ hbase org.apache.hadoop.hbase.util.Merge testtable \
>  testtable,row-20,1309614509041.e7c16267eb30e147e5d988c63d40f982. \
>  testtable,row-30,1309614509041.a9cde1cbc7d1a21b1aca2ac7fda30ad8.
>
> But I get consistently errors:
>
> 11/07/02 07:20:49 INFO util.Merge: Merging regions
> testtable,row-20,1309613053987.23a35ac696bdf4a8023dcc4c5b8419e0. and
> testtable,row-30,1309613053987.3664920956c30ac5ff2a7726e4e6 in table
> testtable
> 11/07/02 07:20:49 INFO wal.HLog: HLog configuration: blocksize=32 MB,
> rollsize=30.4 MB, enabled=true, optionallogflushinternal=1000ms
> 11/07/02 07:20:49 INFO wal.HLog: New hlog
> /Volumes/Macintosh-HD/Users/larsgeorge/.logs_1309616449171/hlog.1309616449181
> 11/07/02 07:20:49 INFO wal.HLog: getNumCurrentReplicas--HDFS-826 not
> available; hdfs_out=org.apache.hadoop.fs.FSDataOutputStream@25961581,
> exception=org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.getNumCurrentReplicas()
> 11/07/02 07:20:49 INFO regionserver.HRegion: Setting up tabledescriptor
> config now ...
> 11/07/02 07:20:49 INFO regionserver.HRegion: Onlined -ROOT-,,0.70236052;
> next sequenceid=1
> info: null
> region1: [B@48fd918a
> region2: [B@7f5e2075
> 11/07/02 07:20:49 FATAL util.Merge: Merge failed
> java.io.IOException: Could not find meta region for
> testtable,row-20,1309613053987.23a35ac696bdf4a8023dcc4c5b8419e0.
>        at
> org.apache.hadoop.hbase.util.Merge.mergeTwoRegions(Merge.java:211)
>        at org.apache.hadoop.hbase.util.Merge.run(Merge.java:111)
>        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>        at org.apache.hadoop.hbase.util.Merge.main(Merge.java:386)
> 11/07/02 07:20:49 INFO regionserver.HRegion: Setting up tabledescriptor
> config now ...
> 11/07/02 07:20:49 INFO regionserver.HRegion: Onlined .META.,,1.1028785192;
> next sequenceid=1
> 11/07/02 07:20:49 INFO regionserver.HRegion: Closed -ROOT-,,0.70236052
> 11/07/02 07:20:49 INFO wal.HLog: main.logSyncer exiting
> 11/07/02 07:20:49 ERROR util.Merge: exiting due to error
> java.lang.NullPointerException
>        at org.apache.hadoop.hbase.util.Merge$1.processRow(Merge.java:119)
>        at
> org.apache.hadoop.hbase.util.MetaUtils.scanMetaRegion(MetaUtils.java:229)
>        at
> org.apache.hadoop.hbase.util.MetaUtils.scanMetaRegion(MetaUtils.java:258)
>        at org.apache.hadoop.hbase.util.Merge.run(Merge.java:116)
>        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>        at org.apache.hadoop.hbase.util.Merge.main(Merge.java:386)
>
> After which I most of the times have shot .META. with an error
>
> 2011-07-02 06:42:10,763 WARN org.apache.hadoop.hbase.master.HMaster: Failed
> getting all descriptors
> java.io.FileNotFoundException: No status for
> hdfs://localhost:8020/hbase/.corrupt
>        at
> org.apache.hadoop.hbase.util.FSUtils.getTableInfoModtime(FSUtils.java:888)
>        at
> org.apache.hadoop.hbase.util.FSTableDescriptors.get(FSTableDescriptors.java:122)
>        at
> org.apache.hadoop.hbase.util.FSTableDescriptors.getAll(FSTableDescriptors.java:149)
>        at
> org.apache.hadoop.hbase.master.HMaster.getHTableDescriptors(HMaster.java:1429)
>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>        at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>        at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>        at java.lang.reflect.Method.invoke(Method.java:597)
>        at
> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:312)
>        at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1065)
>
> Lars

--bcaec531469b8a804804a71c59f7--