hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jean-Daniel Cryans <jdcry...@apache.org>
Subject Re: copy_table.rb doesn't work
Date Fri, 10 Jun 2011 17:33:22 GMT
The ruby scripts have a tendency to rot since they aren't tested as
part of our unit test suite... we should probably just delete
copy_table.rb since IIRC it was a hacked up script that Stack needed
for Powerset in 2008.

There's no practical way of copying a table, in the most recent
version there's the CopyTable mapreduce job that you could use but in
0.20 I think you could distcp the table's folder to another cluster
and try it there or have another hbase instance using the same HDFS
but root to a different folder. In both cases you'll have to use the
add_table.rb script.

J-D

On Fri, Jun 10, 2011 at 10:40 AM, James Hammerton
<james.hammerton@mendeley.com> wrote:
> Hi,
>
> I did some digging around in the HBase source and found that HLogEdit last
> existed in version 0.18 (we're on 0.20.6) and
> HConstants.COL_REGIONINFO_ARRAY last existed in version 0.2.
>
> After this digging around and making some educated guesses I made a new
> version of the script which I've attached to this message.
>
> I've tried running this script on a small 525M (on disc) table in a test
> instance of HBase and it's been running for over 12 mins, and is repeatedly
> outputting messages like the following:
>
> 11/06/10 11:27:05 DEBUG hfile.LruBlockCache: Cache Stats: Sizes:
> Total=1.6478577MB (1727904), Free=197.52715MB (207122224), Max=199.175MB
> (208850128), Counts: Blocks=3, Access=3, Hit=0, Miss=3, Evictions=0,
> Evicted=0, Ratios: Hit Ratio=0.0%, Miss Ratio=100.0%, Evicted/Run=NaN
> 11/06/10 11:28:05 DEBUG hfile.LruBlockCache: Cache Stats: Sizes:
> Total=1.6478577MB (1727904), Free=197.52715MB (207122224), Max=199.175MB
> (208850128), Counts: Blocks=3, Access=3, Hit=0, Miss=3, Evictions=0,
> Evicted=0, Ratios: Hit Ratio=0.0%, Miss Ratio=100.0%, Evicted/Run=NaN
>
> The du command tells me that the copied table's directory so far takes up
> just 4k on disk. Is my hacked script actually doing anything useful?
>
> James
>
> On Fri, Jun 10, 2011 at 11:19 AM, James Hammerton
> <james.hammerton@mendeley.com> wrote:
>>
>> [Re-sending as I'm not sure this got through]
>>
>> Hi,
>>
>> Before trying to merge regions on a table in our live database we decided
>> to copy the table and merge the regions on the copy first to test the
>> merging code works before risking our live data.
>>
>> However when we try to run the copy_table.rb script it fails due to the
>> following error (this is actually from a test HBase instance but the error
>> from the live instance is the same, modulo different paths):
>>
>>
>> file:/home/james/hbase-0.20.6/lib/jruby-complete-1.2.0.jar!/META-INF/jruby.home/lib/ruby/site_ruby/1.8/builtin/javasupport/core_ext/object.rb:33:in
>> `get_proxy_or_package_under_package': cannot load Java class
>> org.apache.hadoop.hbase.regionserver.HLogEdit (NameError)
>>     from
>> file:/home/james/hbase-0.20.6/lib/jruby-complete-1.2.0.jar!/META-INF/jruby.home/lib/ruby/site_ruby/1.8/builtin/javasupport/java.rb:51:in
>> `method_missing'
>>     from /home/james/hbase/bin/copy_table.rb:40
>>
>> Looking through the copy_table.rb file I could so no reference to HLogEdit
>> other than the import, so I tried removing it. Now running the script
>> produces the error below:
>>
>>
>> file:/home/james/hbase-0.20.6/lib/jruby-complete-1.2.0.jar!/META-INF/jruby.home/lib/ruby/site_ruby/1.8/builtin/java/ast.rb:80:in
>> `const_missing': uninitialized constant
>> Java::OrgApacheHadoopHbase::HConstants::COL_REGIONINFO_ARRAY (NameError)
>>     from /home/james/hbase/bin/copy_table.rb:124
>>     from
>> file:/home/james/hbase-0.20.6/lib/jruby-complete-1.2.0.jar!/META-INF/jruby.home/lib/ruby/site_ruby/1.8/builtin/java/collections.rb:47:in
>> `call'
>>     from
>> file:/home/james/hbase-0.20.6/lib/jruby-complete-1.2.0.jar!/META-INF/jruby.home/lib/ruby/site_ruby/1.8/builtin/java/collections.rb:47:in
>> `each'
>>     from /home/james/hbase/bin/copy_table.rb:118
>>
>> I looked at the latest version of the copy_table.rb script and it also
>> references both HLogEdit and HConstants::COL_REGIONINFO_
>> ARRAY...
>> Any ideas how to fix this?
>>
>> Regards,
>>
>> James
>>
>> --
>> James Hammerton | Senior Data Mining Engineer
>> www.mendeley.com/profiles/james-hammerton
>>
>> Mendeley Limited | London, UK | www.mendeley.com
>> Registered in England and Wales | Company Number 6419015
>>
>>
>>
>
>
>
> --
> James Hammerton | Senior Data Mining Engineer
> www.mendeley.com/profiles/james-hammerton
>
> Mendeley Limited | London, UK | www.mendeley.com
> Registered in England and Wales | Company Number 6419015
>
>
>
>

Mime
View raw message