Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: neutral (nike.apache.org: 209.85.192.52 is neither permitted nor
 denied by domain of discord@uw.edu)
MIME-Version: 1.0
Date: Thu, 7 Aug 2014 14:00:37 -0700
Message-ID: 
 <CAB-gU_v1nKUFXxCxGOQXWvYqGrq+nB2aOdqwsU_6J-9VeeQy3w@mail.gmail.com>
Subject: Large discrepancy in hdfs hbase rootdir size after copytable
 operation.
From: Colin Kincaid Williams <discord@uw.edu>
To: user@hbase.apache.org
Content-Type: multipart/alternative; boundary=001a11c1275066a7e105001061c3

--001a11c1275066a7e105001061c3
Content-Type: text/plain; charset=UTF-8

I used the copy table command to copy a database between the original
cluster A and a new cluster B. I have noticed that the rootdir is larger
than 2X the size of the original. I am trying to account for such a large
difference. The following are some details about the table.


I'm trying to figure out why my copied table is more than 2X the size of
the original table. Could the bloomfilter itself account for this?

The guide I used as a reference:
http://blog.pivotal.io/pivotal/products/migrating-an-apache-hbase-table-between-different-clusters


Supposedly the original command used to create the table on cluster A:

create 'ADMd5', {NAME => 'a', BLOOMFILTER => 'ROW', VERSIONS => '1',
COMPRESSION => 'SNAPPY', MIN_VERSIONS => '0'}


How I created the target table on cluster B:

create 'ADMd5','a',{


BLOOMFILTER => 'ROW',
VERSIONS => '1',
COMPRESSION => 'SNAPPY',
MIN_VERSIONS => '0',
SPLITS =>['/++ASUZm4u7YsTcF/VtK6Q==',
'/zyuFR1VmhJyF4rbWsFnEg==',
'0sZYnBd83ul58d1O8I2JnA==',
'2+03N7IicZH3ltrqZUX6kQ==',
'4+/slRQtkBDU7Px6C9MAbg==',
'6+1dGCQ/IBrCsrNQXe/9xQ==',
'7+2pvtpHUQHWkZJoouR9wQ==',
'8+4n2deXhzmrpe//2Fo6Fg==',
'9+4SKW/BmNzpL68cXwKV1Q==',
'A+4ajStFkjEMf36cX5D9xg==',
'B+6Zm6Kccb3l6iM2L0epxQ==',
'C+6lKKDiOWl5qrRn72fNCw==',
'D+6dZMyn7m+NhJ7G07gqaw==',
'E+6BrimmrpAd92gZJ5hyMw==',
'G+5tisu4xWZMOJnDHeYBJg==',
'I+7fRy4dvqcM/L6dFRQk9g==',
'J+8ECMw1zeOyjfOg/ypXJA==',
'K+7tenLYn6a1aNLniL6tbg==']}


How the tables now appear in hbase shell:

table A:

describe 'ADMd5'
DESCRIPTION

  ENABLED

 {NAME => 'ADMd5', FAMILIES => [{NAME => 'a', BLOOMFILTER => 'NONE',
REPLICATION_SCOPE => '0', VERSIONS => '3', COMPRESSION => 'NONE', MIN_VER
true

 SIONS => '0', TTL => '2147483647', BLOCKSIZE => '65536', IN_MEMORY =>
'false', BLOCKCACHE => 'true'}]}


1 row(s) in 0.0370 seconds


table B:

hbase(main):003:0> describe 'ADMd5'
DESCRIPTION

  ENABLED

 {NAME => 'ADMd5', FAMILIES => [{NAME => 'a', BLOOMFILTER => 'ROW',
REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'SNAPPY', MIN_VE
true

 RSIONS => '0', TTL => '2147483647', BLOCKSIZE => '65536', IN_MEMORY =>
'false', BLOCKCACHE => 'true'}]}


1 row(s) in 0.0280 seconds


The containing foldersize in hdfs:
table A:
sudo -u hdfs hadoop fs -dus -h /a_d
dus: DEPRECATED: Please use 'du -s' instead.
227.4g  /a_d

table B:
sudo -u hdfs hadoop fs -dus -h /a_d
dus: DEPRECATED: Please use 'du -s' instead.
501.0g  /a_d


https://gist.github.com/drocsid/80bba7b6b19d64fde6c2

--001a11c1275066a7e105001061c3--