hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Buttler, David" <buttl...@llnl.gov>
Subject RE: split not working
Date Thu, 04 Nov 2010 23:01:13 GMT
I have tried it from the shell and using the API.  The same thing happens: the log messages
report that a split request was sent, but then no further updates.
Is it possible that having GZ compression enabled hampers this operation?

Log messages attached.
Dave

Master:
-------------
2010-11-04 15:48:04,028 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers:
Rowscanned=4, rowsOffline=0
2010-11-04 15:48:30,757 DEBUG org.apache.hadoop.hbase.master.RegionManager: Adding operation
TABLE_SPLIT from tasklist
2010-11-04 15:48:30,757 DEBUG org.apache.hadoop.hbase.master.RegionManager: Adding operation
TABLE_SPLIT from tasklist
2010-11-04 15:48:30,757 DEBUG org.apache.hadoop.hbase.master.RegionManager: Adding operation
TABLE_SPLIT from tasklist
2010-11-04 15:48:30,757 DEBUG org.apache.hadoop.hbase.master.RegionManager: Adding operation
TABLE_SPLIT from tasklist
2010-11-04 15:48:31,135 DEBUG org.apache.hadoop.hbase.master.RegionManager: Sending MSG_REGION_SPLIT
REGION => {NAME => 'dbpedia,3fdc590551f245675682a4f1baedb59eca4c698d,128820517403
8.26fe0f3fc176da9403ab4cd94dc56f41.', STARTKEY => '3fdc590551f245675682a4f1baedb59eca4c698d',
ENDKEY => '7fcb1391fda134a4b2a17b494b52f65db2bad69e', ENCODED => 26fe0f3fc176da9403ab4cd
94dc56f41, TABLE => {{NAME => 'dbpedia', FAMILIES => [{NAME => 'annotations',
BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '3', COMPRESSION =>
'GZ', TTL => '214748364
7', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME =>
'meta', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '3', COMPRESSION
=> 'GZ', TTL => 
'2147483647', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'},
{NAME => 'src', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '3',
COMPRESSION => 'GZ'
, TTL => '2147483647', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE =>
'true'}, {NAME => 'text', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS
=> '3', COMPRESSI
ON => 'GZ', TTL => '2147483647', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE
=> 'true'}]}} to 10.220.5.20:60020
2010-11-04 15:48:31,520 DEBUG org.apache.hadoop.hbase.master.RegionManager: Sending MSG_REGION_SPLIT
REGION => {NAME => 'dbpedia,7fcb1391fda134a4b2a17b494b52f65db2bad69e,128820529378
4.2b6ce7d6ac234856cb808ac29b65e88a.', STARTKEY => '7fcb1391fda134a4b2a17b494b52f65db2bad69e',
ENDKEY => 'bffc513271d8d61b3510cb4780efddaa212481a0', ENCODED => 2b6ce7d6ac234856cb808ac
29b65e88a, TABLE => {{NAME => 'dbpedia', FAMILIES => [{NAME => 'annotations',
BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '3', COMPRESSION =>
'GZ', TTL => '214748364
7', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME =>
'meta', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '3', COMPRESSION
=> 'GZ', TTL => 
'2147483647', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'},
{NAME => 'src', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '3',
COMPRESSION => 'GZ'
, TTL => '2147483647', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE =>
'true'}, {NAME => 'text', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS
=> '3', COMPRESSI
ON => 'GZ', TTL => '2147483647', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE
=> 'true'}]}} to 10.220.5.17:60020
2010-11-04 15:48:31,521 DEBUG org.apache.hadoop.hbase.master.RegionManager: Sending MSG_REGION_SPLIT
REGION => {NAME => 'dbpedia,bffc513271d8d61b3510cb4780efddaa212481a0,128820529378
4.ae02b30fad134ad4e07b0bad5f3e1154.', STARTKEY => 'bffc513271d8d61b3510cb4780efddaa212481a0',
ENDKEY => '', ENCODED => ae02b30fad134ad4e07b0bad5f3e1154, TABLE => {{NAME =>
'dbpedia',
 FAMILIES => [{NAME => 'annotations', BLOOMFILTER => 'NONE', REPLICATION_SCOPE =>
'0', VERSIONS => '3', COMPRESSION => 'GZ', TTL => '2147483647', BLOCKSIZE => '65536',
IN_MEMORY => '
false', BLOCKCACHE => 'true'}, {NAME => 'meta', BLOOMFILTER => 'NONE', REPLICATION_SCOPE
=> '0', VERSIONS => '3', COMPRESSION => 'GZ', TTL => '2147483647', BLOCKSIZE =>
'65536', IN_M
EMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'src', BLOOMFILTER => 'NONE',
REPLICATION_SCOPE => '0', VERSIONS => '3', COMPRESSION => 'GZ', TTL => '2147483647',
BLOCKSIZE => '655
36', IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'text', BLOOMFILTER =>
'NONE', REPLICATION_SCOPE => '0', VERSIONS => '3', COMPRESSION => 'GZ', TTL =>
'2147483647', BLOCKSI
ZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}} to 10.220.5.17:60020
2010-11-04 15:48:31,536 DEBUG org.apache.hadoop.hbase.master.RegionManager: Sending MSG_REGION_SPLIT
REGION => {NAME => 'dbpedia,,1288205174038.5ed51d83d2db52b019c9650fc08316ec.', ST
ARTKEY => '', ENDKEY => '3fdc590551f245675682a4f1baedb59eca4c698d', ENCODED => 5ed51d83d2db52b019c9650fc08316ec,
TABLE => {{NAME => 'dbpedia', FAMILIES => [{NAME => 'annotations', BL
OOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '3', COMPRESSION =>
'GZ', TTL => '2147483647', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE
=> 'true'}, {NAME => 
CKCACHE => 'true'}]}} to 10.220.5.38:60020
2010-11-04 15:48:47,476 INFO org.apache.hadoop.hbase.master.ServerManager: 30 region servers,
0 dead, average load 63.06666666666667


-----------
Node containing a region:
2010-11-04 15:41:59,658 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_SPLIT:
dbpedia,3fdc590551f245675682a4f1baedb59eca4c698d,1288205174038.26fe0f3fc176da9403ab4cd94dc56f41.
2010-11-04 15:41:59,659 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_SPLIT:
dbpedia,3fdc590551f245675682a4f1baedb59eca4c698d,1288205174038.26fe0f3fc176da9403ab4cd94dc56f41.
2010-11-04 15:41:59,659 DEBUG org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction
requested for region dbpedia,3fdc590551f245675682a4f1baedb59eca4c698d,1288205174038.26fe0f3fc176da9403ab4cd94dc56f41.
because: MSG_REGION_SPLIT
2010-11-04 15:41:59,659 INFO org.apache.hadoop.hbase.regionserver.HRegion: Starting compaction
on region dbpedia,3fdc590551f245675682a4f1baedb59eca4c698d,1288205174038.26fe0f3fc176da9403ab4cd94dc56f41.
2010-11-04 15:41:59,659 DEBUG org.apache.hadoop.hbase.regionserver.Store: annotations: no
store files to compact
2010-11-04 15:41:59,659 DEBUG org.apache.hadoop.hbase.regionserver.Store: meta: no store files
to compact
2010-11-04 15:41:59,663 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed
on region dbpedia,3fdc590551f245675682a4f1baedb59eca4c698d,1288205174038.26fe0f3fc176da9403ab4cd94dc56f41.
in 0sec
2010-11-04 15:46:36,345 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=59.2
MB, free=739.13 MB, max=798.34 MB, blocks=830, accesses=67548, hits=0, hitRatio=0.00%%, evictions=0,
evicted=0, evictedPerRun=NaN
2010-11-04 15:48:09,624 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_SPLIT:
dbpedia,3fdc590551f245675682a4f1baedb59eca4c698d,1288205174038.26fe0f3fc176da9403ab4cd94dc56f41.
2010-11-04 15:48:09,624 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_SPLIT:
dbpedia,3fdc590551f245675682a4f1baedb59eca4c698d,1288205174038.26fe0f3fc176da9403ab4cd94dc56f41.
2010-11-04 15:48:09,624 DEBUG org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction
requested for region dbpedia,3fdc590551f245675682a4f1baedb59eca4c698d,1288205174038.26fe0f3fc176da9403ab4cd94dc56f41.
because: MSG_REGION_SPLIT
2010-11-04 15:48:09,624 INFO org.apache.hadoop.hbase.regionserver.HRegion: Starting compaction
on region dbpedia,3fdc590551f245675682a4f1baedb59eca4c698d,1288205174038.26fe0f3fc176da9403ab4cd94dc56f41.
2010-11-04 15:48:09,624 DEBUG org.apache.hadoop.hbase.regionserver.Store: annotations: no
store files to compact
2010-11-04 15:48:09,624 DEBUG org.apache.hadoop.hbase.regionserver.Store: meta: no store files
to compact
2010-11-04 15:48:09,626 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed
on region dbpedia,3fdc590551f245675682a4f1baedb59eca4c698d,1288205174038.26fe0f3fc176da9403ab4cd94dc56f41.
in 0sec
2010-11-04 15:51:36,345 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=59.2
MB, free=739.13 MB, max=798.34 MB, blocks=830, accesses=67548, hits=0, hitRatio=0.00%%, evictions=0,
evicted=0, evictedPerRun=NaN
2010-11-04 15:56:36,345 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=59.2
MB, free=739.13 MB, max=798.34 MB, blocks=830, accesses=67548, hits=0, hitRatio=0.00%%, evictions=0,
evicted=0, evictedPerRun=NaN

-----Original Message-----
From: saint.ack@gmail.com [mailto:saint.ack@gmail.com] On Behalf Of Stack
Sent: Wednesday, November 03, 2010 8:26 PM
To: user@hbase.apache.org
Subject: Re: split not working

Try it from the shell.  Try splitting individual regions (Scan .META.
first to see list or check your UI where it lists the regions in a
table).

St.Ack

On Wed, Nov 3, 2010 at 6:22 PM, Buttler, David <buttler1@llnl.gov> wrote:
> Hi all,
> I have a small table (3M rows) that is using 4 regions.  I want to split the table so
that I can take advantage of more nodes in my cluster with map/reduce tasks and the TableInputFormat.
> The split button on the web page sends split messages to the master/region server, but
nothing seems to happen - my regions don't split.
>
> I am using the latest cloudera branch hadoop-0.20.2+737, and hbase-089.20100924+28
>
> Alternatively, is there a way that I can specify a minimum number of splits for the TableInputFormat?
> Dave
>
>

Mime
View raw message