hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Nick Dimiduk (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-8803) region_mover.rb should move multiple regions at a time
Date Wed, 17 Jul 2013 22:02:51 GMT

    [ https://issues.apache.org/jira/browse/HBASE-8803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13711694#comment-13711694
] 

Nick Dimiduk commented on HBASE-8803:
-------------------------------------

bq. Are you sure you are using the patched version?

Definitely running v4. Looks like this patch includes the thread count in the load command,
but not in unload.

bq. also, how many active nodes do you have in you HBase cluster?

This cluster has 9 RS's, so your logic makes sense. It looks like unloading regions is still
operating 8 at a time for me.

{noformat}
Unloading hor18n39.gq1.ygridcore.net region(s)
...
Valid region move targets:                                                               
                                                                                         
             [392/465]
hor18n37.gq1.ygridcore.net,60020,1374094975268
hor17n37.gq1.ygridcore.net,60020,1374094975264
hor18n35.gq1.ygridcore.net,60020,1374094975327
hor17n39.gq1.ygridcore.net,60020,1374094975281
hor18n36.gq1.ygridcore.net,60020,1374094975254
hor17n36.gq1.ygridcore.net,60020,1374094975277
hor17n34.gq1.ygridcore.net,60020,1374094975291
hor18n38.gq1.ygridcore.net,60020,1374094975259
13/07/17 21:44:38 INFO region_mover: Moving 330 region(s) from hor18n39.gq1.ygridcore.net,60020,1374094975326
during this cycle
13/07/17 21:44:38 INFO region_mover: Moving region b59050cf97aabcef838e3c50e93e6d13 (1 of
330) to server=hor18n37.gq1.ygridcore.net,60020,1374094975268
13/07/17 21:44:38 INFO region_mover: Moving region 1f1a21f370963529326e1e8647c77977 (2 of
330) to server=hor17n37.gq1.ygridcore.net,60020,1374094975264
13/07/17 21:44:38 INFO region_mover: Moving region 214a5b6dd53cb94e5fe549712df10918 (3 of
330) to server=hor18n35.gq1.ygridcore.net,60020,1374094975327
13/07/17 21:44:38 INFO region_mover: Moving region 157d5afaffe570ef1d5535162ee61a73 (4 of
330) to server=hor17n39.gq1.ygridcore.net,60020,1374094975281
13/07/17 21:44:38 INFO region_mover: Moving region af5448c90e78a8f0d935efb0b380502e (5 of
330) to server=hor18n36.gq1.ygridcore.net,60020,1374094975254
13/07/17 21:44:38 INFO region_mover: Moving region bbe7e56f1e895379bfb6bbc2ef3ebec7 (6 of
330) to server=hor17n36.gq1.ygridcore.net,60020,1374094975277
13/07/17 21:44:38 INFO region_mover: Moving region cdf96aa8d79255991c82267052cf9870 (7 of
330) to server=hor17n34.gq1.ygridcore.net,60020,1374094975291
13/07/17 21:44:38 INFO region_mover: Moving region 7d0a02f452c334a12026b45346a87d36 (8 of
330) to server=hor18n38.gq1.ygridcore.net,60020,1374094975259
13/07/17 21:45:41 INFO region_mover: Moving region 01408a1987f7a0cb7fb657a19bddc2d5 (9 of
330) to server=hor18n37.gq1.ygridcore.net,60020,1374094975268
13/07/17 21:45:41 INFO region_mover: Moving region 45d7cf6b5997310b022f315e52f64da2 (10 of
330) to server=hor17n37.gq1.ygridcore.net,60020,1374094975264
13/07/17 21:45:41 INFO region_mover: Moving region 353a09c07053144a598be04a53afa82e (11 of
330) to server=hor18n35.gq1.ygridcore.net,60020,1374094975327
13/07/17 21:45:41 INFO region_mover: Moving region 5f4e8165d58c0672c8c6e58ddbff486c (12 of
330) to server=hor17n39.gq1.ygridcore.net,60020,1374094975281
13/07/17 21:45:41 INFO region_mover: Moving region 234265b7ec4e535cecafcdd83f09dab7 (13 of
330) to server=hor18n36.gq1.ygridcore.net,60020,1374094975254
13/07/17 21:45:41 INFO region_mover: Moving region 9f54ff65021922f7c13f4372dbcb8f98 (14 of
330) to server=hor17n36.gq1.ygridcore.net,60020,1374094975277
13/07/17 21:45:41 INFO region_mover: Moving region 0db22200ccea02196dfea00b33811413 (15 of
330) to server=hor17n34.gq1.ygridcore.net,60020,1374094975291
13/07/17 21:45:41 INFO region_mover: Moving region 905210e29ef683ecf78f01050c4b1148 (16 of
330) to server=hor18n38.gq1.ygridcore.net,60020,1374094975259
...
{noformat}

Loading regions is launching threads, at least according to your new log messages. Here I
would expect 30 threads to spin up all at once. Instead, I see one at a time:

{noformat}
Reloading hor18n39.gq1.ygridcore.net region(s)
...
13/07/17 21:54:27 INFO region_mover: Moving 330 regions to hor18n39.gq1.ygridcore.net,60020,1374098064602
13/07/17 21:56:47 INFO region_mover: Moving region 7d0a02f452c334a12026b45346a87d36 (1 of
330) to server=hor18n39.gq1.ygridcore.net,60020,1374098064602 in thread 0
13/07/17 21:56:54 INFO region_mover: Moving region af5448c90e78a8f0d935efb0b380502e (2 of
330) to server=hor18n39.gq1.ygridcore.net,60020,1374098064602 in thread 1
13/07/17 21:57:01 INFO region_mover: Moving region 157d5afaffe570ef1d5535162ee61a73 (3 of
330) to server=hor18n39.gq1.ygridcore.net,60020,1374098064602 in thread 2
13/07/17 21:57:06 INFO region_mover: Moving region cdf96aa8d79255991c82267052cf9870 (4 of
330) to server=hor18n39.gq1.ygridcore.net,60020,1374098064602 in thread 3
13/07/17 21:57:12 INFO region_mover: Moving region 214a5b6dd53cb94e5fe549712df10918 (5 of
330) to server=hor18n39.gq1.ygridcore.net,60020,1374098064602 in thread 4
13/07/17 21:57:18 INFO region_mover: Moving region b59050cf97aabcef838e3c50e93e6d13 (6 of
330) to server=hor18n39.gq1.ygridcore.net,60020,1374098064602 in thread 5
13/07/17 21:57:23 INFO region_mover: Moving region bbe7e56f1e895379bfb6bbc2ef3ebec7 (7 of
330) to server=hor18n39.gq1.ygridcore.net,60020,1374098064602 in thread 6
13/07/17 21:57:28 INFO region_mover: Moving region 1f1a21f370963529326e1e8647c77977 (8 of
330) to server=hor18n39.gq1.ygridcore.net,60020,1374098064602 in thread 7
13/07/17 21:57:33 INFO region_mover: Moving region 9f54ff65021922f7c13f4372dbcb8f98 (9 of
330) to server=hor18n39.gq1.ygridcore.net,60020,1374098064602 in thread 8
13/07/17 21:57:37 INFO region_mover: Moving region 905210e29ef683ecf78f01050c4b1148 (10 of
330) to server=hor18n39.gq1.ygridcore.net,60020,1374098064602 in thread 9
13/07/17 21:57:42 INFO region_mover: Moving region 234265b7ec4e535cecafcdd83f09dab7 (11 of
330) to server=hor18n39.gq1.ygridcore.net,60020,1374098064602 in thread 10
...
13/07/17 21:59:17 INFO region_mover: Moving region da0f100db394d18e5b6311938097ec3e (35 of
330) to server=hor18n39.gq1.ygridcore.net,60020,1374098064602 in thread 28
13/07/17 21:59:20 INFO region_mover: Moving region 900effc2e95064173fc0a3fab047b119 (36 of
330) to server=hor18n39.gq1.ygridcore.net,60020,1374098064602 in thread 29
13/07/17 21:59:28 INFO region_mover: Moving region 1b78694f962d7c315e4f6b64f07af4aa (37 of
330) to server=hor18n39.gq1.ygridcore.net,60020,1374098064602 in thread 0
13/07/17 21:59:32 INFO region_mover: Moving region 3e47c04afce79818e5ead7d8f46e7128 (38 of
330) to server=hor18n39.gq1.ygridcore.net,60020,1374098064602 in thread 1
13/07/17 21:59:36 INFO region_mover: Moving region 7cce3224f05e438f0e011d6f26b738c0 (39 of
330) to server=hor18n39.gq1.ygridcore.net,60020,1374098064602 in thread 2
13/07/17 21:59:39 INFO region_mover: Region t1,1735ed70,1373581705674.ef04ef42d7eb7d7e472671661cc48ee0.
(40 of 330) already on target server=hor18n39.gq1.ygridcore.net,60020,1374098064602
13/07/17 21:59:43 INFO region_mover: Region t1,2888871d,1373581705680.483e522559ecd6b0fdd96dbd6b593c7d.
(41 of 330) already on target server=hor18n39.gq1.ygridcore.net,60020,1374098064602
13/07/17 21:59:46 INFO region_mover: Moving region 6337b0857ccc6d995082582841e467d7 (42 of
330) to server=hor18n39.gq1.ygridcore.net,60020,1374098064602 in thread 3
...
{noformat}
                
> region_mover.rb should move multiple regions at a time
> ------------------------------------------------------
>
>                 Key: HBASE-8803
>                 URL: https://issues.apache.org/jira/browse/HBASE-8803
>             Project: HBase
>          Issue Type: Bug
>          Components: Usability
>    Affects Versions: 0.98.0, 0.94.8, 0.95.1
>            Reporter: Jean-Marc Spaggiari
>            Assignee: Jean-Marc Spaggiari
>             Fix For: 0.98.0, 0.95.2, 0.94.11
>
>         Attachments: HBASE-8803-v0-trunk.patch, HBASE-8803-v1-0.94.patch, HBASE-8803-v1-trunk.patch,
HBASE-8803-v2-0.94.patch, HBASE-8803-v2-0.94.patch, HBASE-8803-v3-0.94.patch, HBASE-8803-v4-0.94.patch
>
>   Original Estimate: 48h
>  Remaining Estimate: 48h
>
> When there is many regions in a cluster, rolling_restart can take hours because region_mover
is moving the regions one by one.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message