hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "jiraposter@reviews.apache.org (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-5712) Parallelize load of .regioninfo files in diagnostic/repair portion of hbck.
Date Mon, 30 Apr 2012 06:42:54 GMT

    [ https://issues.apache.org/jira/browse/HBASE-5712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13264691#comment-13264691
] 

jiraposter@reviews.apache.org commented on HBASE-5712:
------------------------------------------------------



bq.  On 2012-04-27 23:27:20, Michael Stack wrote:
bq.  > src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java, line 204
bq.  > <https://reviews.apache.org/r/4883/diff/1/?file=104442#file104442line204>
bq.  >
bq.  >     This'll work but why not ConcurrentSkipListMap?

Sure, changed.


bq.  On 2012-04-27 23:27:20, Michael Stack wrote:
bq.  > src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java, line 418
bq.  > <https://reviews.apache.org/r/4883/diff/1/?file=104442#file104442line418>
bq.  >
bq.  >     +1 on suggested change

done.


bq.  On 2012-04-27 23:27:20, Michael Stack wrote:
bq.  > src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java, line 2835
bq.  > <https://reviews.apache.org/r/4883/diff/1/?file=104442#file104442line2835>
bq.  >
bq.  >     Is this flag needed?  Why not just check thread is alive?  I see we can return
with an error.  What happens if the return on 2816 happens?  Will the wait at #643 above be
for ever?

This is not a thread but actually fed to an executor (thread pool) at line 637.  If the return
happens on 2816, this is in a finally which will always mark the workitem as done. 

There are two other instances of this pattern that were originally in this code before I got
to it -- I'd have used Futures (and have filed a follow on issue for it) but it works.


- jmhsieh


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4883/#review7337
-----------------------------------------------------------


On 2012-04-26 01:42:01, jmhsieh wrote:
bq.  
bq.  -----------------------------------------------------------
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/4883/
bq.  -----------------------------------------------------------
bq.  
bq.  (Updated 2012-04-26 01:42:01)
bq.  
bq.  
bq.  Review request for hbase, Ted Yu and Jimmy Xiang.
bq.  
bq.  
bq.  Summary
bq.  -------
bq.  
bq.  * Parallelized load of .regioninfo files
bq.  * changed TreeMap to SortedMap in method signatures
bq.  * renamed a test's name.
bq.  
bq.  
bq.  This addresses bug HBASE-5712.
bq.      https://issues.apache.org/jira/browse/HBASE-5712
bq.  
bq.  
bq.  Diffs
bq.  -----
bq.  
bq.    src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java 66156c2 
bq.    src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java 6b64f10 
bq.  
bq.  Diff: https://reviews.apache.org/r/4883/diff
bq.  
bq.  
bq.  Testing
bq.  -------
bq.  
bq.  Ran patch 10x on trunk, passes.  Ran 1x on 0.92 and 0.94.
bq.  
bq.  Ther 0.90 version that is nearly identical except for ignoring changes near lines HBaseFsck
lines 671-680.
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  jmhsieh
bq.  
bq.


                
> Parallelize load of .regioninfo files in diagnostic/repair portion of hbck.
> ---------------------------------------------------------------------------
>
>                 Key: HBASE-5712
>                 URL: https://issues.apache.org/jira/browse/HBASE-5712
>             Project: HBase
>          Issue Type: Improvement
>          Components: hbck
>    Affects Versions: 0.90.7, 0.92.2, 0.94.0, 0.96.0
>            Reporter: Jonathan Hsieh
>            Assignee: Jonathan Hsieh
>             Fix For: 0.90.7, 0.92.2, 0.94.0, 0.96.0
>
>         Attachments: hbase-5712-90-v2.patch, hbase-5712-90.patch, hbase-5712-v2.patch,
hbase-5712.patch
>
>
> On heavily loaded hdfs's some dfs nodes may not respond quickly and backs off for 60s
before attempting to read data from another datanode.  Portions of the information gathered
from hdfs (.regioninfo files) are loaded serially.  With HBase with clusters with 100's, or
1000's, or 10000's regions encountering these 60s delay blocks progress and can be very painful.
 
> There is already some parallelization of portions of the hdfs information load operations
and the goal here is move the reading of .regioninfos into the parallelized sections..

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message