hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hadoop QA (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-1912) Datanode should support block replacement
Date Fri, 26 Oct 2007 03:16:50 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-1912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12537819

Hadoop QA commented on HADOOP-1912:

-1 overall.  Here are the results of testing the latest attachment 
against trunk revision r588341.

    @author +1.  The patch does not contain any @author tags.

    javadoc +1.  The javadoc tool did not generate any warning messages.

    javac +1.  The applied patch does not generate any new compiler warnings.

    findbugs +1.  The patch does not introduce any new Findbugs warnings.

    core tests -1.  The patch failed core unit tests.

    contrib tests +1.  The patch passed contrib unit tests.

Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/998/testReport/
Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/998/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/998/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/998/console

This message is automatically generated.

> Datanode should support block replacement
> -----------------------------------------
>                 Key: HADOOP-1912
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1912
>             Project: Hadoop
>          Issue Type: New Feature
>          Components: dfs
>    Affects Versions: 0.14.1
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.16.0
>         Attachments: replace.patch, replace1.patch, replace2.patch, replace3.patch, replace4.patch,
replace5.patch, replace6.patch
> This jira Data Node's support for rebalancing (HADOOP-1652). When a balancer decides
to move a block B from Source S to Destination D. It also chooses a proxy source PS, which
contains a replica of B, to speed up block copy.  The block placement is carried in the following
> 1. A block copy command is sent to datanode PS in the format of  "OP_BLOCK_COPY <block_id_of_B>
<source S> <destination D>". It requests PS to copy B to datanode D.
> 2. PS then transfers block B to datanode D with a block replacement command to D in the
format of "OP_BLOCK_REPLACEMENT <block_id_of_B> <source S> <data_of_B>".

> 3. Datanode D writes the block B to its disk and then sends a name node a blockReceived
RPC informing the namenode that a block B is received and please delete a replica of B from
source S if there is any excessive replica.
> 4. The namenode then adds datanode D to block B's map and removes an exesive replicas
of B in favor of datanode S.
> In addition, each data node has a limited bandwidth for rebalancing. The default value
for the bandwidth is 5MB/s. Throttling is done at both source & destination sides. Each
data node limits maximum number of concurrent data transfers (including both sending and receiving)
for the rebalancing purpose to be 5. In the worst case, each data transfer has a limited bandwidth
of 1MB/s. Each sender & receiver has a Throttler. The primary method of the class is "throttle(
int numOfBytes )". The parameter numOfBytes indicates the total number of bytes that the caller
has sent or received since the last throttle is called. The method calculates the caller's
I/O rate. If the rate is faster than the bandwidth limit, it sleeps to slow down the data
transfer. After it wakes up, it adjusts its bandwidth limit if the number of concurrent data
transfers is changed. 

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message