hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Colin Patrick McCabe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-3731) 2.0 release upgrade must handle blocks being written from 1.0
Date Tue, 21 Aug 2012 01:41:38 GMT

    [ https://issues.apache.org/jira/browse/HDFS-3731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13438387#comment-13438387
] 

Colin Patrick McCabe commented on HDFS-3731:
--------------------------------------------

bq. Colin, the approach and your patch looks good to me. What's the latest on testing? Eg
you verified a v1 install with hsync files correctly upgraded (eg leases recovered) to v2
or trunk build with this patch?

I installed branch-1 and started a one-node cluster.  I have a small test program which creates
a file, calls {{hsync}} on it, and then sleeps for a few minutes.  During its sleep, I brought
down the cluster.  Then I had a block in blocksBeingWritten corresponding to the last block
of that file that was hsync'ed.  (I guess it's called {{sync}} in branch-1).

Then installed a hacked version of trunk running this patch.  The hack was to decrease the
lease hard recovery time.  It is normally hard-coded to be an hour and I didn't want to wait
that long.

I ran {{./bin/start-dfs.sh -upgrade}}.  I confirmed that the last block of the file was recovered
and moved into {{rbw}}, then into {{finalized/}} and then could be read by me.

Then I stopped the cluster, ran ./bin/namenode -finalize, answered 'Y', and restarted the
cluster.  The {{blocksBeingWritten}} directory was gone, as well as the {{previous}} directory.

The unit test does basically the same thing, in a more systematic way.  Hope this helps.
                
> 2.0 release upgrade must handle blocks being written from 1.0
> -------------------------------------------------------------
>
>                 Key: HDFS-3731
>                 URL: https://issues.apache.org/jira/browse/HDFS-3731
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: data-node
>    Affects Versions: 2.0.0-alpha
>            Reporter: Suresh Srinivas
>            Assignee: Colin Patrick McCabe
>            Priority: Blocker
>         Attachments: HDFS-3731.002.patch, HDFS-3731.003.patch
>
>
> Release 2.0 upgrades must handle blocks being written to (bbw) files from 1.0 release.
Problem reported by Brahma Reddy.
> The {{DataNode}} will only have one block pool after upgrading from a 1.x release.  (This
is because in the 1.x releases, there were no block pools-- or equivalently, everything was
in the same block pool).  During the upgrade, we should hardlink the block files from the
{{blocksBeingWritten}} directory into the {{rbw}} directory of this block pool.  Similarly,
on {{-finalize}}, we should delete the {{blocksBeingWritten}} directory.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message