Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hdfs-issues@hadoop.apache.org
Message-ID: <577832.48401277417931379.JavaMail.jira@thor>
Date: Thu, 24 Jun 2010 18:18:51 -0400 (EDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: hdfs-issues@hadoop.apache.org
Subject: [jira] Commented: (HDFS-1186) 0.20: DNs should interrupt writers at
 start of recovery
In-Reply-To: <8074260.166471275595439491.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HDFS-1186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12882365#action_12882365 ] 

Todd Lipcon commented on HDFS-1186:
-----------------------------------

Hi Sam, thanks for taking a look. I think you're right that in some really weird timing scenarios we might have a problem:

{noformat}
writer writes offset 1 and syncs, gs=1

NN recovery starts:
  - interrupts writer, gets metadata (len 1)
  - recovering DN hangs for a little bit

writer recovery starts, picks a different primary DN:
  - interrupts writer (noop)
  - gets metadata (len 1)
  - gets new GS=2
  - syncs blocks to GS=2 len=1
  - restarts pipeline
  - writes and syncs some more data to block with GS=2

NN-directed recovery proceeds:
  - gets new GS=3   (this has to be at least 10 seconds after above due to lastRecoveryTime check)
  - calls updateBlock on all DNs, which truncates files
{noformat}

I think the issue here is that the genstamp can be incremented in between startBlockRecovery() and updateBlock(), and thus updateBlock is allowing an update based on stale recovery info. If we simply added a check in tryUpdateBlock() that oldblock.getGenerationStamp() == oldgs, I think we'd be safe. What do you think?


> 0.20: DNs should interrupt writers at start of recovery
> -------------------------------------------------------
>
>                 Key: HDFS-1186
>                 URL: https://issues.apache.org/jira/browse/HDFS-1186
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: data-node
>    Affects Versions: 0.20-append
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Blocker
>         Attachments: hdfs-1186.txt
>
>
> When block recovery starts (eg due to NN recovering lease) it needs to interrupt any writers currently writing to those blocks. Otherwise, an old writer (who hasn't realized he lost his lease) can continue to write+sync to the blocks, and thus recovery ends up truncating data that has been sync()ed.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.