hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kihwal Lee <kih...@yahoo-inc.com.INVALID>
Subject Re: RollingUpgrade Rollback openfiles issue
Date Thu, 25 May 2017 16:46:10 GMT
Hi Vinay,
If I rephrase the question,
Does a RU rollback snapshot provide a consistent snapshot of the distributed file system?
I don't think we aimed it to be a completely consistent snapshot. It is meant to be a safe
place to go back with the old version of software.  This is normally used as a last resort.
By design, the datanodes will have extra blocks on rollback, which will be invalidated quickly.
 But the short presence of blocks with "future" block ids still can interfere with block
allocations after rolling back, if the cluster is used right away.  As you pointed out, the
under-construction block length is not recorded either.
>But now, extra bytes are seen after rollback. Is this correct?I think it is a reasonable
compromise.  If you can make a general argument against it, we can revisit the design and
try to fix it.
Kihwal

      From: Vinayakumar B <vinayakumar.ba@huawei.com>
 To: "hdfs-dev@hadoop.apache.org" <hdfs-dev@hadoop.apache.org> 
 Sent: Thursday, May 25, 2017 8:10 AM
 Subject: RollingUpgrade Rollback openfiles issue
   

Hi all,

Have a doubt with expected behavior in case of RollingUpgrade and rollback.

Scenario:

1. file was being written before rolling upgrade started and written some bytes, say X, with
hsync().
2. Rolling upgrade done and writer continued to write and added some more bytes and file closed
with X+Y bytes.
3. Now rollback done.

i. Current state of the File is UNDERCONSTRUCTION.
Ii. getFileStatus() returns with size X. BUT in replicas there is a FINALIZED replica with
size X+Y.
iii. recoverLease() on the file closes the file with X+Y bytes.

Question:
  What should be the size here after rollback + recoverLease()?
  Since user always writes with hsync(), application might have some other track of how much
bytes written. But now, extra bytes are seen after rollback. Is this correct?

-vinay

   
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message