hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ajay Kumar (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HDFS-12942) Synchronization issue in FSDataSetImpl#moveBlock
Date Fri, 12 Jan 2018 21:59:00 GMT

    [ https://issues.apache.org/jira/browse/HDFS-12942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16324607#comment-16324607
] 

Ajay Kumar edited comment on HDFS-12942 at 1/12/18 9:58 PM:
------------------------------------------------------------

{quote}While cleaning up the new replica on failure, we call volume.onBlockFileDeletion which
decrements the space used for the block on the new target volume. However I am not sure that
the space used was incremented in the first place by copyReplicaToVolume. This may be a pre-existing
bug actually.{quote}

[~arpitagarwal],[~virajith] good catch. 
{code} try (AutoCloseableLock lock = datasetLock.acquire()) {
      // Increment numBlocks here as this block moved without knowing to BPS
      FsVolumeImpl volume = (FsVolumeImpl) newReplicaInfo.getVolume();
      volume.incrNumBlocks(block.getBlockPoolId());
    }{code}
Seems in case of success we should increment the "dfs used" and "no of blocks" for new volume
and decrement the same for old volume. Currently we only increment no of blocks. Is this bug
or intentional? [~arpitagarwal], thanks for offline discussion on this. 
In case of failure we should free up memory on disk.  (will update patch for same)
{quote}Adding to this, in patch v4, the generation stamp check in finalizeReplica is done
after a call to v.addFinalizedBlock which seems wasted work if the check doesn't pass. Can
the generation stamp check be done before the call to v.addFinalizedBlock?{quote}
Agree,Will move check in start of finalize call.


was (Author: ajayydv):
{quote}While cleaning up the new replica on failure, we call volume.onBlockFileDeletion which
decrements the space used for the block on the new target volume. However I am not sure that
the space used was incremented in the first place by copyReplicaToVolume. This may be a pre-existing
bug actually.{quote}

[~arpitagarwal],[~virajith] good catch. 
{code} try (AutoCloseableLock lock = datasetLock.acquire()) {
      // Increment numBlocks here as this block moved without knowing to BPS
      FsVolumeImpl volume = (FsVolumeImpl) newReplicaInfo.getVolume();
      volume.incrNumBlocks(block.getBlockPoolId());
    }{code}
Seems in case of success we should increment the dfs used and no of blocks for new volume
and decrement the same for old block. Currently we only increment no of blocks. Is this bug
or intentional? [~arpitagarwal], thanks for offline discussion on this. 
In case of failure we should free up memory on disk.  (will update patch for same)
{quote}Adding to this, in patch v4, the generation stamp check in finalizeReplica is done
after a call to v.addFinalizedBlock which seems wasted work if the check doesn't pass. Can
the generation stamp check be done before the call to v.addFinalizedBlock?{quote}
Agree,Will move check in start of finalize call.

> Synchronization issue in FSDataSetImpl#moveBlock
> ------------------------------------------------
>
>                 Key: HDFS-12942
>                 URL: https://issues.apache.org/jira/browse/HDFS-12942
>             Project: Hadoop HDFS
>          Issue Type: Bug
>            Reporter: Ajay Kumar
>            Assignee: Ajay Kumar
>         Attachments: HDFS-12942.001.patch, HDFS-12942.002.patch, HDFS-12942.003.patch,
HDFS-12942.004.patch
>
>
> FSDataSetImpl#moveBlock  works in following following 3 steps:
> # first creates a new replicaInfo object
> # calls finalizeReplica to finalize it.
> # Calls removeOldReplica to remove oldReplica.
> A client can potentially append to the old replica between step 1 and 2.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message