hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-11182) Update DataNode to use DatasetVolumeChecker
Date Mon, 19 Dec 2016 22:04:58 GMT

    [ https://issues.apache.org/jira/browse/HDFS-11182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15762444#comment-15762444
] 

ASF GitHub Bot commented on HDFS-11182:
---------------------------------------

Github user arp7 commented on a diff in the pull request:

    https://github.com/apache/hadoop/pull/168#discussion_r93129689
  
    --- Diff: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/checker/DatasetVolumeChecker.java
---
    @@ -299,26 +327,35 @@ public void checkVolume(
       private class ResultHandler
           implements FutureCallback<VolumeCheckResult> {
         private final FsVolumeReference reference;
    -    private final Set<StorageLocation> failedVolumes;
    -    private final Set<StorageLocation> healthyVolumes;
    -    private final CountDownLatch latch;
    -    private final AtomicLong numVolumes;
    +    private final Set<FsVolumeSpi> failedVolumes;
    +    private final Set<FsVolumeSpi> healthyVolumes;
    +    private final Semaphore semaphore;
     
         @Nullable
         private final Callback callback;
     
    +    /**
    +     * 
    +     * @param reference FsVolumeReference to be released when the check is
    +     *                  complete.
    +     * @param healthyVolumes set of healthy volumes. If the disk check is
    +     *                       successful, add the volume here.
    +     * @param failedVolumes set of failed volumes. If the disk check fails,
    +     *                      add the volume here.
    +     * @param semaphore semaphore used to trigger callback invocation.
    --- End diff --
    
    CountDownLatch#countDown returns no value so there is no easy way to detect when the count
falls to zero and the callback can be invoked (it must be invoked once only). I was using
an AtomicLong to detect the 0->1 transition but it had a bug.
    
    The semaphore approach fixes it. We still need the CountDownLatch which we can use as
an event. I could have used an Object mutex instead but that would have required extra code
to deal with the spurious wakeup problem which CountDownLatch does not suffer from.


> Update DataNode to use DatasetVolumeChecker
> -------------------------------------------
>
>                 Key: HDFS-11182
>                 URL: https://issues.apache.org/jira/browse/HDFS-11182
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: datanode
>            Reporter: Arpit Agarwal
>            Assignee: Arpit Agarwal
>
> Update DataNode to use the DatasetVolumeChecker class introduced in HDFS-11149 to parallelize
disk checks.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message