hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Colin Patrick McCabe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-7758) Retire FsDatasetSpi#getVolumes() and use FsDatasetSpi#getVolumeRefs() instead
Date Fri, 06 Mar 2015 22:43:38 GMT

    [ https://issues.apache.org/jira/browse/HDFS-7758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14351058#comment-14351058

Colin Patrick McCabe commented on HDFS-7758:

   * Returns a list of volume references.
   * The caller must release the reference of each volume by calling
   * {@link FsVolumeReference#close}.
  public List<FsVolumeReference> getVolumeRefs();

  /** Returns a reference of a given volume, specified by the index. */
  public FsVolumeReference getVolumeRef(int idx) throws IOException;

This is still the wrong interface.  {{getVolumeRef(int)}} encourages people to assume that
the number of volumes is never going to change.  What happens if it does?

Instead of doing this, let's have an {{Iterator}} that we can use.  Something like this:

public Iterator<FsVolumeRef> getVolumeRefIterator();

private static class FsVolumeRefIterator
      implements Iterator<FsVolumeRef>, Closeable {
  private final List<FsVolumeRef> list;

  private int idx = 0;

  FsVolumeRefIterator(List<FsVolumeSpi> spiList) {
    this.list = new ArrayList<FsVolumeRef>();
    for (FsVolumeSpi volume : spiList) {
      try {
      } catch (ClosedChannelException e) {
        LOG.info("Can't obtain a reference to {} because it is closed.",

  public boolean hasNext() {
    return (idx < list.size());

  public FsVolumeRef next() {
    int i = idx++;
    return list.get(i);

  public void remove() {
    throw UnsupportedOperationException();

  public void close() throws IOException {
    for (FsVolumeRef ref : list) {

Then we can get rid of {{getVolumeRefs}} and {{getVolumeRef}}.  Since the {{Iterator}} implements
{{java.io.Closeable}}, findbugs will remind us that we need to close it (and free the refs)
in any function we use it in.

> Retire FsDatasetSpi#getVolumes() and use FsDatasetSpi#getVolumeRefs() instead
> -----------------------------------------------------------------------------
>                 Key: HDFS-7758
>                 URL: https://issues.apache.org/jira/browse/HDFS-7758
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: datanode
>    Affects Versions: 2.6.0
>            Reporter: Lei (Eddy) Xu
>            Assignee: Lei (Eddy) Xu
>         Attachments: HDFS-7758.000.patch, HDFS-7758.001.patch
> HDFS-7496 introduced reference-counting  the volume instances being used to prevent race
condition when hot swapping a volume.
> However, {{FsDatasetSpi#getVolumes()}} can still leak the volume instance without increasing
its reference count. In this JIRA, we retire the {{FsDatasetSpi#getVolumes()}} and propose
{{FsDatasetSpi#getVolumeRefs()}} and etc. method to access {{FsVolume}}. Thus it makes sure
that the consumer of {{FsVolume}} always has correct reference count.

This message was sent by Atlassian JIRA

View raw message