hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mingliang Liu (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-11903) Ozone: Cleaning up local storage when closing MiniOzoneCluster
Date Tue, 30 May 2017 22:56:04 GMT

    [ https://issues.apache.org/jira/browse/HDFS-11903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16030302#comment-16030302
] 

Mingliang Liu commented on HDFS-11903:
--------------------------------------

Thanks [~anu] for review and for the suggestion. Yes I noticed that I can set the {{OzoneConfigKeys.OZONE_LOCALSTORAGE_ROOT}}
in OzoneConfig and build a new MiniOzoneCluster with that config object.

One possible problem is that, the {{OzoneMetadataManager}} is singleton, so only the first
OzoneConfig is used to check the local {{OzoneMetadataManager::storageRoot}} so the {{/metadata.db}}
and {{/user.db}} will be shared. I currently don't need to create the two {{OzoneMiniCluster}}
now, but I assume that will be useful someday, say running distcp between them. So overall,
current test will fail with duplicate volume/bucket exception:
{code}
  @Test
  public void testCloseWillCleanFiles() throws Exception {
    final String vName = "test-close-will-clean-files-volume";
    final String bName = "test-close-will-clean-files-bucket";
    try (MiniOzoneCluster c1 = new MiniOzoneCluster.Builder(new OzoneConfiguration())
        .numDataNodes(1)
        .setHandlerType("local")
        .build()) {
      OzoneVolume v = c1.createOzoneClient().createVolume(vName, "L", "1TB");
      v.createBucket(bName);
    }
    try (MiniOzoneCluster c2 = new MiniOzoneCluster.Builder(new OzoneConfiguration())
        .numDataNodes(1)
        .setHandlerType("local")
        .build()) {
      OzoneVolume v = c2.createOzoneClient().createVolume(vName, "L", "1TB");
      v.createBucket(bName);
    }
  }
{code}
And it should fail even when we set different {{OzoneConfigKeys.OZONE_LOCALSTORAGE_ROOT}}
I assume (not verified).

> Ozone: Cleaning up local storage when closing MiniOzoneCluster
> --------------------------------------------------------------
>
>                 Key: HDFS-11903
>                 URL: https://issues.apache.org/jira/browse/HDFS-11903
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: ozone
>            Reporter: Mingliang Liu
>            Assignee: Mingliang Liu
>         Attachments: HDFS-11903-HDFS-7240.000.patch
>
>
> Currently we don't delete the local storage when closing {{MiniOzoneCluster}}, which
will cause creating volume with the same name to fail, even in a new JVM process. Let's delete
the local storage files: {{/tmp/ozone/metadata.db}} and {{/tmp/ozone/user.db}} when closing.
> Please note this is not a complete solution. As the {{OzoneMetadataManager}}  currently
is a singleton, we are not able to create two independent MiniOzoneCluster in a single JVM.
Data maybe shared unexpectedly. Let's address that in another JIRA, if needed.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message