hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "cpeel@kence.org (Updated) (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HDFS-2635) NNBench creates duplicate files if multiple maps are run from the same client
Date Tue, 06 Dec 2011 20:09:39 GMT

     [ https://issues.apache.org/jira/browse/HDFS-2635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

cpeel@kence.org updated HDFS-2635:
----------------------------------

    Target Version/s: 0.20.203.0
              Status: Patch Available  (was: Open)

--- NNBench.java	2011-12-06 12:03:49.000000000 -0800
+++ NNBench.new	2011-12-06 12:03:31.000000000 -0800
@@ -722,16 +722,16 @@
       if (barrier()) {
         if (op.equals(OP_CREATE_WRITE)) {
           startTimeTPmS = System.currentTimeMillis();
-          doCreateWriteOp("file_" + hostName + "_", reporter);
+          doCreateWriteOp("file_" + hostName + "_" + key, reporter);
         } else if (op.equals(OP_OPEN_READ)) {
           startTimeTPmS = System.currentTimeMillis();
-          doOpenReadOp("file_" + hostName + "_", reporter);
+          doOpenReadOp("file_" + hostName + "_" + key, reporter);
         } else if (op.equals(OP_RENAME)) {
           startTimeTPmS = System.currentTimeMillis();
-          doRenameOp("file_" + hostName + "_", reporter);
+          doRenameOp("file_" + hostName + "_" + key, reporter);
         } else if (op.equals(OP_DELETE)) {
           startTimeTPmS = System.currentTimeMillis();
-          doDeleteOp("file_" + hostName + "_", reporter);
+          doDeleteOp("file_" + hostName + "_" + key, reporter);
         }
         
         endTimeTPms = System.currentTimeMillis();
                
> NNBench creates duplicate files if multiple maps are run from the same client
> -----------------------------------------------------------------------------
>
>                 Key: HDFS-2635
>                 URL: https://issues.apache.org/jira/browse/HDFS-2635
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.20.205.0, 0.20.204.0, 0.20.203.0
>            Reporter: cpeel@kence.org
>            Priority: Minor
>
> NNBench creates files in the format:
>   file_<hostname>__<filenum>
> This works seamlessly as long as all of the Hadoop clients in the cluster are each running
with a single map slot. If multiple map slots are available on a single client, each mapper
tries to create the same set of files. This can result in lock contention on some non-Hadoop
HDFS implementations thereby defeating the purpose of the NNBench test.
> Making the files unique per mapper, not per host, resolves this.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message