hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "zhoutai.zt (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HADOOP-15020) NNBench not support run more than one map task on one host
Date Tue, 07 Nov 2017 07:30:00 GMT
zhoutai.zt created HADOOP-15020:
-----------------------------------

             Summary: NNBench not support run more than one map task on one host
                 Key: HADOOP-15020
                 URL: https://issues.apache.org/jira/browse/HADOOP-15020
             Project: Hadoop Common
          Issue Type: Improvement
          Components: benchmarks
    Affects Versions: 2.7.2
         Environment: Hadoop 2.7.2
            Reporter: zhoutai.zt
            Priority: Minor


When benchmark NameNode performance with NNBench. I start with pseudo distributed deploy.
Everything goes well with "-maps 1". BUT with -maps N (n>1) and -operation create_write,
many exceptions meet during the benchmark.

Hostname is part of the file path, which can  differentiate hosts. With more than two map
tasks run on the same host, more than two map tasks may operate on the same file, leading
to exceptions.

17/11/07 15:22:32 INFO hdfs.NNBench:                  RAW DATA: AL Total #1: 84
17/11/07 15:22:32 INFO hdfs.NNBench:                  RAW DATA: AL Total #2: 43
17/11/07 15:22:32 INFO hdfs.NNBench:               RAW DATA: TPS Total (ms): 2570
17/11/07 15:22:32 INFO hdfs.NNBench:        RAW DATA: Longest Map Time (ms): 814.0
17/11/07 15:22:32 INFO hdfs.NNBench:                    RAW DATA: Late maps: 0
17/11/07 15:22:32 INFO hdfs.NNBench:              {color:red}RAW DATA: # of exceptions: 3000{color}

2017-11-07 14:54:08,082 INFO org.apache.hadoop.hdfs.NNBench: Exception recorded in op: Create/Write/Close
2017-11-07 14:54:08,082 INFO org.apache.hadoop.hdfs.NNBench: Exception recorded in op: Create/Write/Close
2017-11-07 14:54:08,083 INFO org.apache.hadoop.hdfs.NNBench: Exception recorded in op: Create/Write/Close




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message