hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Lixiang Ao <aolixi...@gmail.com>
Subject Re: Benchmark Failure
Date Sat, 22 Mar 2014 08:56:28 GMT
Checked the logs, and turned out to be configuration problem. Just
set dfs.namenode.fs-limits.min-block-size to 1 and it's fixed

Thanks.


On Wed, Mar 19, 2014 at 2:51 PM, Brahma Reddy Battula <
brahmareddy.battula@huawei.com> wrote:

>  Seems to be this is issue, which is logged..Please check following jirafor same....Hope
you also facing same issue...
>
>
>
> https://issues.apache.org/jira/browse/HDFS-4929
>
>
>
>
>
>
>
> Thanks & Regards
>
>
>
> Brahma Reddy Battula
>
>
>   ------------------------------
> *From:* Lixiang Ao [aolixiang@gmail.com]
> *Sent:* Tuesday, March 18, 2014 10:34 AM
> *To:* user@hadoop.apache.org
> *Subject:* Re: Benchmark Failure
>
>   the version is release 2.2.0
> 2014年3月18日 上午12:26于 "Lixiang Ao" <aolixiang@gmail.com>写道:
>
>>  Hi all,
>>
>>  I'm running jobclient tests(on single node), other tests like
>> TestDFSIO, mrbench succeed except nnbench.
>>
>>  I got a lot of Exceptions but without any explanation(see below).
>>
>>  Could anyone tell me what might went wrong?
>>
>>  Thanks!
>>
>>
>>  14/03/17 23:54:22 INFO hdfs.NNBench: Waiting in barrier for: 112819 ms
>> 14/03/17 23:54:23 INFO mapreduce.Job: Job job_local2133868569_0001
>> running in uber mode : false
>> 14/03/17 23:54:23 INFO mapreduce.Job:  map 0% reduce 0%
>> 14/03/17 23:54:28 INFO mapred.LocalJobRunner: hdfs://
>> 0.0.0.0:9000/benchmarks/NNBench-aolx-PC/control/NNBench_Controlfile_10:0+125>
map
>> 14/03/17 23:54:29 INFO mapreduce.Job:  map 6% reduce 0%
>> 14/03/17 23:56:15 INFO hdfs.NNBench: Exception recorded in op:
>> Create/Write/Close
>> 14/03/17 23:56:15 INFO hdfs.NNBench: Exception recorded in op:
>> Create/Write/Close
>> 14/03/17 23:56:15 INFO hdfs.NNBench: Exception recorded in op:
>> Create/Write/Close
>> 14/03/17 23:56:15 INFO hdfs.NNBench: Exception recorded in op:
>> Create/Write/Close
>> 14/03/17 23:56:15 INFO hdfs.NNBench: Exception recorded in op:
>> Create/Write/Close
>> 14/03/17 23:56:15 INFO hdfs.NNBench: Exception recorded in op:
>> Create/Write/Close
>> 14/03/17 23:56:15 INFO hdfs.NNBench: Exception recorded in op:
>> Create/Write/Close
>> 14/03/17 23:56:15 INFO hdfs.NNBench: Exception recorded in op:
>> Create/Write/Close
>> (1000 Exceptions)
>> .
>> .
>> .
>> results:
>>
>>  File System Counters
>> FILE: Number of bytes read=18769411
>> FILE: Number of bytes written=21398315
>> FILE: Number of read operations=0
>> FILE: Number of large read operations=0
>> FILE: Number of write operations=0
>> HDFS: Number of bytes read=11185
>> HDFS: Number of bytes written=19540
>> HDFS: Number of read operations=325
>> HDFS: Number of large read operations=0
>> HDFS: Number of write operations=13210
>> Map-Reduce Framework
>> Map input records=12
>> Map output records=95
>> Map output bytes=1829
>> Map output materialized bytes=2091
>> Input split bytes=1538
>> Combine input records=0
>> Combine output records=0
>> Reduce input groups=8
>> Reduce shuffle bytes=0
>> Reduce input records=95
>> Reduce output records=8
>> Spilled Records=214
>> Shuffled Maps =0
>> Failed Shuffles=0
>> Merged Map outputs=0
>> GC time elapsed (ms)=211
>> CPU time spent (ms)=0
>> Physical memory (bytes) snapshot=0
>> Virtual memory (bytes) snapshot=0
>> Total committed heap usage (bytes)=4401004544
>> File Input Format Counters
>> Bytes Read=1490
>> File Output Format Counters
>> Bytes Written=170
>>  14/03/17 23:56:18 INFO hdfs.NNBench: -------------- NNBench
>> -------------- :
>> 14/03/17 23:56:18 INFO hdfs.NNBench:
>>  Version: NameNode Benchmark 0.4
>> 14/03/17 23:56:18 INFO hdfs.NNBench:                            Date &
>> time: 2014-03-17 23:56:18,619
>> 14/03/17 23:56:18 INFO hdfs.NNBench:
>> 14/03/17 23:56:18 INFO hdfs.NNBench:                         Test
>> Operation: create_write
>> 14/03/17 23:56:18 INFO hdfs.NNBench:                             Start
>> time: 2014-03-17 23:56:15,521
>> 14/03/17 23:56:18 INFO hdfs.NNBench:                            Maps to
>> run: 12
>> 14/03/17 23:56:18 INFO hdfs.NNBench:                         Reduces to
>> run: 6
>> 14/03/17 23:56:18 INFO hdfs.NNBench:                     Block Size
>> (bytes): 1
>> 14/03/17 23:56:18 INFO hdfs.NNBench:                         Bytes to
>> write: 0
>> 14/03/17 23:56:18 INFO hdfs.NNBench:                     Bytes per
>> checksum: 1
>> 14/03/17 23:56:18 INFO hdfs.NNBench:                        Number of
>> files: 1000
>> 14/03/17 23:56:18 INFO hdfs.NNBench:                     Replication
>> factor: 3
>> 14/03/17 23:56:18 INFO hdfs.NNBench:             Successful file
>> operations: 0
>> 14/03/17 23:56:18 INFO hdfs.NNBench:
>> 14/03/17 23:56:18 INFO hdfs.NNBench:         # maps that missed the
>> barrier: 11
>> 14/03/17 23:56:18 INFO hdfs.NNBench:                           #
>> exceptions: 1000
>> 14/03/17 23:56:18 INFO hdfs.NNBench:
>> 14/03/17 23:56:18 INFO hdfs.NNBench:                TPS:
>> Create/Write/Close: 0
>> 14/03/17 23:56:18 INFO hdfs.NNBench: Avg exec time (ms):
>> Create/Write/Close: Infinity
>> 14/03/17 23:56:18 INFO hdfs.NNBench:             Avg Lat (ms):
>> Create/Write: NaN
>> 14/03/17 23:56:18 INFO hdfs.NNBench:                    Avg Lat (ms):
>> Close: NaN
>> 14/03/17 23:56:18 INFO hdfs.NNBench:
>> 14/03/17 23:56:18 INFO hdfs.NNBench:                  RAW DATA: AL Total
>> #1: 0
>> 14/03/17 23:56:18 INFO hdfs.NNBench:                  RAW DATA: AL Total
>> #2: 0
>> 14/03/17 23:56:18 INFO hdfs.NNBench:               RAW DATA: TPS Total
>> (ms): 1131
>> 14/03/17 23:56:18 INFO hdfs.NNBench:        RAW DATA: Longest Map Time
>> (ms): 1.395071776653E12
>> 14/03/17 23:56:18 INFO hdfs.NNBench:                    RAW DATA: Late
>> maps: 11
>> 14/03/17 23:56:18 INFO hdfs.NNBench:              RAW DATA: # of
>> exceptions: 1000
>> 14/03/17 23:56:18 INFO hdfs.NNBench:
>>
>>

Mime
View raw message