Return-Path: X-Original-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id E394C10A88 for ; Tue, 18 Mar 2014 05:04:49 +0000 (UTC) Received: (qmail 45071 invoked by uid 500); 18 Mar 2014 05:04:40 -0000 Delivered-To: apmail-hadoop-mapreduce-user-archive@hadoop.apache.org Received: (qmail 44771 invoked by uid 500); 18 Mar 2014 05:04:38 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 44764 invoked by uid 99); 18 Mar 2014 05:04:37 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Mar 2014 05:04:37 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,NORMAL_HTTP_TO_IP,RCVD_IN_DNSWL_LOW,SPF_PASS,WEIRD_PORT X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of aolixiang@gmail.com designates 209.85.219.51 as permitted sender) Received: from [209.85.219.51] (HELO mail-oa0-f51.google.com) (209.85.219.51) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Mar 2014 05:04:33 +0000 Received: by mail-oa0-f51.google.com with SMTP id i4so6771381oah.10 for ; Mon, 17 Mar 2014 22:04:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=WAbiOJqO0gujHOdpZhGLpTFpmNUK1FSQRTaCVT/tgeQ=; b=vcaHQ8vHu3xwc3qHp8haocWN+4xcc4AYKw23K09TxeM58r8bZoXir7xHc+TCKvA+3M ZEOnGXCIXSHJ77exOs7loRYpiZJ4+2kOISQIwYU1nBD6nOwMuxUbHvb+Im43WiTkbs4o YayOsY1UNgTx0CiesbSbZ0Ugu20dJSTw5L+15xaZ/YVwoMNRF0FbT0ZzVkTyvHKncxz6 O9idug0r706nttBJZBlk7knfLLhQL5OnwtZTpy6lsgD4MzlZHLPH0n6yOoZ7ox+FrsKN /vh4jF9BfNW/2SimYFrDb7ED/sT+LpIbuQLlA71QgEBRvzZjXrTB+rT52hvk7RVPRSq/ /xrg== MIME-Version: 1.0 X-Received: by 10.60.116.74 with SMTP id ju10mr23972454oeb.6.1395119052475; Mon, 17 Mar 2014 22:04:12 -0700 (PDT) Received: by 10.60.77.68 with HTTP; Mon, 17 Mar 2014 22:04:12 -0700 (PDT) Received: by 10.60.77.68 with HTTP; Mon, 17 Mar 2014 22:04:12 -0700 (PDT) In-Reply-To: References: Date: Tue, 18 Mar 2014 13:04:12 +0800 Message-ID: Subject: Re: Benchmark Failure From: Lixiang Ao To: user@hadoop.apache.org Content-Type: multipart/alternative; boundary=089e0116150a8ad63a04f4da7703 X-Virus-Checked: Checked by ClamAV on apache.org --089e0116150a8ad63a04f4da7703 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable the version is release 2.2.0 2014=E5=B9=B43=E6=9C=8818=E6=97=A5 =E4=B8=8A=E5=8D=8812:26=E4=BA=8E "Lixian= g Ao" =E5=86=99=E9=81=93=EF=BC=9A > Hi all, > > I'm running jobclient tests(on single node), other tests like TestDFSIO, > mrbench succeed except nnbench. > > I got a lot of Exceptions but without any explanation(see below). > > Could anyone tell me what might went wrong? > > Thanks! > > > 14/03/17 23:54:22 INFO hdfs.NNBench: Waiting in barrier for: 112819 ms > 14/03/17 23:54:23 INFO mapreduce.Job: Job job_local2133868569_0001 runnin= g > in uber mode : false > 14/03/17 23:54:23 INFO mapreduce.Job: map 0% reduce 0% > 14/03/17 23:54:28 INFO mapred.LocalJobRunner: hdfs:// > 0.0.0.0:9000/benchmarks/NNBench-aolx-PC/control/NNBench_Controlfile_10:0+= 125> map > 14/03/17 23:54:29 INFO mapreduce.Job: map 6% reduce 0% > 14/03/17 23:56:15 INFO hdfs.NNBench: Exception recorded in op: > Create/Write/Close > 14/03/17 23:56:15 INFO hdfs.NNBench: Exception recorded in op: > Create/Write/Close > 14/03/17 23:56:15 INFO hdfs.NNBench: Exception recorded in op: > Create/Write/Close > 14/03/17 23:56:15 INFO hdfs.NNBench: Exception recorded in op: > Create/Write/Close > 14/03/17 23:56:15 INFO hdfs.NNBench: Exception recorded in op: > Create/Write/Close > 14/03/17 23:56:15 INFO hdfs.NNBench: Exception recorded in op: > Create/Write/Close > 14/03/17 23:56:15 INFO hdfs.NNBench: Exception recorded in op: > Create/Write/Close > 14/03/17 23:56:15 INFO hdfs.NNBench: Exception recorded in op: > Create/Write/Close > (1000 Exceptions) > . > . > . > results: > > File System Counters > FILE: Number of bytes read=3D18769411 > FILE: Number of bytes written=3D21398315 > FILE: Number of read operations=3D0 > FILE: Number of large read operations=3D0 > FILE: Number of write operations=3D0 > HDFS: Number of bytes read=3D11185 > HDFS: Number of bytes written=3D19540 > HDFS: Number of read operations=3D325 > HDFS: Number of large read operations=3D0 > HDFS: Number of write operations=3D13210 > Map-Reduce Framework > Map input records=3D12 > Map output records=3D95 > Map output bytes=3D1829 > Map output materialized bytes=3D2091 > Input split bytes=3D1538 > Combine input records=3D0 > Combine output records=3D0 > Reduce input groups=3D8 > Reduce shuffle bytes=3D0 > Reduce input records=3D95 > Reduce output records=3D8 > Spilled Records=3D214 > Shuffled Maps =3D0 > Failed Shuffles=3D0 > Merged Map outputs=3D0 > GC time elapsed (ms)=3D211 > CPU time spent (ms)=3D0 > Physical memory (bytes) snapshot=3D0 > Virtual memory (bytes) snapshot=3D0 > Total committed heap usage (bytes)=3D4401004544 > File Input Format Counters > Bytes Read=3D1490 > File Output Format Counters > Bytes Written=3D170 > 14/03/17 23:56:18 INFO hdfs.NNBench: -------------- NNBench -------------= - > : > 14/03/17 23:56:18 INFO hdfs.NNBench: > Version: NameNode Benchmark 0.4 > 14/03/17 23:56:18 INFO hdfs.NNBench: Date & > time: 2014-03-17 23:56:18,619 > 14/03/17 23:56:18 INFO hdfs.NNBench: > 14/03/17 23:56:18 INFO hdfs.NNBench: Test > Operation: create_write > 14/03/17 23:56:18 INFO hdfs.NNBench: Start > time: 2014-03-17 23:56:15,521 > 14/03/17 23:56:18 INFO hdfs.NNBench: Maps to > run: 12 > 14/03/17 23:56:18 INFO hdfs.NNBench: Reduces to > run: 6 > 14/03/17 23:56:18 INFO hdfs.NNBench: Block Size > (bytes): 1 > 14/03/17 23:56:18 INFO hdfs.NNBench: Bytes to > write: 0 > 14/03/17 23:56:18 INFO hdfs.NNBench: Bytes per > checksum: 1 > 14/03/17 23:56:18 INFO hdfs.NNBench: Number of > files: 1000 > 14/03/17 23:56:18 INFO hdfs.NNBench: Replication > factor: 3 > 14/03/17 23:56:18 INFO hdfs.NNBench: Successful file > operations: 0 > 14/03/17 23:56:18 INFO hdfs.NNBench: > 14/03/17 23:56:18 INFO hdfs.NNBench: # maps that missed the > barrier: 11 > 14/03/17 23:56:18 INFO hdfs.NNBench: # > exceptions: 1000 > 14/03/17 23:56:18 INFO hdfs.NNBench: > 14/03/17 23:56:18 INFO hdfs.NNBench: TPS: > Create/Write/Close: 0 > 14/03/17 23:56:18 INFO hdfs.NNBench: Avg exec time (ms): > Create/Write/Close: Infinity > 14/03/17 23:56:18 INFO hdfs.NNBench: Avg Lat (ms): > Create/Write: NaN > 14/03/17 23:56:18 INFO hdfs.NNBench: Avg Lat (ms): > Close: NaN > 14/03/17 23:56:18 INFO hdfs.NNBench: > 14/03/17 23:56:18 INFO hdfs.NNBench: RAW DATA: AL Total > #1: 0 > 14/03/17 23:56:18 INFO hdfs.NNBench: RAW DATA: AL Total > #2: 0 > 14/03/17 23:56:18 INFO hdfs.NNBench: RAW DATA: TPS Total > (ms): 1131 > 14/03/17 23:56:18 INFO hdfs.NNBench: RAW DATA: Longest Map Time > (ms): 1.395071776653E12 > 14/03/17 23:56:18 INFO hdfs.NNBench: RAW DATA: Late > maps: 11 > 14/03/17 23:56:18 INFO hdfs.NNBench: RAW DATA: # of > exceptions: 1000 > 14/03/17 23:56:18 INFO hdfs.NNBench: > > --089e0116150a8ad63a04f4da7703 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable

the version is release 2.2.0

2014=E5=B9=B43=E6=9C=8818=E6=97=A5 =E4=B8=8A=E5= =8D=8812:26=E4=BA=8E "Lixiang Ao" <aolixiang@gmail.com>=E5=86=99=E9=81=93=EF=BC=9A
Hi all,

I'm running jobc= lient=C2=A0tests(on single node), other tests like TestDFSIO, mrbench succe= ed except nnbench.=C2=A0

I got a lot of Exceptions= but without any explanation(see below).

Could anyone tell me what might went wrong?
<= br>
Thanks!


14/03/17 23:5= 4:22 INFO hdfs.NNBench: Waiting in barrier for: 112819 ms
14/03/1= 7 23:54:23 INFO mapreduce.Job: Job job_local2133868569_0001 running in uber= mode : false
14/03/17 23:54:23 INFO mapreduce.Job: =C2=A0map 0% reduce 0%
14/03/17 23:54:28 INFO mapred.LocalJobRunner: hdfs://0.0.0.0:9000/benchmarks/NNBench-aolx-PC/control/NNBench_C= ontrolfile_10:0+125 > map
14/03/17 23:54:29 INFO mapreduce.Job: =C2=A0map 6% reduce 0%
14/03/17 23:56:15 INFO hdfs.NNBench: Exception recorded in op: Create/Writ= e/Close
14/03/17 23:56:15 INFO hdfs.NNBench: Exception recorded i= n op: Create/Write/Close
14/03/17 23:56:15 INFO hdfs.NNBench: Exception recorded in op: Create/= Write/Close
14/03/17 23:56:15 INFO hdfs.NNBench: Exception record= ed in op: Create/Write/Close
14/03/17 23:56:15 INFO hdfs.NNBench:= Exception recorded in op: Create/Write/Close
14/03/17 23:56:15 INFO hdfs.NNBench: Exception recorded in op: Create/= Write/Close
14/03/17 23:56:15 INFO hdfs.NNBench: Exception record= ed in op: Create/Write/Close
14/03/17 23:56:15 INFO hdfs.NNBench:= Exception recorded in op: Create/Write/Close
(1000 Exceptions)
.
.
.
result= s:

= File System Counters
= FILE: Number of bytes read=3D18769411
FILE: Number of bytes wr= itten=3D21398315
FI= LE: Number of read operations=3D0
FILE: Number of large read operations=3D0
FILE: Number of write op= erations=3D0
HDFS: = Number of bytes read=3D11185
HDFS: Number of bytes written=3D19540
HDFS: Number of read ope= rations=3D325
HDFS:= Number of large read operations=3D0
HDFS: Number of write operations=3D13210
Map-Reduce Framework
Map input records=3D12<= /div>
Map output records= =3D95
Map output bytes=3D1829<= /div>
Map output material= ized bytes=3D2091
I= nput split bytes=3D1538
Combine input records=3D= 0
Combine output re= cords=3D0
Reduce in= put groups=3D8
Reduce shuffle bytes=3D0=
Reduce input recor= ds=3D95
Reduce outp= ut records=3D8
Spilled Records=3D214
Shuffled Maps =3D0
Failed Shuffles=3D0
Merged Map outputs=3D0
GC time elapsed (ms)= =3D211
CPU time spe= nt (ms)=3D0
Physical memory (bytes) = snapshot=3D0
Virtua= l memory (bytes) snapshot=3D0
Total committed heap usage (bytes)=3D4401004544
File Input Format Counter= s=C2=A0
Bytes Read= =3D1490
File Output = Format Counters=C2=A0
Bytes Written=3D170
14/03/17 23:56:18 INFO hdfs.NNBench: -------------- NNBenc= h -------------- :=C2=A0
14/03/17 23:56:18 INFO hdfs.NNBench: =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Version: NameNode Benchmark 0.4
14/03/17 23:56:18 INFO hdfs.NNBench: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Date &= ; time: 2014-03-17 23:56:18,619
14/03/17 23:56:18 INFO hdfs.NNBen= ch:=C2=A0
14/03/17 23:56:18 INFO hdfs.NNBench: =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 Test Ope= ration: create_write
14/03/17 23:56:18 INFO hdfs.NNBench: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 Start ti= me: 2014-03-17 23:56:15,521
14/03/17 23:56:18 INFO hdfs.NNBench: = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0Maps to run: 12
14/03/17 23:56:18 INFO hd= fs.NNBench: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 Reduces to run: 6
14/03/17 23:56:18 INFO hdfs.NNBench: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 Block Size (bytes): 1
14/0= 3/17 23:56:18 INFO hdfs.NNBench: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 Bytes to write: 0
14/03= /17 23:56:18 INFO hdfs.NNBench: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 Bytes per checksum: 1
14/03/17 23:56:18 INFO hdfs.NNBench: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Number of files: 1000
14/03/17 23:56:18 INFO hdfs.NNBench: =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 Replication factor: 3
1= 4/03/17 23:56:18 INFO hdfs.NNBench: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 Successful file operations: 0
14/03/17 23:56:18 INFO hdfs.NNBench:=C2=A0
14/03/17 23:56:18= INFO hdfs.NNBench: =C2=A0 =C2=A0 =C2=A0 =C2=A0 # maps that missed the barr= ier: 11
14/03/17 23:56:18 INFO hdfs.NNBench: =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 # ex= ceptions: 1000
14/03/17 23:56:18 INFO hdfs.NNBench:=C2=A0
14/03/17 23:56:18= INFO hdfs.NNBench: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= TPS: Create/Write/Close: 0
14/03/17 23:56:18 INFO hdfs.NNBench: A= vg exec time (ms): Create/Write/Close: Infinity
14/03/17 23:56:18 INFO hdfs.NNBench: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 Avg Lat (ms): Create/Write: NaN
14/03/17 23:56:18 INFO= hdfs.NNBench: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0Avg Lat (ms): Close: NaN
14/03/17 23:56:18 INFO hdfs.NN= Bench:=C2=A0
14/03/17 23:56:18 INFO hdfs.NNBench: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0RAW DATA: AL Total #1: 0
14/03/17 = 23:56:18 INFO hdfs.NNBench: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0RAW DATA: AL Total #2: 0
14/03/17 23:56:18 INFO = hdfs.NNBench: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 RAW DATA: TP= S Total (ms): 1131
14/03/17 23:56:18 INFO hdfs.NNBench: =C2=A0 =C2=A0 =C2=A0 =C2=A0RAW DA= TA: Longest Map Time (ms): 1.395071776653E12
14/03/17 23:56:18 IN= FO hdfs.NNBench: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0RAW DATA: Late maps: 11
14/03/17 23:56:18 INFO hdfs.= NNBench: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0RAW DATA: # of exc= eptions: 1000
14/03/17 23:56:18 INFO hdfs.NNBench:=C2=A0

<= /div>
--089e0116150a8ad63a04f4da7703--