Return-Path: 
 <common-user-return-29894-apmail-hadoop-common-user-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-user-archive@www.apache.org
Delivered-To: apmail-hadoop-common-user-archive@www.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C3FB483DE
	for <apmail-hadoop-common-user-archive@www.apache.org>;
 Fri,  2 Sep 2011 00:27:59 +0000 (UTC)
Received: (qmail 43193 invoked by uid 500); 2 Sep 2011 00:27:56 -0000
Delivered-To: apmail-hadoop-common-user-archive@hadoop.apache.org
Received: (qmail 43126 invoked by uid 500); 2 Sep 2011 00:27:55 -0000
Mailing-List: contact common-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-user-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-user-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-user@hadoop.apache.org>
List-Id: <common-user.hadoop.apache.org>
Reply-To: common-user@hadoop.apache.org
Delivered-To: mailing list common-user@hadoop.apache.org
Received: (qmail 43118 invoked by uid 99); 2 Sep 2011 00:27:55 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Sep 2011 00:27:55 +0000
X-ASF-Spam-Status: No, hits=0.7 required=5.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.11.129.193] (HELO mta00.prxy.net) (209.11.129.193)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Sep 2011 00:27:47 +0000
Received: from localhost (localhost.localdomain [127.0.0.1])
	by mta00.prxy.net (Postfix) with ESMTP id DC8CFC8A64
	for <common-user@hadoop.apache.org>; Thu,  1 Sep 2011 17:26:22 -0700 (PDT)
X-Virus-Scanned: amavisd-new at mta00.prxy.net
Received: from mta00.prxy.net ([127.0.0.1])
	by localhost (mta00.prxy.net [127.0.0.1]) (amavisd-new, port 10024)
	with ESMTP id aEoJck859YwN for <common-user@hadoop.apache.org>;
	Thu,  1 Sep 2011 17:26:22 -0700 (PDT)
Received: from [192.168.1.107] (c-76-20-56-43.hsd1.ca.comcast.net
 [76.20.56.43])
	by mta00.prxy.net (Postfix) with ESMTPSA id BA8BEC8A62
	for <common-user@hadoop.apache.org>; Thu,  1 Sep 2011 17:26:22 -0700 (PDT)
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1084)
Subject: Re: TestDFSIO failure
From: Ken Krugler <kkrugler_lists@transpac.com>
In-Reply-To: 
 <E236CD55EC618B4884CA7273B31AA07806A17A91@stlwexmbxprd04.na.ds.monsanto.com>
Date: Thu, 1 Sep 2011 17:27:25 -0700
Content-Transfer-Encoding: quoted-printable
Message-Id: <7AF72C75-8A0D-4A37-9BC8-930A52C724D8@transpac.com>
References: 
 <E236CD55EC618B4884CA7273B31AA07806A17A91@stlwexmbxprd04.na.ds.monsanto.com>
To: common-user@hadoop.apache.org
X-Mailer: Apple Mail (2.1084)
X-Virus-Checked: Checked by ClamAV on apache.org

Hi Matt,

On Jun 20, 2011, at 1:46pm, GOEKE, MATTHEW (AG/1000) wrote:

> Has anyone else run into issues using output compression (in our case =
lzo) on TestDFSIO and it failing to be able to read the metrics file? I =
just assumed that it would use the correct decompression codec after it =
finishes but it always returns with a 'File not found' exception.

Yes, I've run into the same issue on 0.20.2 and CHD3u0

I don't see any Jira issue that covers this problem, so unless I hear =
otherwise I'll file one.

The problem is that the post-job code doesn't handle getting the =
<path>.deflate or <path>.lzo (for you) file from HDFS, and then =
decompressing it.

> Is there a simple way around this without spending the time to =
recompile a cluster/codec specific version?


You can use "hadoop fs -text <path reported in exception>.lzo"

This will dump out the file, which looks like:

f:rate  171455.11
f:sqrate        2981174.8
l:size  10485760000
l:tasks 10
l:time  590537

If you take f:rate/1000/l:tasks, that should give you the average =
MB/sec.

E.g. for the example above, that would be 171455/1000/10 =3D 17MB/sec.

-- Ken

--------------------------
Ken Krugler
+1 530-210-6378
http://bixolabs.com
custom big data solutions & training
Hadoop, Cascading, Mahout & Solr