Return-Path: 
 <user-return-5807-apmail-hadoop-common-user-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-user-archive@www.apache.org
Delivered-To: apmail-hadoop-common-user-archive@www.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id E755CEB0E
	for <apmail-hadoop-common-user-archive@www.apache.org>;
 Thu,  7 Mar 2013 03:58:35 +0000 (UTC)
Received: (qmail 11550 invoked by uid 500); 7 Mar 2013 03:58:30 -0000
Delivered-To: apmail-hadoop-common-user-archive@hadoop.apache.org
Received: (qmail 11073 invoked by uid 500); 7 Mar 2013 03:58:30 -0000
Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:user-help@hadoop.apache.org>
List-Unsubscribe: <mailto:user-unsubscribe@hadoop.apache.org>
List-Post: <mailto:user@hadoop.apache.org>
List-Id: <user.hadoop.apache.org>
Reply-To: user@hadoop.apache.org
Delivered-To: mailing list user@hadoop.apache.org
Received: (qmail 11026 invoked by uid 99); 7 Mar 2013 03:58:28 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 07 Mar 2013 03:58:28 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,UNPARSEABLE_RELAY
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: local policy)
Received: from [195.245.230.174] (HELO mail1.bemta3.messagelabs.com)
 (195.245.230.174)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 07 Mar 2013 03:58:20 +0000
Received: from [195.245.230.115:46467] by server-14.bemta-3.messagelabs.com id
 88/E5-27076-64018315; Thu, 07 Mar 2013 03:57:58 +0000
X-Env-Sender: ivanovd@rsc.org
X-Msg-Ref: server-8.tower-57.messagelabs.com!1362628677!26810424!1
X-Originating-IP: [135.196.236.26]
X-StarScan-Received: 
X-StarScan-Version: 6.8.6.1; banners=rsc.org,-,-
X-VirusChecked: Checked
Received: (qmail 28490 invoked from network); 7 Mar 2013 03:57:58 -0000
Received: from unknown (HELO COBLE.rsc-network.org) (135.196.236.26)
  by server-8.tower-57.messagelabs.com with AES128-SHA encrypted SMTP;
 7 Mar 2013 03:57:58 -0000
Received: from TRAWLER.rsc-network.org ([fe80::c136:da7a:952f:c8d6]) by
 COBLE.rsc-network.org ([fe80::e8f9:d309:8350:6593%11]) with mapi id
 14.02.0309.002; Thu, 7 Mar 2013 03:57:57 +0000
From: Dmitriy Ivanov <ivanovd@rsc.org>
To: "user@hadoop.apache.org" <user@hadoop.apache.org>
Subject: Reading partitioned sequence file from hdfs throws
 filenotfoundexception
Thread-Topic: Reading partitioned sequence file from hdfs throws
 filenotfoundexception
Thread-Index: Ac4a6ACLX6yr515gSHG19sbUg+wXtA==
Date: Thu, 7 Mar 2013 03:57:57 +0000
Message-ID: <7FAD2B761D88324AB0B720A68656C34110EA5070@Trawler.rsc-network.org>
Accept-Language: en-GB, en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
x-originating-ip: [71.178.177.98]
Content-Type: multipart/alternative;
	boundary="_000_7FAD2B761D88324AB0B720A68656C34110EA5070Trawlerrscnetwo_"
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

--_000_7FAD2B761D88324AB0B720A68656C34110EA5070Trawlerrscnetwo_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

Hello,

I'm using hadoop 1.1.1 and run into unexpected complication with partition=
ed file. The file itself is the result of map-reduce task.

Here is code I'm using to read the file:

        try (SequenceFile.Reader reader =3D new SequenceFile.Reader(fs, pa=
th, conf)) {
                // skipped code.
        }

This is exception:

java.io.FileNotFoundException: File does not exist: /users/ivanovd/1.624be=
3e5-5932-468d-9ce4-f73078836936.cvsp
        at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.fetchLocatedBlo=
cks(DFSClient.java:1975)
        at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.openInfo(DFSCli=
ent.java:1944)
        at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.<init>(DFSClien=
t.java:1936)
        at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:731)
        at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFi=
leSystem.java:165)
        at org.apache.hadoop.io.SequenceFile$Reader.openFile(SequenceFile.=
java:1499)
        at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.ja=
va:1486)
        at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.ja=
va:1479)
        at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.ja=
va:1474)
        at HadoopTask.exportResults(HadoopTask.java:163)

The file itself exists (in partitioned form):
./hadoop fs -ls /users/ivanovd/1.2a8b1a9c-47de-4631-8013-f0dd3e096036.cvsp=
/
-rw-r--r--   3 ivanovd supergroup          0 2013-03-06 22:17 /users/ivano=
vd/1.2a8b1a9c-47de-4631-8013-f0dd3e096036.cvsp/_SUCCESS
drwxr-xr-x   - ivanovd supergroup          0 2013-03-06 22:16 /users/ivano=
vd/1.2a8b1a9c-47de-4631-8013-f0dd3e096036.cvsp/_logs
-rw-r--r--   3 ivanovd supergroup      63301 2013-03-06 22:17 /users/ivano=
vd/1.2a8b1a9c-47de-4631-8013-f0dd3e096036.cvsp/part-r-00000

Also hadoop fs -getmerge works fine.

Did anyone encounter this problem with hdfs SequenceFile.Reader? What am I=
 doing wrong?

Thanks,
/DI


DISCLAIMER:

This communication (including any attachments) is intended for the use of =
the addressee only and may contain confidential, privileged or copyright m=
aterial. It may not be relied upon or disclosed to any other person withou=
t the consent of the RSC. If you have received it in error, please contact=
 us immediately. Any advice given by the RSC has been carefully formulated=
 but is necessarily based on the information available, and the RSC cannot=
 be held responsible for accuracy or completeness. In this respect, the RS=
C owes no duty of care and shall not be liable for any resulting damage or=
 loss. The RSC acknowledges that a disclaimer cannot restrict liability at=
 law for personal injury or death arising through a finding of negligence.=
 The RSC does not warrant that its emails or attachments are Virus-free: P=
lease rely on your own screening. The Royal Society of Chemistry is a char=
ity, registered in England and Wales, number 207890 - Registered office: T=
homas Graham House, Science Park, Milton Road, Cambridge CB4 0WF
--_000_7FAD2B761D88324AB0B720A68656C34110EA5070Trawlerrscnetwo_
Content-Type: text/html; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

<html xmlns:v=3D"urn:schemas-microsoft-com:vml" xmlns:o=3D"urn:schemas-mic=
rosoft-com:office:office" xmlns:w=3D"urn:schemas-microsoft-com:office:word=
" xmlns:x=3D"urn:schemas-microsoft-com:office:excel" xmlns:m=3D"http://sch=
emas.microsoft.com/office/2004/12/omml" xmlns=3D"http://www.w3.org/TR/REC-=
html40">
<head>
<meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Dus-ascii=
">
<meta name=3D"Generator" content=3D"Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
=09{font-family:"Cambria Math";
=09panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
=09{font-family:Calibri;
=09panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
=09{margin:0in;
=09margin-bottom:.0001pt;
=09font-size:11.0pt;
=09font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
=09{mso-style-priority:99;
=09color:blue;
=09text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
=09{mso-style-priority:99;
=09color:purple;
=09text-decoration:underline;}
span.EmailStyle17
=09{mso-style-type:personal-compose;
=09font-family:"Calibri","sans-serif";
=09color:windowtext;}
.MsoChpDefault
=09{mso-style-type:export-only;
=09font-family:"Calibri","sans-serif";}
@page WordSection1
=09{size:8.5in 11.0in;
=09margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
=09{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext=3D"edit" spidmax=3D"1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext=3D"edit">
<o:idmap v:ext=3D"edit" data=3D"1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang=3D"EN-US" link=3D"blue" vlink=3D"purple">
<div class=3D"WordSection1">
<p class=3D"MsoNormal">Hello,<o:p></o:p></p>
<p class=3D"MsoNormal"><o:p>&nbsp;</o:p></p>
<p class=3D"MsoNormal">I&#8217;m using hadoop 1.1.1 and run into unexpecte=
d complication with partitioned file. The file itself is the result of map=
-reduce task.<o:p></o:p></p>
<p class=3D"MsoNormal"><o:p>&nbsp;</o:p></p>
<p class=3D"MsoNormal">Here is code I&#8217;m using to read the file:<o:p>=
</o:p></p>
<p class=3D"MsoNormal"><o:p>&nbsp;</o:p></p>
<p class=3D"MsoNormal">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; try (Seq=
uenceFile.Reader reader =3D new SequenceFile.Reader(fs, path, conf)) {<o:p=
></o:p></p>
<p class=3D"MsoNormal">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nb=
sp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; // skipped code.<o:p></o:p></p>
<p class=3D"MsoNormal">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; }<o:p></=
o:p></p>
<p class=3D"MsoNormal"><o:p>&nbsp;</o:p></p>
<p class=3D"MsoNormal">This is exception:<o:p></o:p></p>
<p class=3D"MsoNormal"><o:p>&nbsp;</o:p></p>
<p class=3D"MsoNormal">java.io.FileNotFoundException: File does not exist:=
 /users/ivanovd/1.624be3e5-5932-468d-9ce4-f73078836936.cvsp<o:p></o:p></p>=

<p class=3D"MsoNormal">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at org.a=
pache.hadoop.hdfs.DFSClient$DFSInputStream.fetchLocatedBlocks(DFSClient.ja=
va:1975)<o:p></o:p></p>
<p class=3D"MsoNormal">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at org.a=
pache.hadoop.hdfs.DFSClient$DFSInputStream.openInfo(DFSClient.java:1944)<o=
:p></o:p></p>
<p class=3D"MsoNormal">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at org.a=
pache.hadoop.hdfs.DFSClient$DFSInputStream.&lt;init&gt;(DFSClient.java:193=
6)<o:p></o:p></p>
<p class=3D"MsoNormal">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at org.a=
pache.hadoop.hdfs.DFSClient.open(DFSClient.java:731)<o:p></o:p></p>
<p class=3D"MsoNormal">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at org.a=
pache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:16=
5)<o:p></o:p></p>
<p class=3D"MsoNormal">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at org.a=
pache.hadoop.io.SequenceFile$Reader.openFile(SequenceFile.java:1499)<o:p><=
/o:p></p>
<p class=3D"MsoNormal">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at org.a=
pache.hadoop.io.SequenceFile$Reader.&lt;init&gt;(SequenceFile.java:1486)<o=
:p></o:p></p>
<p class=3D"MsoNormal">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at org.a=
pache.hadoop.io.SequenceFile$Reader.&lt;init&gt;(SequenceFile.java:1479)<o=
:p></o:p></p>
<p class=3D"MsoNormal">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at org.a=
pache.hadoop.io.SequenceFile$Reader.&lt;init&gt;(SequenceFile.java:1474)<o=
:p></o:p></p>
<p class=3D"MsoNormal">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; &nbsp;at Hadoo=
pTask.exportResults(HadoopTask.java:163)<o:p></o:p></p>
<p class=3D"MsoNormal"><o:p>&nbsp;</o:p></p>
<p class=3D"MsoNormal">The file itself exists (in partitioned form):<o:p><=
/o:p></p>
<p class=3D"MsoNormal">./hadoop fs -ls /users/ivanovd/1.2a8b1a9c-47de-4631=
-8013-f0dd3e096036.cvsp/<o:p></o:p></p>
<p class=3D"MsoNormal">-rw-r--r--&nbsp;&nbsp; 3 ivanovd supergroup&nbsp;&n=
bsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 0 2013-03-06 22:17 /users/i=
vanovd/1.2a8b1a9c-47de-4631-8013-f0dd3e096036.cvsp/_SUCCESS<o:p></o:p></p>=

<p class=3D"MsoNormal">drwxr-xr-x&nbsp;&nbsp; - ivanovd supergroup&nbsp;&n=
bsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 0 2013-03-06 22:16 /users/i=
vanovd/1.2a8b1a9c-47de-4631-8013-f0dd3e096036.cvsp/_logs<o:p></o:p></p>
<p class=3D"MsoNormal">-rw-r--r--&nbsp;&nbsp; 3 ivanovd supergroup&nbsp;&n=
bsp;&nbsp;&nbsp;&nbsp; 63301 2013-03-06 22:17 /users/ivanovd/1.2a8b1a9c-47=
de-4631-8013-f0dd3e096036.cvsp/part-r-00000<o:p></o:p></p>
<p class=3D"MsoNormal"><o:p>&nbsp;</o:p></p>
<p class=3D"MsoNormal">Also hadoop fs -getmerge works fine.<o:p></o:p></p>=

<p class=3D"MsoNormal"><o:p>&nbsp;</o:p></p>
<p class=3D"MsoNormal">Did anyone encounter this problem with hdfs Sequenc=
eFile.Reader? What am I doing wrong?<o:p></o:p></p>
<p class=3D"MsoNormal"><o:p>&nbsp;</o:p></p>
<p class=3D"MsoNormal">Thanks,<o:p></o:p></p>
<p class=3D"MsoNormal">/DI<o:p></o:p></p>
<p class=3D"MsoNormal"><o:p>&nbsp;</o:p></p>
</div>
<br clear=3D"both">
DISCLAIMER:<BR>
<BR>
This communication (including any attachments) is intended for the use of =
the addressee only and may contain confidential, privileged or copyright m=
aterial. It may not be relied upon or disclosed to any other person withou=
t the consent of the RSC. If you have received it in error, please contact=
 us immediately. Any advice given by the RSC has been carefully formulated=
 but is necessarily based on the information available, and the RSC cannot=
 be held responsible for accuracy or completeness. In this respect, the RS=
C owes no duty of care and shall not be liable for any resulting damage or=
 loss. The RSC acknowledges that a disclaimer cannot restrict liability at=
 law for personal injury or death arising through a finding of negligence.=
 The RSC does not warrant that its emails or attachments are Virus-free: P=
lease rely on your own screening. The Royal Society of Chemistry is a char=
ity, registered in England and Wales, number 207890 - Registered office: T=
homas Graham House, Science Park, Milton Road, Cambridge CB4 0WF<BR>
</body>
</html>

--_000_7FAD2B761D88324AB0B720A68656C34110EA5070Trawlerrscnetwo_--