hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Abdelrhman Shettia <ashet...@hortonworks.com>
Subject Re: Reading partitioned sequence file from hdfs throws filenotfoundexception
Date Thu, 07 Mar 2013 05:56:38 GMT
Hi All , 

Try to give the full path for the file such as : 

> /users/ivanovd/1.2a8b1a9c-47de-4631-8013-f0dd3e096036.cvsp/part-r-00000

if the job is producing lots of files and there is a need to setup the number of mappers more
than one. A file crusher utility may be the best option here to implement between 1st M-R
job and 2nd M-R job. 

- Abdelrahman 

On Mar 6, 2013, at 7:57 PM, Dmitriy Ivanov <ivanovd@rsc.org> wrote:

> Hello,
> I’m using hadoop 1.1.1 and run into unexpected complication with partitioned file.
The file itself is the result of map-reduce task.
> Here is code I’m using to read the file:
>         try (SequenceFile.Reader reader = new SequenceFile.Reader(fs, path, conf)) {
>                 // skipped code.
>         }
> This is exception:
> java.io.FileNotFoundException: File does not exist: /users/ivanovd/1.624be3e5-5932-468d-9ce4-f73078836936.cvsp
>         at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.fetchLocatedBlocks(DFSClient.java:1975)
>         at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.openInfo(DFSClient.java:1944)
>         at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.<init>(DFSClient.java:1936)
>         at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:731)
>         at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:165)
>         at org.apache.hadoop.io.SequenceFile$Reader.openFile(SequenceFile.java:1499)
>         at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1486)
>         at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1479)
>         at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1474)
>         at HadoopTask.exportResults(HadoopTask.java:163)
> The file itself exists (in partitioned form):
> ./hadoop fs -ls /users/ivanovd/1.2a8b1a9c-47de-4631-8013-f0dd3e096036.cvsp/
> -rw-r--r--   3 ivanovd supergroup          0 2013-03-06 22:17 /users/ivanovd/1.2a8b1a9c-47de-4631-8013-f0dd3e096036.cvsp/_SUCCESS
> drwxr-xr-x   - ivanovd supergroup          0 2013-03-06 22:16 /users/ivanovd/1.2a8b1a9c-47de-4631-8013-f0dd3e096036.cvsp/_logs
> -rw-r--r--   3 ivanovd supergroup      63301 2013-03-06 22:17 /users/ivanovd/1.2a8b1a9c-47de-4631-8013-f0dd3e096036.cvsp/part-r-00000
> Also hadoop fs -getmerge works fine.
> Did anyone encounter this problem with hdfs SequenceFile.Reader? What am I doing wrong?
> Thanks,
> /DI
> This communication (including any attachments) is intended for the use of the addressee
only and may contain confidential, privileged or copyright material. It may not be relied
upon or disclosed to any other person without the consent of the RSC. If you have received
it in error, please contact us immediately. Any advice given by the RSC has been carefully
formulated but is necessarily based on the information available, and the RSC cannot be held
responsible for accuracy or completeness. In this respect, the RSC owes no duty of care and
shall not be liable for any resulting damage or loss. The RSC acknowledges that a disclaimer
cannot restrict liability at law for personal injury or death arising through a finding of
negligence. The RSC does not warrant that its emails or attachments are Virus-free: Please
rely on your own screening. The Royal Society of Chemistry is a charity, registered in England
and Wales, number 207890 - Registered office: Thomas Graham House, Science Park, Milton Road,
Cambridge CB4 0WF

View raw message