hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From JOAQUIN GUANTER GONZALBEZ <x...@tid.es>
Subject Reading a directory in standalone Hadoop
Date Fri, 31 Aug 2012 08:11:16 GMT
Hello Hadoopers,

I am trying to write a test that runs some MR jobs. One of these jobs needs to read a file
produced by another job with a SequenceFile.Reader class. The job that produces this file
has its output path set to "/folder/timezone". After running in my standalone Hadoop environment,
it produces 4 files in this folder:


-          .SUCCESS.crc

-          .part-r-00000.crc

-          _SUCCESS

-          part-r-00000

So far, so good. The problem is that when the next job tries to read "/folder/timezone" with
a SequenceFile.Reader class, it gets a FileNotFoundException, presumably because it is trying
to open the directory as a "file" in my local filesystem. Here's the stacktrace I get:

java.io.FileNotFoundException: /folder/timezone (Acess denied)
                at java.io.FileInputStream.open(Native Method)
                at java.io.FileInputStream.&lt;init&gt;(FileInputStream.java:138)
                at org.apache.hadoop.fs.RawLocalFileSystem$TrackingFileInputStream.&lt;init&gt;(RawLocalFileSystem.java:72)
                at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileInputStream.&lt;init&gt;(RawLocalFileSystem.java:108)
                at org.apache.hadoop.fs.RawLocalFileSystem.open(RawLocalFileSystem.java:178)
                at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.&lt;init&gt;(ChecksumFileSystem.java:127)
                at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:284)
                at org.apache.hadoop.io.SequenceFile$Reader.openFile(SequenceFile.java:1508)
                at org.apache.hadoop.io.SequenceFile$Reader.&lt;init&gt;(SequenceFile.java:1487)
                at org.apache.hadoop.io.SequenceFile$Reader.&lt;init&gt;(SequenceFile.java:1480)
                at org.apache.hadoop.io.SequenceFile$Reader.&lt;init&gt;(SequenceFile.java:1475)
                at es.tid.smartsteps.footfalls.lookups.CartesianConverterMapperBase.setup(CartesianConverterMapperBase.java:41)
                at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:142)
                at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:647)
                at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323)
                at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:210)

Any idea on how to solve this issue? Is this a supported scenario by SequenceFile.Reader or
am I out of luck?

Many thanks!
Ximo.

________________________________
Este mensaje se dirige exclusivamente a su destinatario. Puede consultar nuestra pol?tica
de env?o y recepci?n de correo electr?nico en el enlace situado m?s abajo.
This message is intended exclusively for its addressee. We only send and receive email on
the basis of the terms set out at:
http://www.tid.es/ES/PAGINAS/disclaimer.aspx

Mime
View raw message