hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sean Shanny <ssha...@tripadvisor.com>
Subject Having trouble accessing MapFiles in the DistributedCache
Date Thu, 25 Dec 2008 05:10:21 GMT
To all,

Version:  hadoop-0.17.2.1-core.jar

I created a MapFile on a local node.

I  put the files into the HDFS using the following commands:

$ bin/hadoop fs -copyFromLocal /tmp/ur/data    /2008-12-19/url/data
$ bin/hadoop fs -copyFromLocal /tmp/ur/index  /2008-12-19/url/index

and placed them in the DistributedCache using the following calls in  
the JobConf class:

DistributedCache.addCacheFile(new URI("/2008-12-19/url/data"), conf);
DistributedCache.addCacheFile(new URI("/2008-12-19/url/index"), conf);

What I cannot figure out how to do is actually access the MapFile now  
within my Map code.  I tried the following but I am getting file not  
found errors when I try to run the job.

private FileSystem              fs;
private MapFile.Reader     myReader;
private Path[]                        localFiles;

....

  public void configure(JobConf conf)
     {
         String[] s = conf.getStrings("map.input.file");
         m_sFileName = s[0];

        try
         {
             localFiles = DistributedCache.getLocalCacheFiles(conf);

             for (Path localFile : localFiles)
             {
                 String sFileName = localFile.getName();

                 if (sFileName.equalsIgnoreCase("data"))
                 {
                     System.out.println("Full Path: " +  
localFile.toString());
                     System.out.println("Parent: " +  
localFile.getParent().toString());

                     fs = FileSystem.get(localFile.toUri(), conf);
                     myReader = new MapFile.Reader(fs,  
localFile.getParent().toString(), conf);
                 }
             }
         }
         catch (IOException e)
         {
             // TODO Auto-generated catch block
             e.printStackTrace();
         }

The following exception is thrown and I cannot figure out why it is  
adding the extra data element at the end of the path.  The data is  
actually at

Task Logs: 'task_200812250002_0001_m_000000_0'

stdout logs
Full Path: /tmp/hadoop-root/mapred/local/taskTracker/archive/hdp01n/ 
2008-12-19/url/data/data
Parent: /tmp/hadoop-root/mapred/local/taskTracker/archive/hdp01n/ 
2008-12-19/url/data
stderr logs
java.io.FileNotFoundException: File does not exist: /tmp/hadoop-root/ 
mapred/local/taskTracker/archive/hdp01n/2008-12-19/url/data/data at  
org 
.apache 
.hadoop 
.dfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java: 
369) at org.apache.hadoop.fs.FileSystem.getLength(FileSystem.java:628)  
at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java: 
1431) at org.apache.hadoop.io.SequenceFile 
$Reader.<init>(SequenceFile.java:1426) at org.apache.hadoop.io.MapFile 
$Reader.createDataFileReader(MapFile.java:301) at  
org.apache.hadoop.io.MapFile$Reader.open(MapFile.java:283) at  
org.apache.hadoop.io.MapFile$Reader.<init>(MapFile.java:272) at  
org.apache.hadoop.io.MapFile$Reader.<init>(MapFile.java:259) at  
org.apache.hadoop.io.MapFile$Reader.<init>(MapFile.java:252) at  
com 
.TripResearch 
.warehouse.etl.EtlTestUrlMapLookup.configure(EtlTestUrlMapLookup.java: 
84) at  
org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java: 
58) at  
org 
.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java: 
82) at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:33)  
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java: 
58) at  
org 
.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java: 
82) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:215) at  
org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2122)

The files do exist but I don't understand why they were placed in  
their own directories.  I would have expected both files to exist at / 
2008-12-19/url/ not /2008-12-19/url/data/ and /2008-12-19/url/index/

ls -la /tmp/hadoop-root/mapred/local/taskTracker/archive/hdp01n/ 
2008-12-19/url/data
total 740640
drwxr-xr-x 2 root root 4096 Dec 24 23:49 .
drwxr-xr-x 4 root root 4096 Dec 24 23:49 ..
-rwxr-xr-x 1 root root 751776245 Dec 24 23:49 data
-rw-r--r-- 1 root root 5873260 Dec 24 23:49 .data.crc	

[root@hdp01n warehouse]# ls -la /tmp/hadoop-root/mapred/local/ 
taskTracker/archive/hdp01n/2008-12-19/url/index
total 2148
drwxr-xr-x 2 root root    4096 Dec 25 00:04 .
drwxr-xr-x 4 root root    4096 Dec 25 00:04 ..
-rwxr-xr-x 1 root root 2165220 Dec 25 00:04 index
-rw-r--r-- 1 root root   16924 Dec 25 00:04 .index.crc

....

I know I must be doing something really stupid here as I am sure this  
has been done by lots of folks prior to my feeble attempt.  I did a  
google search but really could not come up with any examples of using  
a MapFile on the DistributedCache.

Thanks.

--sean





Mime
View raw message