hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shi Yu <sh...@uchicago.edu>
Subject DistributedCache in NewAPI on 0.20.X branch
Date Thu, 15 Dec 2011 22:39:15 GMT
Hi,

I am using 0.20.X branch.  However, I need to use the new API because it 
has the cleanup(context) method in Mapper.  However, I am confused about 
how to load the cached files in mapper.   I could load the 
DistributedCache files using old API (JobConf),  but in new API it 
always returns null.   I read some previous discussions that "on 0.20.X 
branch, calling DistributedCache using old API is encouraged."  My 
question is:  Is it possible to use DistributedCache using new API, or 
the only possible way is to upgrade to higher version?    Or I use a 
wrong method in my code?  Thanks!


The relevant code is something as follows:

class Map extends Mapper <K1, V1, K2, V2>{

public void setup(Mapper<K1,V1, K2, V2>.Context context) throws IOException{
   try{
    Path[] localFiles;
    localFiles = 
DistributedCache.getLocalCacheFiles(context.getConfiguration());
    String file1 = localFiles[0].toUri.getPath();   // return null?
     ....

   }


}

}

public int run(String[] args) throws IOException, URISyntaxException{
    ...
   //set the configuration

   Job job = new Job(conf,"Myjob");
   DistributeCache.addCacheFile(new URI(hdfs://server:9000/file1), 
job.getConfiguration);

}

BR,


Shi


Mime
View raw message