hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michel Segel <michael_se...@hotmail.com>
Subject Re: Issue with DistributedCache
Date Thu, 24 Nov 2011 13:13:58 GMT
Silly question... Why do you need to use the distributed cache for the word count program?
 What are you trying to accomplish?

I've only had to play with it for one project where we had to push out a bunch of c++ code
to the nodes as part of a job...

Sent from a remote device. Please excuse any typos...

Mike Segel

On Nov 24, 2011, at 7:05 AM, Denis Kreis <de.kreis@gmail.com> wrote:

> Hi Bejoy
> 
> 1. Old API:
> The Map and Reduce classes are the same as in the example, the main
> method is as follows
> 
> public static void main(String[] args) throws IOException,
> InterruptedException {
>        UserGroupInformation ugi =
> UserGroupInformation.createProxyUser("<remote user name>",
> UserGroupInformation.getLoginUser());
>        ugi.doAs(new PrivilegedExceptionAction<Void>() {
>            public Void run() throws Exception {
>                
>                JobConf conf = new JobConf(WordCount.class);
>                conf.setJobName("wordcount");
>                
>                conf.setOutputKeyClass(Text.class);
>                conf.setOutputValueClass(IntWritable.class);
>                
>                conf.setMapperClass(Map.class);
>                conf.setCombinerClass(Reduce.class);
>                conf.setReducerClass(Reduce.class);
>            
>                conf.setInputFormat(TextInputFormat.class);
>                conf.setOutputFormat(TextOutputFormat.class);
> 
>                FileInputFormat.setInputPaths(conf, new Path("<path to input dir>"));
>                FileOutputFormat.setOutputPath(conf, new Path("<path to
> output dir>"));
>                
>                conf.set("mapred.job.tracker", "<ip:8021>");
>                
>                FileSystem fs = FileSystem.get(new URI("hdfs://<ip>:8020"),
> new Configuration());
>                fs.mkdirs(new Path("<remote path>"));
>                fs.copyFromLocalFile(new Path("<local path>/test.jar"), new
> Path("<remote path>"));
>                
> 

Mime
View raw message