hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rakhi Khatwani <rakhi.khatw...@gmail.com>
Subject Using chainmapper and chain reducer in hadoop
Date Fri, 17 Jul 2009 13:06:49 GMT
Hi
        I am trying out a simple example in which you have two maps and one
reduce, Map1 and Map2 and Reduce1

now i wanna execute my job in the following fashion

Map1 -> Map2 -> Reduce1 (o/p of map1 goes into map2 and o/p of map2 goes
into reduce1)


i have declared my conf as follows:


JobConf conf1 = new JobConf(ChainingJobsMain.class);
        conf1.setJobName("Job1");

        JobConf mapper1Conf = new JobConf();
        JobConf mapper2Conf = new JobConf();
        JobConf reducerConf = new JobConf();

ChainMapper chainMapper = new ChainMapper();
        ChainReducer chainReducer = new ChainReducer();
        chainMapper.addMapper(conf1, Map1.class, LongWritable.class,
Text.class, Text.class, Text.class, false, mapper1Conf);
        chainMapper.addMapper(conf1, Map2.class, Text.class, Text.class,
Text.class, Text.class, false, mapper2Conf);
        chainReducer.setReducer(conf1, Reduce1.class, LongWritable.class,
Text.class, Text.class, Text.class, false, reducerConf);

    //Input and output format
        conf1.setInputFormat(TextInputFormat.class);
        conf1.setOutputFormat(NullOutputFormat.class);
        FileInputFormat.setInputPaths(conf1, "/user/root/input");


        try {
            JobClient.runJob(conf1);
        } catch (IOException e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        }


but i get an output of a single map reduce. (i.e. map1 and reduce1.... i
dont see any map2 code executing).

Has anyone tried this out? and am i going wrong anywhere?? or does anyone
have a simple example which runs chainmapper???

Regards,
Raakhi

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message