hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Xiance SI(司宪策) <adam...@gmail.com>
Subject Re: Confused by new API & MultipleOutputFormats using Hadoop 0.20.1
Date Sat, 07 Nov 2009 14:45:26 GMT
I just fall back to old mapred.* APIs, seems MultipleOutputs only works for
the old API.

wishes,
Xiance

On Mon, Nov 2, 2009 at 9:12 AM, Paul Smith <psmith@aconex.com> wrote:

> Totally stuck here, I can't seem to find a way to resolve this, but I can't
> use the new API _and_ use the MultipleOutputFormats class.
>
> I found this thread which is related, but doesn't seem to help me (or I
> missed something completely, certainly possible):
>
>
> http://markmail.org/message/u4wz5nbcn5rawydq#query:hadoop%20MultipleTextOutputFormat%20OutputFormat%20Job%20JobConf+page:1+mid:5wy63oqa2vs6bj7b+state:results
>
> My controller Job class is simple, but I get a compile error trying to add
> the new MultipleOutputs:
>
> public class ControllerMetricGrinder {
>
>    public static class MetricNameMultipleTextOutputFormat extends
>            MultipleTextOutputFormat<String, ControllerMetric> {
>
>        @Override
>        protected String generateFileNameForKeyValue(String key,
> ControllerMetric value, String name) {
>            return key;
>        }
>
>    }
>    public static void main(String[] args) throws Exception {
>
>        Job job = new Job();
>        job.setJarByClass(ControllerMetricGrinder.class);
>
>        job.setOutputKeyClass(Text.class);
>        job.setOutputValueClass(ControllerMetric.class);
>
>        job.setMapperClass(ControllerMetricMapper.class);
>
>        job.setCombinerClass(ControllerMetricReducer.class);
>        job.setReducerClass(ControllerMetricReducer.class);
>
>        // COMPILE ERROR HERE
>        MultipleOutputs.addMultiNamedOutput(job, "metrics",
>                MetricNameMultipleTextOutputFormat.class,
>                Text.class, ControllerMetric.class);
>
>        job.setNumReduceTasks(5);
>
>        FileInputFormat.addInputPath(job, new Path(args[0]));
>        FileOutputFormat.setOutputPath(job, new Path(args[1]));
>
>        System.exit(job.waitForCompletion(true) ? 0 : 1);
>    }
> }
>
> (mappers and reducers are using the new API, and are in separate classes).
>
> MultipleOutputs doesn't take a Job, it only takes a JobConf.  Any ideas?
>  I'd prefer to use the new API (because I've written it that way), but I'm
> guessing now I'll have to go and rework everything to the OLD API to get
> this to work.
>
> I'm trying to create a File-per-metric name (there's only 5).
>
> thoughts?
>
> Paul
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message