avro-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From snikhil0 <snik...@telenav.com>
Subject Re: Avro Map Reduce Question: GenericRecord, renaming reduce output
Date Wed, 13 Jun 2012 07:26:52 GMT
Ok this things looks like a map-reduce api issue:

I went back to the old style of map-reduce api: now I get a good avro header
but no datums. Sheesh! can someone please help !

The main function:

        final static Schema IN_SCHEMA =
LogshedCollectorUtils.getResourceSchema();
	final static Schema OUT_SCHEMA = LogshedCollectorUtils.getResourceSchema();
	final static ReflectData reflectData = ReflectData.get();
        final static Schema KEY_SCHEMA =
reflectData.getSchema(LogKeyWritable.class);
	final static Schema MAP_OUT_SCHEMA = Pair.getPairSchema(KEY_SCHEMA,
OUT_SCHEMA);

                Configuration conf =
LogshedCollectorUtils.getLocalHadoopConfiguartion();
		JobConf jobConf = new
JobConf(LogshedCollectorUtils.getLocalHadoopConfiguartion(),
				MuxDemuxJob.class);
		jobConf.setJobName("muxdemux");
		jobConf.setJarByClass(MuxDemuxJob.class);
		
		jobConf.setInputFormat(AvroInputFormat.class);
		jobConf.setOutputFormat(AvroOutputFormat.class);
		
		AvroJob.setInputSchema(jobConf, IN_SCHEMA);
		AvroJob.setMapOutputSchema(jobConf, MAP_OUT_SCHEMA);
		AvroJob.setOutputSchema(jobConf, OUT_SCHEMA);
		
		AvroJob.setMapperClass(jobConf, LogshedMapper.class);
		AvroJob.setReducerClass(jobConf, LogshedReducer.class);
		
		//Job job = new Job(jobConf, "muxdemux");
		
		FileInputFormat.setInputPaths(jobConf, new Path(args[0]));
		Path outPath = new Path(args[1]);
		FileOutputFormat.setOutputPath(jobConf, outPath);

		JobClient.runJob(jobConf);
		return 0;

Nikhil

--
View this message in context: http://apache-avro.679487.n3.nabble.com/Avro-Map-Reduce-Question-GenericRecord-renaming-reduce-output-tp4025105p4025126.html
Sent from the Avro - Users mailing list archive at Nabble.com.

Mime
View raw message