avro-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Wilkes <cwil...@gmail.com>
Subject Re: Problem running an Non Avro to Avro Job
Date Thu, 15 Sep 2011 14:35:29 GMT
In  org.apache.avro.mapred.HadoopMapper:

public void collect(OUT datum) throws IOException {
      if (isMapOnly) {
        // don't require a Pair for datum
      } else {
        // cast datum to a pair
     }
 }

so in your case you probably just want to set the number of reducers
to 0 to create an avro record without a Pair as the output value (you
mention "key" below btw).

Actually re-reading it looks like you are trying to make a Pair
object.  Why are you putting that inside an AvroWrapper?  Just do
  collector.collect(p)
And set your map output schema to TestPair.SCHEMA$ where SCHEMA$ =
Pair.getPairSchema(keySChema, valueSchema)

Chris

On Thu, Sep 15, 2011 at 6:10 AM, Rohini U <rohini.u@gmail.com> wrote:
> Hi All,
>
> I am using avro 15.3 and CDH3. I have a query in running a Non Avro to Avro
> Job. My mapper is non Avro mapper and it outputs a avro data.
> When I run this, I get an error saying not a Pair Schema. Does the key
> always have to be a Pair Schema?
>
> public void map(Text key, Text value,
>                 OutputCollector<AvroWrapper<TestPair>, NullWritable>
output,
>                 Reporter reporter) throws IOException {
>             String valueString = value.toString();
>             String[] toks = valueString.split("\t");
>             TestPair p = new TestPair();
>             p.left = toks[0];
>             p.right = toks[1];
>
>
>             output.collect(new AvroWrapper<TestPair>(p),wt);
>         }
>
>  Thanks,
> -Rohini
>

Mime
View raw message