avro-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Doug Cutting (JIRA)" <j...@apache.org>
Subject [jira] Updated: (AVRO-581) java: add reducer that separates keys and values when map output is pairs
Date Wed, 21 Jul 2010 21:21:50 GMT

     [ https://issues.apache.org/jira/browse/AVRO-581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Doug Cutting updated AVRO-581:

          Status: Resolved  (was: Patch Available)
    Hadoop Flags: [Reviewed]
      Resolution: Fixed

> Could we do a similar thing for the Avro API?

Sure.  I made it so that, with a map-only job, you can call either setOutputSchema() or setMapOutputSchema().

> Otherwise, I'm +1 on this.

Thanks for the reviews!

I just committed this.

> java: add reducer that separates keys and values when map output is pairs
> -------------------------------------------------------------------------
>                 Key: AVRO-581
>                 URL: https://issues.apache.org/jira/browse/AVRO-581
>             Project: Avro
>          Issue Type: New Feature
>          Components: java
>            Reporter: Doug Cutting
>            Assignee: Doug Cutting
>             Fix For: 1.4.0
>         Attachments: AVRO-581.patch, AVRO-581.patch, AVRO-581.patch, AVRO-581.patch,
weather-sorted.avro, weather.avro
> We should add a Pair<K,V> class, implementing SpecificRecord, that combines instances
of two schemas (specific or generic).  Pairs would be compared by key, ignoring value.  The
template for its schema would be:
> {code}
> {"type": "record", "name": "org.apache.avro.mapred.Pair", "fields":[
>   {"name": "key", "type":" <<insert key schema here>>},
>   {"name": "value", "order": "ignore", "type": <<insert value schema>>}
> ]}
> {code}
> When map outputs are instances of this class, a reducer may be used whose reduce method
is something like:
> public abstract void reduce(K key, Iterable<V> values);

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message