crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Stefan De Smit (JIRA)" <>
Subject [jira] [Commented] (CRUNCH-572) Avro schema error in Avros.tableOf() can't redefine org.apache.avro.mapred.Pair
Date Mon, 14 Nov 2016 09:53:58 GMT


Stefan De Smit commented on CRUNCH-572:

We have a similar problem, where we used Pair.getPairSchema() in our own code (so, outside
of PTableType)
Isn't the problem that Pair.makePairSchema always uses the same name "Pair" for different
schema's (as there's a cache for same key/value schema) instead of generating a new name similar
to Avros.createTupleSchema

> Avro schema error in Avros.tableOf() can't redefine org.apache.avro.mapred.Pair
> -------------------------------------------------------------------------------
>                 Key: CRUNCH-572
>                 URL:
>             Project: Crunch
>          Issue Type: Bug
>    Affects Versions: 0.13.0
>            Reporter: Adric Eckstein
>         Attachments: CRUNCH-572.patch
> The Avros.tableOf() method produces a record schema named "org.apache.avro.mapred.Pair",
which cannot be unique across types, whereas the Avros.pairs() method similarly creates a
record schema, but calls Avros.createTupleSchema() which creates a unique name at every call.
> Example that fails on constructing schema:
> AvroType<Pair<String, String>> t1 = Avros.tableOf(Avros.strings(), Avros.strings());
> AvroType<Pair<String, Double>> t2 = Avros.tableOf(Avros.strings(), Avros.doubles());
> AvroType<Pair<Pair<String, String>, Pair<String, Double>>> t3
= Avros.pairs(t1, t2);
> System.out.println("schema: " + t3.getSchema().toString());
> Can the tableOf() method be updated to call the createTupleSchema() method instead of
the avro pair constructor?

This message was sent by Atlassian JIRA

View raw message