crunch-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Micah Whitacre <mkwhita...@gmail.com>
Subject Mixing Writables and Avros PTypes
Date Tue, 19 Mar 2013 23:31:22 GMT
Out of curiosity why can we not mix PTypes between Writables and
Avros?  The situation I'm encountering are cases where someone wants
to create a PTable<UUID, SomeAvro>.  UUID is hidden behind a
UUIDWritable (+ MapFn implementations) and exposed as a PType using
Writables.derive(...).  SomeAvro's PType is exposed using
Avros.records(SomeAvro.class).

If consumers want to create the PTable<UUID, SomeAvro> they can either
do Avros.tableOf(UUID.PTYPE, SomeAvros.PTYPE) or
Writables.tableOf(UUID.PType, SomeAvros.PTYPE) but that will throw an
IllegalArgumentException because in both cases something doesn't
extend the correct type.

Part of the confusion is that the method signatures for both are just
PType and don't document or restrict the allowed types until runtime.
Is there a way that we might be missing to make both flavors of PTypes
play nicely and be able to create tables, pairs, etc easily?

I was kind of hoping most data providers could hide whether they are
Avro, Writable, or whatever by just exposing PTypes for consumer to
use but now it seems consumer might need to implement twice as many
PTypes and wrappers for a mixed pipeline to work well.

Thanks,
Micah

Mime
View raw message