crunch-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Josh Wills <jwi...@cloudera.com>
Subject Re: Mixing Writables and Avros PTypes
Date Wed, 27 Mar 2013 04:02:02 GMT
PTypeFamily has an as(...) method right now that knows how to convert
primitive and collection types from one PTypeFamily to another. I was
thinking of extending it to support derived types and the more common
kind of generic types-- so like Avros.generics and
Writables.writables, which should be able to be converted from one
type family to another with a little bit of magic.

On Mon, Mar 25, 2013 at 7:15 PM, Micah Whitacre <mkwhitacre@gmail.com> wrote:
> Is this what you are thinking when you mentioned the PType.as(...) method?
> http://pastebin.com/cjwss9Hs
> (note i realize not really code complete but just as an example)
>
> Its similar to what you suggested for the UUID support enhancement.
>
> or you were really thinking the PType interface would get an "as(...)"
> method? that took in the converted PTypeFamily (e.g.
> PType.as(PTypeFamily destinationFamily)).
>
> I could see how making the code on pastebin more generic to at least
> support Avros.toWritables(Class<?>)  would be useful.
>
> All of these seem like good enhancements to me.
>
> On Tue, Mar 19, 2013 at 7:43 PM, Josh Wills <jwills@cloudera.com> wrote:
>> Hey Micah,
>>
>> Agreed that this isn't an ideal state of affairs. I think the trick for
>> PTables should be to have the key type determine the value type, and have
>> derived types that would either convert the PType automatically (via the
>> PType.as(...) or some such thing), or convert the Avro type to a
>> BytesWritable (for Writable keys) or the Writable type to use the
>> Avros.writables functionality. I don't think it would be super-complicated
>> to try it out and see if there are any edge cases here that I'm missing.
>>
>> J
>>
>>
>> On Tue, Mar 19, 2013 at 4:31 PM, Micah Whitacre <mkwhitacre@gmail.com>
>> wrote:
>>>
>>> Out of curiosity why can we not mix PTypes between Writables and
>>> Avros?  The situation I'm encountering are cases where someone wants
>>> to create a PTable<UUID, SomeAvro>.  UUID is hidden behind a
>>> UUIDWritable (+ MapFn implementations) and exposed as a PType using
>>> Writables.derive(...).  SomeAvro's PType is exposed using
>>> Avros.records(SomeAvro.class).
>>>
>>> If consumers want to create the PTable<UUID, SomeAvro> they can either
>>> do Avros.tableOf(UUID.PTYPE, SomeAvros.PTYPE) or
>>> Writables.tableOf(UUID.PType, SomeAvros.PTYPE) but that will throw an
>>> IllegalArgumentException because in both cases something doesn't
>>> extend the correct type.
>>>
>>> Part of the confusion is that the method signatures for both are just
>>> PType and don't document or restrict the allowed types until runtime.
>>> Is there a way that we might be missing to make both flavors of PTypes
>>> play nicely and be able to create tables, pairs, etc easily?
>>>
>>> I was kind of hoping most data providers could hide whether they are
>>> Avro, Writable, or whatever by just exposing PTypes for consumer to
>>> use but now it seems consumer might need to implement twice as many
>>> PTypes and wrappers for a mixed pipeline to work well.
>>>
>>> Thanks,
>>> Micah
>>
>>
>>
>>
>> --
>> Director of Data Science
>> Cloudera
>> Twitter: @josh_wills



-- 
Director of Data Science
Cloudera
Twitter: @josh_wills

Mime
View raw message