Mailing-List: contact user-help@crunch.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@crunch.apache.org
Received-SPF: pass (nike.apache.org: domain of jwills@cloudera.com designates
 209.85.215.50 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CADH-_N+w90bGP63mm2wKq4VhXGerbw6kpkTkui-Ue50pS+JZ=g@mail.gmail.com>
References: 
 <CADH-_NLGPyXyFBKq-4y2wM5mbO=JMX1agCXYYWYC5zVDqb5Keg@mail.gmail.com>
 <CAH29n6NJijQ8s9E2GUzmV6SQCT5pd9sXKkCg56xEow2mVNB1kw@mail.gmail.com>
 <CADH-_N+w90bGP63mm2wKq4VhXGerbw6kpkTkui-Ue50pS+JZ=g@mail.gmail.com>
From: Josh Wills <jwills@cloudera.com>
Date: Tue, 26 Mar 2013 21:02:02 -0700
Message-ID: 
 <CAH29n6NP=uBtmOjX62JDjXptOSLZSqq=qzm8AjpwTEtH9yzySA@mail.gmail.com>
Subject: Re: Mixing Writables and Avros PTypes
To: user@crunch.apache.org
Content-Type: text/plain; charset=ISO-8859-1

PTypeFamily has an as(...) method right now that knows how to convert
primitive and collection types from one PTypeFamily to another. I was
thinking of extending it to support derived types and the more common
kind of generic types-- so like Avros.generics and
Writables.writables, which should be able to be converted from one
type family to another with a little bit of magic.

On Mon, Mar 25, 2013 at 7:15 PM, Micah Whitacre <mkwhitacre@gmail.com> wrote:
> Is this what you are thinking when you mentioned the PType.as(...) method?
> http://pastebin.com/cjwss9Hs
> (note i realize not really code complete but just as an example)
>
> Its similar to what you suggested for the UUID support enhancement.
>
> or you were really thinking the PType interface would get an "as(...)"
> method? that took in the converted PTypeFamily (e.g.
> PType.as(PTypeFamily destinationFamily)).
>
> I could see how making the code on pastebin more generic to at least
> support Avros.toWritables(Class<?>)  would be useful.
>
> All of these seem like good enhancements to me.
>
> On Tue, Mar 19, 2013 at 7:43 PM, Josh Wills <jwills@cloudera.com> wrote:
>> Hey Micah,
>>
>> Agreed that this isn't an ideal state of affairs. I think the trick for
>> PTables should be to have the key type determine the value type, and have
>> derived types that would either convert the PType automatically (via the
>> PType.as(...) or some such thing), or convert the Avro type to a
>> BytesWritable (for Writable keys) or the Writable type to use the
>> Avros.writables functionality. I don't think it would be super-complicated
>> to try it out and see if there are any edge cases here that I'm missing.
>>
>> J
>>
>>
>> On Tue, Mar 19, 2013 at 4:31 PM, Micah Whitacre <mkwhitacre@gmail.com>
>> wrote:
>>>
>>> Out of curiosity why can we not mix PTypes between Writables and
>>> Avros?  The situation I'm encountering are cases where someone wants
>>> to create a PTable<UUID, SomeAvro>.  UUID is hidden behind a
>>> UUIDWritable (+ MapFn implementations) and exposed as a PType using
>>> Writables.derive(...).  SomeAvro's PType is exposed using
>>> Avros.records(SomeAvro.class).
>>>
>>> If consumers want to create the PTable<UUID, SomeAvro> they can either
>>> do Avros.tableOf(UUID.PTYPE, SomeAvros.PTYPE) or
>>> Writables.tableOf(UUID.PType, SomeAvros.PTYPE) but that will throw an
>>> IllegalArgumentException because in both cases something doesn't
>>> extend the correct type.
>>>
>>> Part of the confusion is that the method signatures for both are just
>>> PType and don't document or restrict the allowed types until runtime.
>>> Is there a way that we might be missing to make both flavors of PTypes
>>> play nicely and be able to create tables, pairs, etc easily?
>>>
>>> I was kind of hoping most data providers could hide whether they are
>>> Avro, Writable, or whatever by just exposing PTypes for consumer to
>>> use but now it seems consumer might need to implement twice as many
>>> PTypes and wrappers for a mixed pipeline to work well.
>>>
>>> Thanks,
>>> Micah
>>
>>
>>
>>
>> --
>> Director of Data Science
>> Cloudera
>> Twitter: @josh_wills


-- 
Director of Data Science
Cloudera
Twitter: @josh_wills