crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chandan Biswas <cbiswas1...@gmail.com>
Subject Re: Query on Channels.split
Date Tue, 18 Nov 2014 14:42:45 GMT
Hi Josh,
Here is the code for Tuple3 and Tuple4-
https://gist.github.com/cbiswas1983/7018fda426e8b085c6c7
https://gist.github.com/cbiswas1983/b9140376e702876783b6

If it looks good and you all agree with design then I will write the unit
test and send it to you. If any changes are required then suggest me. I
will be happier to do it.

Thanks,
Chandan


On Mon, Nov 17, 2014 at 5:27 PM, Josh Wills <josh.wills@gmail.com> wrote:

> Hey Chandan,
>
> No, it doesn't exist right now, please do send it along.
>
> Josh
> On Mon, Nov 17, 2014 at 12:07 PM Chandan Biswas <cbiswas1983@gmail.com>
> wrote:
>
> > Hi,
> > Currently Channels.split supports only Pair class. But, in our team we
> need
> > Tuple support.  Currently what we are doing is -
> >
> > PCollection<Pair<Pair<A,B>,C> pair1=someDoFn....
> > Pair<PCollection<Pair<A,B>>,PCollection<C>> pair1AfterFirstSplit
=
> > Channels.split(pair1);
> > Pair<PCollection<A>,PCollection<B>> pair1AfterSecondSplit =
> > Channels.split(pair1.first());
> >
> > instead we can do if Tuple is supported -
> >
> > PCollection<Tuple3<A,B,C> firstTuple3=someDoFn....
> > Tuple3<PCollection<A>,PCollection<B>,PCollection<C>>
> > firstTuple3AfterSplit=firstTuple3.split(...);
> >
> > Please let me know is if the functionality is already present in Crunch.
> If
> > yes, it will be great if anybody directs me to it.
> >
> > Following the existing pattern in Channels class, I have written separate
> > Channels classes for Tuple3 and Tuple4. If it's required I can send it.
> >
> > Thanks,
> > Chandan
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message