crunch-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ron Hashimshony <ron.hashimsh...@myheritage.com>
Subject Re: Use of getDettachedValue and Unit Tests
Date Sat, 23 May 2015 06:37:56 GMT
Thanks,
Added a JIRA. Let me know if I can assist further.
Ron.

On Fri, May 22, 2015 at 7:03 PM, Josh Wills <jwills@cloudera.com> wrote:

> Hey Ron,
>
> It's a little tricky, but yeah, I think we could add a mode that would
> simulate object re-use (at the very least during the reduce phase, where it
> tends to cause most problems.) File a JIRA for it?
>
> Thanks,
> J
>
> On Fri, May 22, 2015 at 7:18 AM, Ron Hashimshony <
> ron.hashimshony@myheritage.com> wrote:
>
>> Hi,
>> We love Crunch for the great Unit-Testing capabilities, which gives us a
>> good confidence when running the pipeline on the real data.
>> However, we did find one place in which the Unit-Tests failed us - when
>> we need to add *getDettachedValue* call on *iterable*, unit-tests did
>> behave differently and did not reuse the same objects, as happened in the
>> production pipeline on the whole data.
>> Is there any way to incorporate this validation in unit-tests?
>> We have tests that run a set of *DoFn*s setting the input *MemCollection*
>> s and check the *MemCollection*s out using *MemPipeline*, and other
>> tests for a single *DoFn*'s *process* using *InMemoryEmitter*.
>> Thanks,
>> Ron Hashimshony
>>
>
>
>
> --
> Director of Data Science
> Cloudera <http://www.cloudera.com>
> Twitter: @josh_wills <http://twitter.com/josh_wills>
>



-- 

*Ron Hashimshony*
Back-End developer

Mobile: +972-54-6617722 | ron.h@myheritage.com | www.myheritage.com
MyHeritage Ltd., 3 Ariel Sharon St., Terminal Park, Or Yehuda 60250, Israel

<http://www.myheritage.com/>   <http://blog.myheritage.com/>
<https://www.facebook.com/myheritage> <https://twitter.com/myheritage>

Mime
View raw message