crunch-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeremy Lewi <jer...@lewi.us>
Subject Re: Exception with AvroPathPerKeyTarget
Date Fri, 28 Mar 2014 13:45:56 GMT
This is my first time on a  cluster I'll try what Josh suggests now.

J


On Fri, Mar 28, 2014 at 3:41 AM, Josh Wills <josh.wills@gmail.com> wrote:

>
> On Fri, Mar 28, 2014 at 1:22 AM, Gabriel Reid <gabriel.reid@gmail.com>wrote:
>
>> Hi Jeremy,
>>
>> On Thu, Mar 27, 2014 at 3:26 PM, Jeremy Lewi <jeremy@lewi.us> wrote:
>> > Hi
>> >
>> > I'm hitting the exception pasted below when using AvroPathPerKeyTarget.
>> > Interestingly, my code works just fine when I run on a small dataset
>> using
>> > the LocalJobTracker. However, when I run on a large dataset using a
>> hadoop
>> > cluster I hit the exception.
>> >
>>
>> Have you ever been able to successfully use the AvroPathPerKeyTarget
>> on a real cluster, or is this the first try with it?
>>
>> I'm wondering if this could be a problem that's always been around (as
>> the integration test for AvroPathPerKeyTarget also runs in the local
>> jobtracker), or if this could be something new.
>>
>
> +1-- Jeremy, if you force the job to run w/a single reducer on the cluster
> (i.e., via groupByKey(1)), does it work?
>
>
>>
>> - Gabriel
>>
>
>

Mime
View raw message