crunch-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From John Jensen <jen...@richrelevance.com>
Subject RE: Problem running job with large number of directories
Date Wed, 22 May 2013 23:42:33 GMT

Certainly. Appreciate it.

________________________________
From: Josh Wills [jwills@cloudera.com]
Sent: Wednesday, May 22, 2013 4:38 PM
To: user@crunch.apache.org
Subject: Re: Problem running job with large number of directories

Hey John,

I haven't hit that one before, but I have some hypothesis we could test if you're up for some
trying out some patches I write.

J


On Wed, May 22, 2013 at 4:01 PM, John Jensen <jensen@richrelevance.com<mailto:jensen@richrelevance.com>>
wrote:

I have a curious problem when running a crunch job on (avro) files in a fairly large set of
directories (just slightly less than 100).
After running some fraction of the mappers they start failing with the exception below. Things
work fine with a smaller number of directories.

The magic 'zdHJpbmcifSx7Im5hbWUiOiJ2YWx1ZSIsInR5cGUiOiJzdHJpbmcifV19fSwiZGVmYXVsdCI' string
shows up in the 'crunch.inputs.dir' entry in the job config, so I assume it has something
to do with deserializing that value, but reading through the code I don't see any obvious
way how.

Furthermore, the crunch.inputs.dir config entry is just under 1.5M, so it would not surprise
me if I'm running up against a hadoop limit somewhere.

Has anybody else seen similar issues? (this is 0.5.0, btw).

-- John


java.io.IOException: Split class zdHJp
bmcifSx7Im5hbWUiOiJ2YWx1ZSIsInR5cGUiOiJzdHJpbmcifV19fSwiZGVmYXVsdCI not found
        at org.apache.hadoop.mapred.MapTask.getSplitDetails(MapTask.java:342)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:614)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325)
        at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
        at org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: java.lang.ClassNotFoundException: Class zdHJp
bmcifSx7Im5hbWUiOiJ2YWx1ZSIsInR5cGUiOiJzdHJpbmcifV19fSwiZGVmYXVsdCI not found
        at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1493)
        at org.apache.hadoop.mapred.MapTask.getSplitDetails(MapTask.java:340)
        ... 7 more



--
Director of Data Science
Cloudera<http://www.cloudera.com>
Twitter: @josh_wills<http://twitter.com/josh_wills>

Mime
View raw message