pig-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dmitriy Ryaboy <dvrya...@gmail.com>
Subject Re: Problems loading a datafile..
Date Wed, 02 Mar 2011 17:32:47 GMT
Off the top of my head, I can't think of anything, but you can just grab
everything in Elephant-Bird's lib/ directory and make sure it's on the
classpath on all the task trackers and your client machine (you can
propagate it to the TTs via the register keyword if you don't want to bug
your hadoop sysadmin and restart things).

D

On Wed, Mar 2, 2011 at 9:25 AM, Kris Coward <kris@melon.org> wrote:

>
> Nope; they're reproduced across all the machines. Does the
> LzoTokenizedLoader class have any dependencies that LzoTokenizedStorage
> doesn't (which I may be overlooking)?
>
> -K
>
> On Tue, Mar 01, 2011 at 07:17:10PM -0500, Kris Coward wrote:
> >
> > What's peculiar is that the test script for the loader class that was
> > run a week ago seems also to be failing with the same error. We've added
> > nodes to the cluster; maybe the relevant .jar files haven't been copied
> > over to those nodes. I'll bug our sysadmin about that..
> >
> > Thanks,
> > Kris
> >
> > On Tue, Mar 01, 2011 at 02:08:32PM -0800, Dmitriy Ryaboy wrote:
> > > Kris,
> > > Check the pig log file. Often "unable to create input slice" is caused
> by
> > > errors such as not being able to find your loader class, or some
> dependency
> > > of your loader class.
> > >
> > > D
> > >
> > > On Tue, Mar 1, 2011 at 1:48 PM, Kris Coward <kris@melon.org> wrote:
> > >
> > > >
> > > > I get the output:
> > > >
> > > > rw-r--r--   2 kris supergroup     172694 2011-02-25 01:59
> > > > /path/to/file/item/ex/subdir
> > > >
> > > > -K
> > > >
> > > > On Tue, Mar 01, 2011 at 12:46:31PM -0800, Dmitriy Ryaboy wrote:
> > > > > What happens when you "hadoop fs -lsr" those paths?
> > > > >
> > > > > D
> > > > >
> > > > > On Sun, Feb 27, 2011 at 7:47 PM, Kris Coward <kris@melon.org>
> wrote:
> > > > >
> > > > > >
> > > > > > So I finally got a couple of test scripts running on my cluster
> to take
> > > > > > a sample data file, load it, do a little processing, store it,
> load it,
> > > > > > do a little more processing, and dump the results.
> > > > > >
> > > > > > Once these were working, I set to parsing and storing some real
> data,
> > > > > > but when got an "Unable to create input slice" error when trying
> to
> > > > load
> > > > > > this data back out again. This happened with each of:
> > > > > >
> > > > > > foo = LOAD '/path/to/file/{item,list,glob}/*/subdir' USING
> > > > > > com.twitter.elephantbird.pig.load.LzoTokenizedLoader(',') AS
> > > > (schema:...);
> > > > > > foo = LOAD '/path/to/file/item/*/subdir' USING
> > > > > > com.twitter.elephantbird.pig.load.LzoTokenizedLoader(',') AS
> > > > (schema:...);
> > > > > > foo = LOAD '/path/to/file/item/ex/subdir' USING
> > > > > > com.twitter.elephantbird.pig.load.LzoTokenizedLoader(',') AS
> > > > (schema:...);
> > > > > >
> > > > > > and yielded the error (the same each time, except for the
> name/glob
> > > > > > used):
> > > > > >
> > > > > > ERROR 2997: Unable to recreate exception from backend error:
> > > > > > org.apache.pig.backend.executionengine.ExecException: ERROR
2118:
> > > > Unable to
> > > > > > create input slice for:
> > > > > > hdfs://master.hadoop:9000//path/to/file/item/ex/subdir
> > > > > > org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066:
> Unable
> > > > to
> > > > > > open iterator for alias foo
> > > > > >        at
> org.apache.pig.PigServer.openIterator(PigServer.java:482)
> > > > > >        at
> > > > > >
> > > >
> org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:539)
> > > > > >        at
> > > > > >
> > > >
> org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:241)
> > > > > >        at
> > > > > >
> > > >
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:168)
> > > > > >        at
> > > > > >
> > > >
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:144)
> > > > > >        at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:75)
> > > > > >        at org.apache.pig.Main.main(Main.java:352)
> > > > > > Caused by: org.apache.pig.backend.executionengine.ExecException:
> ERROR
> > > > > > 2997: Unable to recreate exception from backend error:
> > > > > > org.apache.pig.backend.executionengine.ExecException: ERROR
2118:
> > > > Unable to
> > > > > > create input slice for:
> > > > > > hdfs://master.hadoop:9000/path/to/file/item/ex/subdir
> > > > > >        at
> > > > > >
> > > >
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Launcher.getStats(Launcher.java:176)
> > > > > >        at
> > > > > >
> > > >
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:253)
> > > > > >        at
> > > > > >
> > > >
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.execute(HExecutionEngine.java:249)
> > > > > >        at
> > > > > >
> org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:781)
> > > > > >        at org.apache.pig.PigServer.store(PigServer.java:529)
> > > > > >        at
> org.apache.pig.PigServer.openIterator(PigServer.java:465)
> > > > > >        ... 6 more
> > > > > >
> > > > > >
> > > > > > Anyone have any suggestions why this may be happening and how
to
> fix
> > > > it?
> > > > > >
> > > > > > Thanks,
> > > > > > Kris
> > > > > >
> > > > > > --
> > > > > > Kris Coward
> > > > http://unripe.melon.org/
> > > > > > GPG Fingerprint: 2BF3 957D 310A FEEC 4733  830E 21A4 05C7 1FEB
> 12B3
> > > > > >
> > > >
> > > > --
> > > > Kris Coward
> http://unripe.melon.org/
> > > > GPG Fingerprint: 2BF3 957D 310A FEEC 4733  830E 21A4 05C7 1FEB 12B3
> > > >
> >
> > --
> > Kris Coward                                   http://unripe.melon.org/
> > GPG Fingerprint: 2BF3 957D 310A FEEC 4733  830E 21A4 05C7 1FEB 12B3
>
> --
> Kris Coward                                     http://unripe.melon.org/
> GPG Fingerprint: 2BF3 957D 310A FEEC 4733  830E 21A4 05C7 1FEB 12B3
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message