crunch-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pavithra Siriwardena <thilini.gunasek...@gmail.com>
Subject Re: Input source Text() does not exist!
Date Mon, 03 Nov 2014 11:09:32 GMT
Hi Gabriel,

Thanks for your quick reply. You are correct, I have mixed up the two
modes, being in pseudo-distributed mode, I thought I was  running the job
in local mode, and tried to access the local file system. I've switched
back to the standalone mode and now it's working.

Thanks & Regards,
-Pavithra

On Mon, Nov 3, 2014 at 3:16 PM, Gabriel Reid <gabriel.reid@gmail.com> wrote:

> Hi Pavithra,
>
> This looks like there is probably a difference in where you're putting
> your input files and where Hadoop is expecting to find them.
> Are your input files on your local file system, or on HDFS? And is
> Hadoop set up to work with your local file system or HDFS?
>
> The path "/home/pavithra/in" looks like it is likely a local path
> (i.e. on the drive of your machine where you're running the command).
> It could be that when hadoop is trying to read the files, it's
> connecting to HDFS, and not your local file system.
>
> An easy way to check this is to type "hadoop dfs -ls
> /home/pavithra/in". That command should list the input files you want
> to use for your Crunch job, but I'm expecting that it will say that
> that path does not exist (because it's looking on HDFS). Could you try
> this out and then post back what you get?
>
> - Gabriel
>
>
> On Mon, Nov 3, 2014 at 10:05 AM, Pavithra Siriwardena
> <thilini.gunasekara@gmail.com> wrote:
> > Hi folks,
> >
> > I am trying to run my first Crunch pipeline, using the example project
> given
> > in Crunch Home Page. I'm using Hadoop 2.51 and the Crunch0.11.0.
> >
> > When I try to run it locally with using Hadoop's launcher script,
> >
> > hadoop jar target/crunch-demo-1.0-SNAPSHOT-job.jar /home/pavithra/in
> > /home/pavithra/out
> >
> > I get the error following error saying 'Input source Text() does not
> > exist!'; Note that the input text files are in '/home/pavithra/in'.
> >
> > Exception in thread "main" java.lang.IllegalStateException: Input source
> > Text(/home/pavithra/in) does not exist!
> > at
> >
> org.apache.crunch.impl.dist.collect.BaseInputCollection.getSizeInternal(BaseInputCollection.java:66)
> > at
> >
> org.apache.crunch.impl.dist.collect.PCollectionImpl.getSize(PCollectionImpl.java:315)
> > at
> >
> org.apache.crunch.impl.dist.collect.BaseDoCollection.getSizeInternal(BaseDoCollection.java:49)
> > at
> >
> org.apache.crunch.impl.dist.collect.PCollectionImpl.getSize(PCollectionImpl.java:315)
> > at
> >
> org.apache.crunch.impl.dist.collect.BaseDoCollection.getSizeInternal(BaseDoCollection.java:49)
> > at
> >
> org.apache.crunch.impl.dist.collect.PCollectionImpl.getSize(PCollectionImpl.java:315)
> > at
> >
> org.apache.crunch.util.PartitionUtils.getRecommendedPartitions(PartitionUtils.java:51)
> > at
> >
> org.apache.crunch.util.PartitionUtils.getRecommendedPartitions(PartitionUtils.java:40)
> > at org.apache.crunch.lib.Aggregate.count(Aggregate.java:55)
> > at
> >
> org.apache.crunch.impl.dist.collect.PCollectionImpl.count(PCollectionImpl.java:245)
> > at com.example.WordCount.run(WordCount.java:54)
> > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> > at com.example.WordCount.main(WordCount.java:22)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > at java.lang.reflect.Method.invoke(Method.java:606)
> > at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
> >
> >  Can you please guide me on this problem...
> >
> > Thanks
> > -Pavithra
> >
> >
>



-- 
-Pavithra

Mime
View raw message