Mailing-List: contact user-help@crunch.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@crunch.apache.org
Received-SPF: pass (nike.apache.org: domain of gabriel.reid@gmail.com
 designates 209.85.212.179 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CABf=UBjOQv_5kkqWiTy7vbzQNLyz9HiwXX1nWayq26d+fFybhQ@mail.gmail.com>
References: 
 <CABf=UBjOQv_5kkqWiTy7vbzQNLyz9HiwXX1nWayq26d+fFybhQ@mail.gmail.com>
Date: Mon, 3 Nov 2014 10:46:15 +0100
Message-ID: 
 <CAA5C_psa22JPLFz+XzoStPNSqMwW7JJMgyuDDH__bAUOhCDzNw@mail.gmail.com>
Subject: Re: Input source Text() does not exist!
From: Gabriel Reid <gabriel.reid@gmail.com>
To: "user@crunch.apache.org" <user@crunch.apache.org>
Content-Type: text/plain; charset=UTF-8

Hi Pavithra,

This looks like there is probably a difference in where you're putting
your input files and where Hadoop is expecting to find them.
Are your input files on your local file system, or on HDFS? And is
Hadoop set up to work with your local file system or HDFS?

The path "/home/pavithra/in" looks like it is likely a local path
(i.e. on the drive of your machine where you're running the command).
It could be that when hadoop is trying to read the files, it's
connecting to HDFS, and not your local file system.

An easy way to check this is to type "hadoop dfs -ls
/home/pavithra/in". That command should list the input files you want
to use for your Crunch job, but I'm expecting that it will say that
that path does not exist (because it's looking on HDFS). Could you try
this out and then post back what you get?

- Gabriel


On Mon, Nov 3, 2014 at 10:05 AM, Pavithra Siriwardena
<thilini.gunasekara@gmail.com> wrote:
> Hi folks,
>
> I am trying to run my first Crunch pipeline, using the example project given
> in Crunch Home Page. I'm using Hadoop 2.51 and the Crunch0.11.0.
>
> When I try to run it locally with using Hadoop's launcher script,
>
> hadoop jar target/crunch-demo-1.0-SNAPSHOT-job.jar /home/pavithra/in
> /home/pavithra/out
>
> I get the error following error saying 'Input source Text() does not
> exist!'; Note that the input text files are in '/home/pavithra/in'.
>
> Exception in thread "main" java.lang.IllegalStateException: Input source
> Text(/home/pavithra/in) does not exist!
> at
> org.apache.crunch.impl.dist.collect.BaseInputCollection.getSizeInternal(BaseInputCollection.java:66)
> at
> org.apache.crunch.impl.dist.collect.PCollectionImpl.getSize(PCollectionImpl.java:315)
> at
> org.apache.crunch.impl.dist.collect.BaseDoCollection.getSizeInternal(BaseDoCollection.java:49)
> at
> org.apache.crunch.impl.dist.collect.PCollectionImpl.getSize(PCollectionImpl.java:315)
> at
> org.apache.crunch.impl.dist.collect.BaseDoCollection.getSizeInternal(BaseDoCollection.java:49)
> at
> org.apache.crunch.impl.dist.collect.PCollectionImpl.getSize(PCollectionImpl.java:315)
> at
> org.apache.crunch.util.PartitionUtils.getRecommendedPartitions(PartitionUtils.java:51)
> at
> org.apache.crunch.util.PartitionUtils.getRecommendedPartitions(PartitionUtils.java:40)
> at org.apache.crunch.lib.Aggregate.count(Aggregate.java:55)
> at
> org.apache.crunch.impl.dist.collect.PCollectionImpl.count(PCollectionImpl.java:245)
> at com.example.WordCount.run(WordCount.java:54)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at com.example.WordCount.main(WordCount.java:22)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
>
>  Can you please guide me on this problem...
>
> Thanks
> -Pavithra
>
>