hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harsh J <ha...@cloudera.com>
Subject Re: Hadoop Terasort Error- "File _partition.lst does not exist"
Date Wed, 25 Jan 2012 12:30:22 GMT
Its not your TaskTracker thats failing, your job itself is running
locally, and not on a JobTracker. This would not work for what you're
trying to run.

Are you sure you have the right mapred-site.xml configuration from
where you launch your job?

On Wed, Jan 25, 2012 at 5:12 PM, Utkarsh Rathore
<utkarsh.bitspilani@gmail.com> wrote:
> Thanks Harsh.
>
> I' ll look into the tasktracker logs to find any issues with mapreduce and
> update this thread accordingly.
>
> (PS: Sorry for the wide circulation. My mails still don't directly land on
> common-user@hadoop.apache.org so tried posting it
> through Nabble and something got broken. I have mailed owner of this
> mailing list so hopefully I'll able to post directly in future).
>
> ~Utkarsh
>
>
>
> On Wed, Jan 25, 2012 at 11:40 AM, Harsh J <harsh@cloudera.com> wrote:
>
>> Apparently, you are running terasort with a local job runner as
>> explained by the presence of "org.apache.hadoop.fs.RawLocalFileSystem"
>> and "LocalJobRunner" in your provided log message.
>>
>> Ensure mapred.job.tracker is properly set in your mapred-site.xml, for
>> your job to reach the MapReduce cluster. You seem to have HDFS
>> configured properly from where you are running the job/commands, but
>> not MapReduce.
>>
>> (P.s. You do not need to hit at every list until someone replies. We
>> do not ignore your questions, but there may be some time before
>> someone can look into your issue and answer back.)
>>
>> On Wed, Jan 25, 2012 at 11:28 AM, rathore87
>> <utkarsh.bitspilani@gmail.com> wrote:
>> >
>> > Folks,
>> >
>> > I have a Hadoop cluster on which I have generated some data using
>> Teragen.
>> > But while running Terasort on this data, it gives following error.
>> >
>> > java.lang.RuntimeException: Error in configuring object
>> >        at org.apache.hadoop.util.
>> > ReflectionUtils.setJobConf(ReflectionUtils.java:93)
>> >        at
>> > org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
>> >        at
>> >
>> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
>> >        at
>> >
>> org.apache.hadoop.mapred.MapTask$OldOutputCollector.<init>(MapTask.java:481)
>> >        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:391)
>> >        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325)
>> >        at
>> > org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:210)
>> > Caused by: java.lang.reflect.InvocationTargetException
>> >        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> >        at
>> >
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> >        at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> >        at java.lang.reflect.Method.invoke(Method.java:597)
>> >        at
>> >
>> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
>> >        ... 6 more
>> >
>> > I have generated some data using Teragen on my Hadoop cluster. But when I
>> > run
>> > Caused by: java.lang.IllegalArgumentException: can't read paritions file
>> >        at
>> >
>> org.apache.hadoop.examples.terasort.TeraSort$TotalOrderPartitioner.configure(TeraSort.java:213)
>> >        ... 11 more
>> > Caused by: java.io.FileNotFoundException: File _partition.lst does not
>> > exist.
>> >        at
>> >
>> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:383)
>> >        at
>> >
>> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:251)
>> >        at org.apache.hadoop.fs.FileSystem.getLength(FileSystem.java:776)
>> >        at
>> > org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1424)
>> >        at
>> > org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1419)
>> >        at
>> >
>> org.apache.hadoop.examples.terasort.TeraSort$TotalOrderPartitioner.readPartitions(TeraSort.java:153)
>> >        at
>> >
>> org.apache.hadoop.examples.terasort.TeraSort$TotalOrderPartitioner.configure(TeraSort.java:210)
>> >        ... 11 more
>> >
>> > HDFS does shows this file in its listing and I’m unable to understand
>> why is
>> > Terasort unable to find this flle.
>> >
>> > bash-3.2$ hadoop dfs -lsr /user/hduser/
>> > drwxrwxrwx   - hdfs supergroup          0 2012-01-24 14:12
>> > /user/hduser/terasort-input1
>> > -rw-r--r--   1 hdfs supergroup             0 2012-01-24 00:38
>> > /user/hduser/terasort-input1/_SUCCESS
>> > -rw-r--r--   1 hdfs supergroup           129 2012-01-24 14:12
>> > /user/hduser/terasort-input1/_partition.lst
>> > -rw-r--r--   1 hdfs supergroup 1000000000000 2012-01-23 15:25
>> > /user/hduser/terasort-input1/part-00000
>> >
>> > I tried changing the file permissions, ownership and copying the
>> > _partition.lst file at root of HDFS (so that relative path does not
>> matter)
>> > but nothing seems to work. Online forums/mailing lists also don't help.
>> >
>> > Any help/pointers on this will be appreciated.
>> >
>> > TIK!
>> > ~Utkarsh
>> >
>> > --
>> > View this message in context:
>> http://old.nabble.com/Hadoop-Terasort-Error--%22File-_partition.lst-does-not-exist%22-tp33199363p33199363.html
>> > Sent from the Hadoop core-user mailing list archive at Nabble.com.
>> >
>>
>>
>>
>> --
>> Harsh J
>> Customer Ops. Engineer, Cloudera
>>



-- 
Harsh J
Customer Ops. Engineer, Cloudera

Mime
View raw message