hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Mawata <chris.maw...@gmail.com>
Subject Re: Partitioning and setup errors
Date Sun, 29 Jun 2014 12:26:19 GMT
It is the part about implementing the tool interface that is the issue.
Does your driver class implement Tool?
Chris
On Jun 29, 2014 8:22 AM, "Chris MacKenzie" <
studio@chrismackenziephotography.co.uk> wrote:

> HI Vinod,
>
> Thanks for your support. I’m packaging my application in Eclipse (Kepler)
> as a runnable jar. I can get the application to debug there in local mode.
>
> The command line I am using for my app is:
>
>  "/usr/local/hadoop-2.4.0/jar_files $ hadoop jar
> /usr/local/hadoop-2.4.0/jar_files/ParallelGeneticAlignment.jar a.txt 10 10”
>
> the jar permissions are -"rwxr-xr-x"
>
> The three parameters on the end of the command line are arbitrary commands
> from my sequential program which I will use as input when I can get this to
> run.
>
> I have downloaded the source and run a 2.4 version of WordCount, it’s not
> getting past the map stage either.
>
> I’m pasting in the whole job output.
>
> Deleted demo_output
>
> /usr/local/hadoop-2.4.0/jar_files $ hadoop jar
> /usr/local/hadoop-2.4.0/jar_files/ParallelGeneticAlignment.jar a.txt 10 10
>
> Start time = 1404044126713
>
> SLF4J: Class path contains multiple SLF4J bindings.
>
> SLF4J: Found binding in [rsrc:org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: Found binding in
> [jar:rsrc:slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
>
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
>
> 14/06/29 13:16:06 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
>
> 14/06/29 13:16:31 INFO client.RMProxy: Connecting to ResourceManager at /
> 0.0.0.0:8032
>
> 14/06/29 13:16:46 WARN mapreduce.JobSubmitter: Hadoop command-line option
> parsing not performed. Implement the Tool interface and execute your
> application with ToolRunner to remedy this.
>
> 14/06/29 13:16:46 WARN mapreduce.JobSubmitter: No job jar file set.  User
> classes may not be found. See Job or Job#setJar(String).
>
> 14/06/29 13:16:46 INFO input.FileInputFormat: Total input paths to process
> : 0
>
> 14/06/29 13:17:10 INFO mapreduce.JobSubmitter: number of splits:0
>
> 14/06/29 13:17:13 INFO mapreduce.JobSubmitter: Submitting tokens for job:
> job_1403953980618_0009
>
> 14/06/29 13:17:13 INFO mapred.YARNRunner: Job jar is not present. Not
> adding any jar to the list of resources.
>
> 14/06/29 13:17:13 INFO impl.YarnClientImpl: Submitted application
> application_1403953980618_0009
>
> 14/06/29 13:17:13 INFO mapreduce.Job: The url to track the job:
> http://admins-MacBook-Pro.local:8088/proxy/application_1403953980618_0009/
>
> 14/06/29 13:17:13 INFO mapreduce.Job: Running job: job_1403953980618_0009
>
> 14/06/29 13:17:19 INFO mapreduce.Job: Job job_1403953980618_0009 running
> in uber mode : false
>
> 14/06/29 13:17:19 INFO mapreduce.Job:  map 0% reduce 0%
>
> 14/06/29 13:17:23 INFO mapreduce.Job: Task Id :
> attempt_1403953980618_0009_r_000000_0, Status : FAILED
>
> Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceReducer not found
>
> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)
>
> at
> org.apache.hadoop.mapreduce.task.JobContextImpl.getReducerClass(JobContextImpl.java:210)
>
> at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:611)
>
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
>
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at javax.security.auth.Subject.doAs(Subject.java:422)
>
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
>
> Caused by: java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceReducer not found
>
> at
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)
>
> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)
>
> ... 8 more
>
>
> 14/06/29 13:17:28 INFO mapreduce.Job: Task Id :
> attempt_1403953980618_0009_r_000000_1, Status : FAILED
>
> Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceReducer not found
>
> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)
>
> at
> org.apache.hadoop.mapreduce.task.JobContextImpl.getReducerClass(JobContextImpl.java:210)
>
> at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:611)
>
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
>
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at javax.security.auth.Subject.doAs(Subject.java:422)
>
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
>
> Caused by: java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceReducer not found
>
> at
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)
>
> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)
>
> ... 8 more
>
>
> 14/06/29 13:17:32 INFO mapreduce.Job: Task Id :
> attempt_1403953980618_0009_r_000000_2, Status : FAILED
>
> Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceReducer not found
>
> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)
>
> at
> org.apache.hadoop.mapreduce.task.JobContextImpl.getReducerClass(JobContextImpl.java:210)
>
> at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:611)
>
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
>
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at javax.security.auth.Subject.doAs(Subject.java:422)
>
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
>
> Caused by: java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceReducer not found
>
> at
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)
>
> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)
>
> ... 8 more
>
>
> 14/06/29 13:17:37 INFO mapreduce.Job:  map 0% reduce 100%
>
> 14/06/29 13:17:37 INFO mapreduce.Job: Job job_1403953980618_0009 failed
> with state FAILED due to: Task failed task_1403953980618_0009_r_000000
>
> Job failed as tasks failed. failedMaps:0 failedReduces:1
>
>
> 14/06/29 13:18:04 INFO mapreduce.Job: Counters: 7
>
> Job Counters
>
> Failed reduce tasks=4
>
> Launched reduce tasks=4
>
> Total time spent by all maps in occupied slots (ms)=0
>
> Total time spent by all reduces in occupied slots (ms)=7786
>
> Total time spent by all reduce tasks (ms)=7786
>
> Total vcore-seconds taken by all reduce tasks=7786
>
> Total megabyte-seconds taken by all reduce tasks=7972864
>
> 14/06/29 13:18:05 INFO ipc.Client: Retrying connect to server:
> admins-MacBook-Pro.local/192.168.0.5:53193. Already tried 0 time(s);
> retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=3,
> sleepTime=1000 MILLISECONDS)
>
> 14/06/29 13:18:06 INFO ipc.Client: Retrying connect to server:
> admins-MacBook-Pro.local/192.168.0.5:53193. Already tried 1 time(s);
> retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=3,
> sleepTime=1000 MILLISECONDS)
>
> 14/06/29 13:18:07 INFO ipc.Client: Retrying connect to server:
> admins-MacBook-Pro.local/192.168.0.5:53193. Already tried 2 time(s);
> retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=3,
> sleepTime=1000 MILLISECONDS)
>
> 14/06/29 13:18:07 INFO mapred.ClientServiceDelegate: Application state is
> completed. FinalApplicationStatus=FAILED. Redirecting to job history server
>
> End time = 1404044288031
>
> Elapsed time = 161318 ms
>
> Finished
>
> /usr/local/hadoop-2.4.0/jar_files $
>
> Warm regards.
>
> Chris
>
> From: Vinod Kumar Vavilapalli <vinodkv@hortonworks.com>
> Reply-To: <user@hadoop.apache.org>
> Date: Sunday, 29 June 2014 00:20
> To: <user@hadoop.apache.org>
> Subject: Re: Partitioning and setup errors
>
> What is happening is the client is not able to pick up the right jar to
> push to the cluster. It looks in the class-path for the jar that contains
> the class ParallelGeneticAlignment.
>
> How are you packaging your code? How are your running your job - paste the
> command line?
>
> +Vinod
>
> On Jun 27, 2014, at 5:15 AM, Chris MacKenzie <
> studio@chrismackenziephotography.co.uk> wrote:
>
> Hi,
>
> I realise my previous question may have been a bit naïve and I also
> realise I am asking an awful lot here, any advice would be greatly
> appreciated.
>
>    - I have been using Hadoop 2.4 in local mode and am sticking to the mapreduce.*
>    side of the track.
>    - I am using a Custom Line reader to read each sequence into a Map
>    - I have a partitioner class which is testing the key from the map
>    class.
>    - I've tried debugging in eclipse with a breakpoint in the partitioner
>    class but getPartition(LongWritable mapKey, Text sequenceString, int
>    numReduceTasks) is not being called.
>
> Could there be any reason for that ?
>
> Because my map and reduce code works in local mode within eclipse, I wondered
> if I may get the partitioner to work if  I changed to Pseudo Distributed
> Mode exporting a runnable jar from Eclipse (Kepler)
>
> I have several faults On my own computer  Pseudo Distributed Mode and the
> university clusters Pseudo Distributed Mode which I set up. I’ve googled
> and read extensively but am not seeing a solution to any of these issues.
>
> I have this line:
> 14/06/27 11:45:27 WARN mapreduce.JobSubmitter: No job jar file set.  User
> classes may not be found. See Job or Job#setJar(String).
> My driver code is:
> private void doParallelConcordance() throws Exception {
>
>
> Path inDir = new Path("input_sequences/10_sequences.txt");
> Path outDir = new Path("demo_output");
>
> Job job = Job.getInstance(new Configuration());
> job.setJarByClass(ParallelGeneticAlignment.class);
>
>  job.setOutputKeyClass(Text.class);
> job.setOutputValueClass(IntWritable.class);
>
> job.setInputFormatClass(CustomFileInputFormat.class);
> job.setMapperClass(ConcordanceMapper.class);
> job.setPartitionerClass(ConcordanceSequencePartitioner.class);
> job.setReducerClass(ConcordanceReducer.class);
>
> FileInputFormat.addInputPath(job, inDir);
> FileOutputFormat.setOutputPath(job, outDir);
>
> job.waitForCompletion(true)
> }
>
> On the university server I am getting this error:
> 4/06/27 11:45:40 INFO mapreduce.Job: Task Id :
> attempt_1403860966764_0003_m_000000_0, Status : FAILED
> Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceMapper not found
>
> On my machine the error is:
> 4/06/27 12:58:03 INFO mapreduce.Job: Task Id :
> attempt_1403864060032_0004_r_000000_2, Status : FAILED
> Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceReducer not found
>
> On the university server I get total paths to process:
> 14/06/27 11:45:27 INFO input.FileInputFormat: Total input paths to process
> : 1
> 14/06/27 11:45:28 INFO mapreduce.JobSubmitter: number of splits:1
>
> On my machine I get total paths to process:
> 14/06/27 12:57:09 INFO input.FileInputFormat: Total input paths to process
> : 0
> 14/06/27 12:57:36 INFO mapreduce.JobSubmitter: number of splits:0
>
> Being new to this community, I thought it polite to introduce myself. I’m
> planning to return to software development via an MSc at Heriot Watt
> University in Edinburgh. My MSc project is based on Fosters Genetic
> Sequence Alignment. I have written a sequential version my goal is now to
> port it to Hadoop.
>
> Thanks in advance,
> Regards,
>
> Chris MacKenzie
>
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>

Mime
View raw message