hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "M. Dale" <medal...@yahoo.com>
Subject Re: Partitioning and setup errors
Date Thu, 03 Jul 2014 01:04:18 GMT
Two error messages seem to stand out:

1) Hadoop command-line option parsing not performed. Implement the Tool 
interface and execute your application with ToolRunner to remedy this.

YourDriver extends Configured implements Tool?

Configuration conf = getConf(); //gets the configuration as parsed by 
GenericOptionsParser via ToolRunner.run
Job job = Job.getInstance(conf);


2) No job jar file set.  User classes may not be found. See Job or 
Job#setJar(String).

job.setJarByClass(YourDriver.class); //see 
http://stackoverflow.com/questions/3912267/hadoop-query-regarding-setjarbyclass-method-of-job-class

Hope this helps,
Markus


On 06/29/2014 08:26 AM, Chris Mawata wrote:
>
> It is the part about implementing the tool interface that is the issue.
> Does your driver class implement Tool?
> Chris
>
> On Jun 29, 2014 8:22 AM, "Chris MacKenzie" 
> <studio@chrismackenziephotography.co.uk 
> <mailto:studio@chrismackenziephotography.co.uk>> wrote:
>
>     HI Vinod,
>
>     Thanks for your support. I’m packaging my application in Eclipse
>     (Kepler) as a runnable jar. I can get the application to debug
>     there in local mode.
>
>     The command line I am using for my app is:
>
>      "/usr/local/hadoop-2.4.0/jar_files $ hadoop jar
>     /usr/local/hadoop-2.4.0/jar_files/ParallelGeneticAlignment.jar
>     a.txt 10 10”
>
>     the jar permissions are -"rwxr-xr-x"
>
>     The three parameters on the end of the command line are arbitrary
>     commands from my sequential program which I will use as input when
>     I can get this to run.
>
>     I have downloaded the source and run a 2.4 version of WordCount,
>     it’s not getting past the map stage either.
>
>     I’m pasting in the whole job output.
>
>     Deleted demo_output
>
>     /usr/local/hadoop-2.4.0/jar_files $ hadoop jar
>     /usr/local/hadoop-2.4.0/jar_files/ParallelGeneticAlignment.jar
>     a.txt 10 10
>
>     Start time = 1404044126713
>
>     SLF4J: Class path contains multiple SLF4J bindings.
>
>     SLF4J: Found binding in [rsrc:org/slf4j/impl/StaticLoggerBinder.class]
>
>     SLF4J: Found binding in
>     [jar:rsrc:slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>
>     SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for
>     an explanation.
>
>     SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
>
>     14/06/29 13:16:06 WARN util.NativeCodeLoader: Unable to load
>     native-hadoop library for your platform... using builtin-java
>     classes where applicable
>
>     14/06/29 13:16:31 INFO client.RMProxy: Connecting to
>     ResourceManager at /0.0.0.0:8032 <http://0.0.0.0:8032>
>
>     14/06/29 13:16:46 WARN mapreduce.JobSubmitter: Hadoop command-line
>     option parsing not performed. Implement the Tool interface and
>     execute your application with ToolRunner to remedy this.
>
>     14/06/29 13:16:46 WARN mapreduce.JobSubmitter: No job jar file
>     set.  User classes may not be found. See Job or Job#setJar(String).
>
>     14/06/29 13:16:46 INFO input.FileInputFormat: Total input paths to
>     process : 0
>
>     14/06/29 13:17:10 INFO mapreduce.JobSubmitter: number of splits:0
>
>     14/06/29 13:17:13 INFO mapreduce.JobSubmitter: Submitting tokens
>     for job: job_1403953980618_0009
>
>     14/06/29 13:17:13 INFO mapred.YARNRunner: Job jar is not present.
>     Not adding any jar to the list of resources.
>
>     14/06/29 13:17:13 INFO impl.YarnClientImpl: Submitted application
>     application_1403953980618_0009
>
>     14/06/29 13:17:13 INFO mapreduce.Job: The url to track the job:
>     http://admins-MacBook-Pro.local:8088/proxy/application_1403953980618_0009/
>
>     14/06/29 13:17:13 INFO mapreduce.Job: Running job:
>     job_1403953980618_0009
>
>     14/06/29 13:17:19 INFO mapreduce.Job: Job job_1403953980618_0009
>     running in uber mode : false
>
>     14/06/29 13:17:19 INFO mapreduce.Job:  map 0% reduce 0%
>
>     14/06/29 13:17:23 INFO mapreduce.Job: Task Id :
>     attempt_1403953980618_0009_r_000000_0, Status : FAILED
>
>     Error: java.lang.RuntimeException:
>     java.lang.ClassNotFoundException: Class
>     par.gene.align.concordance.ConcordanceReducer not found
>
>     at
>     org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)
>
>     at
>     org.apache.hadoop.mapreduce.task.JobContextImpl.getReducerClass(JobContextImpl.java:210)
>
>     at
>     org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:611)
>
>     at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
>
>     at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
>
>     at java.security.AccessController.doPrivileged(Native Method)
>
>     at javax.security.auth.Subject.doAs(Subject.java:422)
>
>     at
>     org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>
>     at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
>
>     Caused by: java.lang.ClassNotFoundException: Class
>     par.gene.align.concordance.ConcordanceReducer not found
>
>     at
>     org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)
>
>     at
>     org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)
>
>     ... 8 more
>
>
>     14/06/29 13:17:28 INFO mapreduce.Job: Task Id :
>     attempt_1403953980618_0009_r_000000_1, Status : FAILED
>
>     Error: java.lang.RuntimeException:
>     java.lang.ClassNotFoundException: Class
>     par.gene.align.concordance.ConcordanceReducer not found
>
>     at
>     org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)
>
>     at
>     org.apache.hadoop.mapreduce.task.JobContextImpl.getReducerClass(JobContextImpl.java:210)
>
>     at
>     org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:611)
>
>     at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
>
>     at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
>
>     at java.security.AccessController.doPrivileged(Native Method)
>
>     at javax.security.auth.Subject.doAs(Subject.java:422)
>
>     at
>     org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>
>     at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
>
>     Caused by: java.lang.ClassNotFoundException: Class
>     par.gene.align.concordance.ConcordanceReducer not found
>
>     at
>     org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)
>
>     at
>     org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)
>
>     ... 8 more
>
>
>     14/06/29 13:17:32 INFO mapreduce.Job: Task Id :
>     attempt_1403953980618_0009_r_000000_2, Status : FAILED
>
>     Error: java.lang.RuntimeException:
>     java.lang.ClassNotFoundException: Class
>     par.gene.align.concordance.ConcordanceReducer not found
>
>     at
>     org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)
>
>     at
>     org.apache.hadoop.mapreduce.task.JobContextImpl.getReducerClass(JobContextImpl.java:210)
>
>     at
>     org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:611)
>
>     at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
>
>     at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
>
>     at java.security.AccessController.doPrivileged(Native Method)
>
>     at javax.security.auth.Subject.doAs(Subject.java:422)
>
>     at
>     org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>
>     at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
>
>     Caused by: java.lang.ClassNotFoundException: Class
>     par.gene.align.concordance.ConcordanceReducer not found
>
>     at
>     org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)
>
>     at
>     org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)
>
>     ... 8 more
>
>
>     14/06/29 13:17:37 INFO mapreduce.Job:  map 0% reduce 100%
>
>     14/06/29 13:17:37 INFO mapreduce.Job: Job job_1403953980618_0009
>     failed with state FAILED due to: Task failed
>     task_1403953980618_0009_r_000000
>
>     Job failed as tasks failed. failedMaps:0 failedReduces:1
>
>
>     14/06/29 13:18:04 INFO mapreduce.Job: Counters: 7
>
>     Job Counters
>
>     Failed reduce tasks=4
>
>     Launched reduce tasks=4
>
>     Total time spent by all maps in occupied slots (ms)=0
>
>     Total time spent by all reduces in occupied slots (ms)=7786
>
>     Total time spent by all reduce tasks (ms)=7786
>
>     Total vcore-seconds taken by all reduce tasks=7786
>
>     Total megabyte-seconds taken by all reduce tasks=7972864
>
>     14/06/29 13:18:05 INFO ipc.Client: Retrying connect to server:
>     admins-MacBook-Pro.local/192.168.0.5:53193
>     <http://192.168.0.5:53193>. Already tried 0 time(s); retry policy
>     is RetryUpToMaximumCountWithFixedSleep(maxRetries=3,
>     sleepTime=1000 MILLISECONDS)
>
>     14/06/29 13:18:06 INFO ipc.Client: Retrying connect to server:
>     admins-MacBook-Pro.local/192.168.0.5:53193
>     <http://192.168.0.5:53193>. Already tried 1 time(s); retry policy
>     is RetryUpToMaximumCountWithFixedSleep(maxRetries=3,
>     sleepTime=1000 MILLISECONDS)
>
>     14/06/29 13:18:07 INFO ipc.Client: Retrying connect to server:
>     admins-MacBook-Pro.local/192.168.0.5:53193
>     <http://192.168.0.5:53193>. Already tried 2 time(s); retry policy
>     is RetryUpToMaximumCountWithFixedSleep(maxRetries=3,
>     sleepTime=1000 MILLISECONDS)
>
>     14/06/29 13:18:07 INFO mapred.ClientServiceDelegate: Application
>     state is completed. FinalApplicationStatus=FAILED. Redirecting to
>     job history server
>
>     End time = 1404044288031
>
>     Elapsed time = 161318 ms
>
>     Finished
>
>     /usr/local/hadoop-2.4.0/jar_files $
>
>     Warm regards.
>
>     Chris
>
>
>     From: Vinod Kumar Vavilapalli <vinodkv@hortonworks.com
>     <mailto:vinodkv@hortonworks.com>>
>     Reply-To: <user@hadoop.apache.org <mailto:user@hadoop.apache.org>>
>     Date: Sunday, 29 June 2014 00:20
>     To: <user@hadoop.apache.org <mailto:user@hadoop.apache.org>>
>     Subject: Re: Partitioning and setup errors
>
>     What is happening is the client is not able to pick up the right
>     jar to push to the cluster. It looks in the class-path for the jar
>     that contains the class ParallelGeneticAlignment.
>
>     How are you packaging your code? How are your running your job -
>     paste the command line?
>
>     +Vinod
>
>     On Jun 27, 2014, at 5:15 AM, Chris MacKenzie
>     <studio@chrismackenziephotography.co.uk
>     <mailto:studio@chrismackenziephotography.co.uk>> wrote:
>
>>     Hi,
>>
>>     I realise my previous question may have been a bit naïve and I
>>     also realise I am asking an awful lot here, any advice would be
>>     greatly appreciated.
>>
>>       * I have been using Hadoop 2.4 in local mode and am sticking to
>>         the mapreduce.* side of the track.
>>       * I am using a Custom Line reader to read each sequence into a Map
>>       * I have a partitioner class which is testing the key from the
>>         map class.
>>       * I've tried debugging in eclipse with a breakpoint in the
>>         partitioner class but getPartition(LongWritable mapKey, Text
>>         sequenceString, intnumReduceTasks) is not being called.
>>
>>     Could there be any reason for that ?
>>
>>     Because my map and reduce code works in local mode within
>>     eclipse, I wondered if I may get the partitioner to work if
>>     I changed to Pseudo Distributed Mode exporting a runnable jar
>>     from Eclipse (Kepler)
>>
>>     I have several faults On my own computer  Pseudo Distributed Mode
>>     and the university clusters Pseudo Distributed Mode which I set
>>     up. I’ve googled and read extensively but am not seeing a
>>     solution to any of these issues.
>>
>>     I have this line:
>>     14/06/27 11:45:27 WARN mapreduce.JobSubmitter: No job jar file
>>     set.  User classes may not be found. See Job or Job#setJar(String).
>>     My driver code is:
>>     private void doParallelConcordance() throws Exception {
>>
>>
>>     Path inDir= newPath("input_sequences/10_sequences.txt");
>>     Path outDir = new Path("demo_output");
>>
>>     Job job = Job.getInstance(new Configuration());
>>     job.setJarByClass(ParallelGeneticAlignment.class);
>>
>>     job.setOutputKeyClass(Text.class);
>>     job.setOutputValueClass(IntWritable.class);
>>
>>     job.setInputFormatClass(CustomFileInputFormat.class);
>>     job.setMapperClass(ConcordanceMapper.class);
>>     job.setPartitionerClass(ConcordanceSequencePartitioner.class);
>>     job.setReducerClass(ConcordanceReducer.class);
>>
>>     FileInputFormat.addInputPath(job, inDir);
>>     FileOutputFormat.setOutputPath(job, outDir);
>>
>>     job.waitForCompletion(true)
>>     }
>>
>>     On the university server I am getting this error:
>>     4/06/27 11:45:40 INFO mapreduce.Job: Task Id :
>>     attempt_1403860966764_0003_m_000000_0, Status : FAILED
>>     Error: java.lang.RuntimeException:
>>     java.lang.ClassNotFoundException: Class
>>     par.gene.align.concordance.ConcordanceMapper not found
>>
>>     On my machine the error is:
>>     4/06/27 12:58:03 INFO mapreduce.Job: Task Id :
>>     attempt_1403864060032_0004_r_000000_2, Status : FAILED
>>     Error: java.lang.RuntimeException:
>>     java.lang.ClassNotFoundException: Class
>>     par.gene.align.concordance.ConcordanceReducer not found
>>
>>     On the university server I get total paths to process:
>>     14/06/27 11:45:27 INFO input.FileInputFormat: Total input paths
>>     to process : 1
>>     14/06/27 11:45:28 INFO mapreduce.JobSubmitter: number of splits:1
>>
>>     On my machine I get total paths to process:
>>     14/06/27 12:57:09 INFO input.FileInputFormat: Total input paths
>>     to process : 0
>>     14/06/27 12:57:36 INFO mapreduce.JobSubmitter: number of splits:0
>>
>>     Being new to this community, I thought it polite to introduce
>>     myself. I’m planning to return to software development via an MSc
>>     at Heriot Watt University in Edinburgh. My MSc project is based
>>     on Fosters Genetic Sequence Alignment. I have written a
>>     sequential version my goal is now to port it to Hadoop.
>>
>>     Thanks in advance,
>>     Regards,
>>
>>     Chris MacKenzie
>
>
>     CONFIDENTIALITY NOTICE
>     NOTICE: This message is intended for the use of the individual or
>     entity to which it is addressed and may contain information that
>     is confidential, privileged and exempt from disclosure under
>     applicable law. If the reader of this message is not the intended
>     recipient, you are hereby notified that any printing, copying,
>     dissemination, distribution, disclosure or forwarding of this
>     communication is strictly prohibited. If you have received this
>     communication in error, please contact the sender immediately and
>     delete it from your system. Thank You.
>


Mime
View raw message