hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vinod Kumar Vavilapalli <vino...@hortonworks.com>
Subject Re: Partitioning and setup errors
Date Sat, 28 Jun 2014 23:20:08 GMT
What is happening is the client is not able to pick up the right jar to push to the cluster.
It looks in the class-path for the jar that contains the class ParallelGeneticAlignment.

How are you packaging your code? How are your running your job - paste the command line?


On Jun 27, 2014, at 5:15 AM, Chris MacKenzie <studio@chrismackenziephotography.co.uk>

> Hi,
> I realise my previous question may have been a bit naïve and I also realise I am asking
an awful lot here, any advice would be greatly appreciated.
> I have been using Hadoop 2.4 in local mode and am sticking to the mapreduce.* side of
the track.
> I am using a Custom Line reader to read each sequence into a Map
> I have a partitioner class which is testing the key from the map class. 
> I've tried debugging in eclipse with a breakpoint in the partitioner class but getPartition(LongWritable
mapKey, Text sequenceString, int numReduceTasks) is not being called.
> Could there be any reason for that ?
> Because my map and reduce code works in local mode within eclipse, I wondered if I may
get the partitioner to work if  I changed to Pseudo Distributed Mode exporting a runnable
jar from Eclipse (Kepler)
> I have several faults On my own computer  Pseudo Distributed Mode and the university
clusters Pseudo Distributed Mode which I set up. I’ve googled and read extensively but am
not seeing a solution to any of these issues.
> I have this line:
> 14/06/27 11:45:27 WARN mapreduce.JobSubmitter: No job jar file set.  User classes may
not be found. See Job or Job#setJar(String).
> My driver code is:
> 	private void doParallelConcordance() throws Exception {
> 		Path inDir = new Path("input_sequences/10_sequences.txt");
> 		Path outDir = new Path("demo_output");
> 		Job job = Job.getInstance(new Configuration());
> 		job.setJarByClass(ParallelGeneticAlignment.class);
> 		job.setOutputKeyClass(Text.class);
> 		job.setOutputValueClass(IntWritable.class);
> 		job.setInputFormatClass(CustomFileInputFormat.class);
> 		job.setMapperClass(ConcordanceMapper.class);
> 		job.setPartitionerClass(ConcordanceSequencePartitioner.class);
> 		job.setReducerClass(ConcordanceReducer.class);
> 		FileInputFormat.addInputPath(job, inDir);
> 		FileOutputFormat.setOutputPath(job, outDir);
> 		job.waitForCompletion(true)
> 	}
> On the university server I am getting this error:
> 4/06/27 11:45:40 INFO mapreduce.Job: Task Id : attempt_1403860966764_0003_m_000000_0,
Status : FAILED
> Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class par.gene.align.concordance.ConcordanceMapper
not found
> On my machine the error is:
> 4/06/27 12:58:03 INFO mapreduce.Job: Task Id : attempt_1403864060032_0004_r_000000_2,
Status : FAILED
> Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class par.gene.align.concordance.ConcordanceReducer
not found
> On the university server I get total paths to process:
> 14/06/27 11:45:27 INFO input.FileInputFormat: Total input paths to process : 1
> 14/06/27 11:45:28 INFO mapreduce.JobSubmitter: number of splits:1
> On my machine I get total paths to process:
> 14/06/27 12:57:09 INFO input.FileInputFormat: Total input paths to process : 0
> 14/06/27 12:57:36 INFO mapreduce.JobSubmitter: number of splits:0
> Being new to this community, I thought it polite to introduce myself. I’m planning
to return to software development via an MSc at Heriot Watt University in Edinburgh. My MSc
project is based on Fosters Genetic Sequence Alignment. I have written a sequential version
my goal is now to port it to Hadoop.
> Thanks in advance, 
> Regards,
> Chris MacKenzie

NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

View raw message