hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Mawata <chris.maw...@gmail.com>
Subject Re: Partitioning and setup errors
Date Fri, 27 Jun 2014 13:11:00 GMT
The new Configuration() is suspicious. Are you setting configuration
information manually?
Chris
On Jun 27, 2014 5:16 AM, "Chris MacKenzie" <
studio@chrismackenziephotography.co.uk> wrote:

> Hi,
>
> I realise my previous question may have been a bit naïve and I also
> realise I am asking an awful lot here, any advice would be greatly
> appreciated.
>
>    - I have been using Hadoop 2.4 in local mode and am sticking to the mapreduce.*
>    side of the track.
>    - I am using a Custom Line reader to read each sequence into a Map
>    - I have a partitioner class which is testing the key from the map
>    class.
>    - I've tried debugging in eclipse with a breakpoint in the partitioner
>    class but getPartition(LongWritable mapKey, Text sequenceString, int
>    numReduceTasks) is not being called.
>
> Could there be any reason for that ?
>
> Because my map and reduce code works in local mode within eclipse, I wondered
> if I may get the partitioner to work if  I changed to Pseudo Distributed
> Mode exporting a runnable jar from Eclipse (Kepler)
>
> I have several faults On my own computer  Pseudo Distributed Mode and the
> university clusters Pseudo Distributed Mode which I set up. I’ve googled
> and read extensively but am not seeing a solution to any of these issues.
>
> I have this line:
> 14/06/27 11:45:27 WARN mapreduce.JobSubmitter: No job jar file set.  User
> classes may not be found. See Job or Job#setJar(String).
> My driver code is:
>
> private void doParallelConcordance() throws Exception {
>
>  Path inDir = new Path("input_sequences/10_sequences.txt");
>
> Path outDir = new Path("demo_output");
>
>
> Job job = Job.getInstance(new Configuration());
>
> job.setJarByClass(ParallelGeneticAlignment.class);
>
>  job.setOutputKeyClass(Text.class);
>
> job.setOutputValueClass(IntWritable.class);
>
>
> job.setInputFormatClass(CustomFileInputFormat.class);
>
> job.setMapperClass(ConcordanceMapper.class);
>
> job.setPartitionerClass(ConcordanceSequencePartitioner.class);
>
> job.setReducerClass(ConcordanceReducer.class);
>
>
> FileInputFormat.addInputPath(job, inDir);
>
> FileOutputFormat.setOutputPath(job, outDir);
>
>
> job.waitForCompletion(true)
>
> }
>
> On the university server I am getting this error:
> 4/06/27 11:45:40 INFO mapreduce.Job: Task Id :
> attempt_1403860966764_0003_m_000000_0, Status : FAILED
> Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceMapper not found
>
> On my machine the error is:
> 4/06/27 12:58:03 INFO mapreduce.Job: Task Id :
> attempt_1403864060032_0004_r_000000_2, Status : FAILED
> Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceReducer not found
>
> On the university server I get total paths to process:
> 14/06/27 11:45:27 INFO input.FileInputFormat: Total input paths to process
> : 1
> 14/06/27 11:45:28 INFO mapreduce.JobSubmitter: number of splits:1
>
> On my machine I get total paths to process:
> 14/06/27 12:57:09 INFO input.FileInputFormat: Total input paths to process
> : 0
> 14/06/27 12:57:36 INFO mapreduce.JobSubmitter: number of splits:0
>
> Being new to this community, I thought it polite to introduce myself. I’m
> planning to return to software development via an MSc at Heriot Watt
> University in Edinburgh. My MSc project is based on Fosters Genetic
> Sequence Alignment. I have written a sequential version my goal is now to
> port it to Hadoop.
>
> Thanks in advance,
> Regards,
>
> Chris MacKenzie
>

Mime
View raw message