hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris MacKenzie <stu...@chrismackenziephotography.co.uk>
Subject Re: Partitioning and setup errors
Date Sat, 28 Jun 2014 09:36:06 GMT
HI Chris,

I¹m away from my books for the weekend. Is that call (extends Configurable
implements Tool) a Hadoop 2 call. Would that mean that I am better sticking
with Hadoop 1.x ?
Warm regards.
Chris
 <http://www.chrismackenziephotography.co.uk/>
Expert in all aspects of photography
telephone: 0131 332 6967 <tel:0131 332 6967>

email: studio@chrismackenziephotography.co.uk
corporate: www.chrismackenziephotography.co.uk
<http://www.chrismackenziephotography.co.uk/>
 <http://plus.google.com/+ChrismackenziephotographyCoUk/posts>
<http://twitter.com/#!/MacKenzieStudio>
<http://www.facebook.com/pages/Chris-MacKenzie-Photography/145946284250>
<http://www.linkedin.com/in/chrismackenziephotography/>
<http://pinterest.com/ChrisMacKenzieP/>

From:  Chris Mawata <chris.mawata@gmail.com>
Reply-To:  <user@hadoop.apache.org>
Date:  Friday, 27 June 2014 23:46
To:  <user@hadoop.apache.org>
Subject:  Re: Partitioning and setup errors


Probably my fault. I was looking for the
extends Configurable implements Tool
part. I will double check when I get home rather than send you on a wild
goose chase.
Cheers
Chris

On Jun 27, 2014 8:16 AM, "Chris MacKenzie"
<studio@chrismackenziephotography.co.uk> wrote:
> Hi,
> 
> I realise my previous question may have been a bit naïve and I also realise I
> am asking an awful lot here, any advice would be greatly appreciated.
> * I have been using Hadoop 2.4 in local mode and am sticking to the
> mapreduce.* side of the track.
> * I am using a Custom Line reader to read each sequence into a Map
> * I have a partitioner class which is testing the key from the map class.
> * I've tried debugging in eclipse with a breakpoint in the partitioner class
> but getPartition(LongWritable mapKey, Text sequenceString, int numReduceTasks)
> is not being called.
> Could there be any reason for that ?
> 
> Because my map and reduce code works in local mode within eclipse, I wondered
> if I may get the partitioner to work if  I changed to Pseudo Distributed Mode
> exporting a runnable jar from Eclipse (Kepler)
> 
> I have several faults On my own computer  Pseudo Distributed Mode and the
> university clusters Pseudo Distributed Mode which I set up. I¹ve googled and
> read extensively but am not seeing a solution to any of these issues.
> 
> I have this line:
> 14/06/27 11:45:27 WARN mapreduce.JobSubmitter: No job jar file set.  User
> classes may not be found. See Job or Job#setJar(String).
> My driver code is:
> private void doParallelConcordance() throws Exception {
> 
> Path inDir = new Path("input_sequences/10_sequences.txt");
> 
> Path outDir = new Path("demo_output");
> 
> 
> 
> Job job = Job.getInstance(new Configuration());
> 
> job.setJarByClass(ParallelGeneticAlignment.class);
> 
> job.setOutputKeyClass(Text.class);
> 
> job.setOutputValueClass(IntWritable.class);
> 
> 
> 
> job.setInputFormatClass(CustomFileInputFormat.class);
> 
> job.setMapperClass(ConcordanceMapper.class);
> 
> job.setPartitionerClass(ConcordanceSequencePartitioner.class);
> 
> job.setReducerClass(ConcordanceReducer.class);
> 
> 
> 
> FileInputFormat.addInputPath(job, inDir);
> 
> FileOutputFormat.setOutputPath(job, outDir);
> 
> 
> 
> job.waitForCompletion(true)
> 
> }
> 
> 
> On the university server I am getting this error:
> 4/06/27 11:45:40 INFO mapreduce.Job: Task Id :
> attempt_1403860966764_0003_m_000000_0, Status : FAILED
> Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceMapper not found
> 
> On my machine the error is:
> 4/06/27 12:58:03 INFO mapreduce.Job: Task Id :
> attempt_1403864060032_0004_r_000000_2, Status : FAILED
> Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
> par.gene.align.concordance.ConcordanceReducer not found
> 
> On the university server I get total paths to process:
> 14/06/27 11:45:27 INFO input.FileInputFormat: Total input paths to process : 1
> 14/06/27 11:45:28 INFO mapreduce.JobSubmitter: number of splits:1
> 
> On my machine I get total paths to process:
> 14/06/27 12:57:09 INFO input.FileInputFormat: Total input paths to process : 0
> 14/06/27 12:57:36 INFO mapreduce.JobSubmitter: number of splits:0
> 
> Being new to this community, I thought it polite to introduce myself. I¹m
> planning to return to software development via an MSc at Heriot Watt
> University in Edinburgh. My MSc project is based on Fosters Genetic Sequence
> Alignment. I have written a sequential version my goal is now to port it to
> Hadoop.
> 
> Thanks in advance,
> Regards,
> 
> Chris MacKenzie



Mime
View raw message