Return-Path: X-Original-To: apmail-hadoop-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id CD1671121E for ; Sat, 28 Jun 2014 09:36:47 +0000 (UTC) Received: (qmail 94135 invoked by uid 500); 28 Jun 2014 09:36:42 -0000 Delivered-To: apmail-hadoop-user-archive@hadoop.apache.org Received: (qmail 94025 invoked by uid 500); 28 Jun 2014 09:36:42 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 94014 invoked by uid 99); 28 Jun 2014 09:36:42 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 28 Jun 2014 09:36:42 +0000 X-ASF-Spam-Status: No, hits=2.9 required=5.0 tests=HTML_MESSAGE,MIME_QP_LONG_LINE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL,T_REMOTE_IMAGE X-Spam-Check-By: apache.org Received-SPF: neutral (nike.apache.org: local policy) Received: from [79.99.40.130] (HELO auth-smtp-01.streamline.net) (79.99.40.130) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 28 Jun 2014 09:36:39 +0000 Received: by auth-smtp-01.streamline.net (Postfix, from userid 500) id CB14F16D0F75; Sat, 28 Jun 2014 10:36:13 +0100 (BST) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on auth-smtp-01.streamline.net Received: from [192.168.0.5] (cpc5-bahd4-2-0-cust46.14-2.cable.virginm.net [80.1.220.47]) (Authenticated sender: studio@chrismackenziephotography.co.uk) by auth-smtp-01.streamline.net (Postfix) with ESMTP id 323CD16D0F75 for ; Sat, 28 Jun 2014 10:36:07 +0100 (BST) User-Agent: Microsoft-MacOutlook/14.4.2.140509 Date: Sat, 28 Jun 2014 10:36:06 +0100 Subject: Re: Partitioning and setup errors From: Chris MacKenzie To: Message-ID: Thread-Topic: Partitioning and setup errors References: In-Reply-To: Mime-version: 1.0 Content-type: multipart/alternative; boundary="B_3486796569_464427" X-Virus-Checked: Checked by ClamAV on apache.org > This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. --B_3486796569_464427 Content-type: text/plain; charset="ISO-8859-1" Content-transfer-encoding: quoted-printable HI Chris, I=B9m away from my books for the weekend. Is that call (extends Configurable implements Tool) a Hadoop 2 call. Would that mean that I am better sticking with Hadoop 1.x ? Warm regards. Chris Expert in all aspects of photography telephone: 0131 332 6967 email: studio@chrismackenziephotography.co.uk corporate: www.chrismackenziephotography.co.uk From: Chris Mawata Reply-To: Date: Friday, 27 June 2014 23:46 To: Subject: Re: Partitioning and setup errors Probably my fault. I was looking for the extends Configurable implements Tool part. I will double check when I get home rather than send you on a wild goose chase. Cheers Chris On Jun 27, 2014 8:16 AM, "Chris MacKenzie" wrote: > Hi, >=20 > I realise my previous question may have been a bit na=EFve and I also reali= se I > am asking an awful lot here, any advice would be greatly appreciated. > * I have been using Hadoop 2.4 in local mode and am sticking to the > mapreduce.* side of the track. > * I am using a Custom Line reader to read each sequence into a Map > * I have a partitioner class which is testing the key from the map class. > * I've tried debugging in eclipse with a breakpoint in the partitioner cl= ass > but getPartition(LongWritable mapKey, Text sequenceString, int numReduceT= asks) > is not being called. > Could there be any reason for that ? >=20 > Because my map and reduce code works in local mode within eclipse, I wond= ered > if I may get the partitioner to work if I changed to Pseudo Distributed = Mode > exporting a runnable jar from Eclipse (Kepler) >=20 > I have several faults On my own computer Pseudo Distributed Mode and the > university clusters Pseudo Distributed Mode which I set up. I=B9ve googled = and > read extensively but am not seeing a solution to any of these issues. >=20 > I have this line: > 14/06/27 11:45:27 WARN mapreduce.JobSubmitter: No job jar file set. User > classes may not be found. See Job or Job#setJar(String). > My driver code is: > private void doParallelConcordance() throws Exception { >=20 > Path inDir =3D new Path("input_sequences/10_sequences.txt"); >=20 > Path outDir =3D new Path("demo_output"); >=20 >=20 >=20 > Job job =3D Job.getInstance(new Configuration()); >=20 > job.setJarByClass(ParallelGeneticAlignment.class); >=20 > job.setOutputKeyClass(Text.class); >=20 > job.setOutputValueClass(IntWritable.class); >=20 >=20 >=20 > job.setInputFormatClass(CustomFileInputFormat.class); >=20 > job.setMapperClass(ConcordanceMapper.class); >=20 > job.setPartitionerClass(ConcordanceSequencePartitioner.class); >=20 > job.setReducerClass(ConcordanceReducer.class); >=20 >=20 >=20 > FileInputFormat.addInputPath(job, inDir); >=20 > FileOutputFormat.setOutputPath(job, outDir); >=20 >=20 >=20 > job.waitForCompletion(true) >=20 > } >=20 >=20 > On the university server I am getting this error: > 4/06/27 11:45:40 INFO mapreduce.Job: Task Id : > attempt_1403860966764_0003_m_000000_0, Status : FAILED > Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Clas= s > par.gene.align.concordance.ConcordanceMapper not found >=20 > On my machine the error is: > 4/06/27 12:58:03 INFO mapreduce.Job: Task Id : > attempt_1403864060032_0004_r_000000_2, Status : FAILED > Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Clas= s > par.gene.align.concordance.ConcordanceReducer not found >=20 > On the university server I get total paths to process: > 14/06/27 11:45:27 INFO input.FileInputFormat: Total input paths to proces= s : 1 > 14/06/27 11:45:28 INFO mapreduce.JobSubmitter: number of splits:1 >=20 > On my machine I get total paths to process: > 14/06/27 12:57:09 INFO input.FileInputFormat: Total input paths to proces= s : 0 > 14/06/27 12:57:36 INFO mapreduce.JobSubmitter: number of splits:0 >=20 > Being new to this community, I thought it polite to introduce myself. I=B9m > planning to return to software development via an MSc at Heriot Watt > University in Edinburgh. My MSc project is based on Fosters Genetic Seque= nce > Alignment. I have written a sequential version my goal is now to port it = to > Hadoop. >=20 > Thanks in advance, > Regards, >=20 > Chris MacKenzie --B_3486796569_464427 Content-type: text/html; charset="ISO-8859-1" Content-transfer-encoding: quoted-printable
HI Chris,

=
I’m away from my books for the weekend. Is that call (exten= ds Configurable implements Tool) a Hadoop 2 call. Would that mean that I am = better sticking with Hadoop 1.x ?

Warm regard= s.

Chris

3D"Chris

Expert in all a= spects of photography

telephone: 0131 332 6967

email: studio@chrismackenziephotography.co.uk

<= p style=3D"font-family: Times; font-size: medium; line-height: 0.5;">corporate= : www.chrismackenziephotography.co.uk=

3D"Google" =3D"Twitter" 3D"Facebook" =3D"Linked 3D"Pinterest"=

From: Chris Mawata <chris.mawata@gma= il.com>
Reply-To: <user@hadoop.apache.org>
Date: Friday, 27 June 2014 23:46
To: <user@hadoop.apache.org>
Subject:= Re: Partitioning and setup errors

Probably my fault. I was looking for the
extends Configurable implements Tool
part. I will double check when I get home rather than send you on a wild go= ose chase.
Cheers
Chris

On Jun 27, 2014 8:16 AM, "Chris MacKenzie= " <studio@chrisma= ckenziephotography.co.uk> wrote:
Hi,
<= br>
I realise my previ= ous question may have been a bit na=EFve and I also realise I am asking an awf= ul lot here, any advice would be greatly appreciated.
  • I<= font color=3D"#444444"> ha= ve been using Hadoop 2.4 in local mode and am sticking to the mapreduce.* side of the track.
  • I am using a Cu= stom Line reader to read each sequence into a Map
  • I have a partitioner = class which is testing the key from the map class. 
    • <= li>I've tried deb= ugging in eclipse with a breakpoint in the partitioner class but getPartition(LongWritable mapKey, Text = sequenceString, int numReduceTasks) is not being called.
Could there be any reas= on for that ?

Because my map and reduce code works in local mode within ecl= ipse, I wondered if&= nbsp;I may get the partit= ioner to work if  = I = changed to Pseudo Distributed Mode exporting a runnable jar from Eclipse (Ke= pler)

I= =  have several faults On my own computer &nbs= p;Pseudo Distributed Mode and the university clusters Pseudo Dis= tributed Mode which I set up. I’ve googled and read extensively but am not seeing= a solution to any of these issues.

I have this line:
14/06/27 11:45:27 WARN mapreduce.JobSubmi= tter: No job jar file set.  User classes may not be found. See Job or J= ob#setJar(String).
My driver code is:

private void = doParallelConcordance() throws Exception = {

Path inDir =3D <= /span>new Pat= h("input_sequences/10_sequences.txt");

Path outDir =3D new Path("demo_output");


Job job =3D Job.get= Instance(new Configuration());

=

job.setJa= rByClass(ParallelGeneticAlignment.class);=

=

job.setOutputKeyClass(Text.cla= ss);

job.setOutputValueClass(IntWritable.class);


job.setInputFormatClass(CustomFileInputF= ormat.class);

job.setM= apperClass(ConcordanceMapper.class);

job.setPartitionerClass(ConcordanceSequencePartitioner.class);

job.setReducerClass(ConcordanceReduce= r.class);


FileInputFormat.addInputPath(job, inDir);

=

FileOutputFormat.setOutputPath(job, outDir);


job.waitForCompletion(true)

}


On the university server I am getting this error:
<= div>
4/06/27 11:45:40 INFO map= reduce.Job: Task Id : attempt_1403860966764_0003_m_000000_0, Status : FAILED=
Error: java.lang= .RuntimeException: java.lang.ClassNotFoundException: Class par.gene.align.co= ncordance.ConcordanceMapper not found

On my machine the error is:
4/06/27 12:58:03 INFO mapreduce.Job: Task Id = : attempt_1403864060032_0004_r_000000_2, Status : FAILED
Error: java.lang.RuntimeException: j= ava.lang.ClassNotFoundException: Class par.gene.align.concordance.Concordanc= eReducer not found

On t= he university server I get total paths to process:
14/06/27 11:45:27 INFO input.FileInputForm= at: Total input paths to process : 1
14/06/27 11:45:28 INFO mapreduce.JobSubmitter: number of= splits:1

On my machine I get tot= al paths to process:
1= 4/06/27 12:57:09 INFO input.FileInputFormat: Total input paths to process : = 0
14/06/27 12:57:36 IN= FO mapreduce.JobSubmitter: number of splits:0

Being new to this community, I thought it polite to introduce myself.= I’m planning to return to software development via an MSc at Heriot W= att University in Edinburgh. My MSc project is based on Fosters Genetic Sequ= ence Alignment. I have written a sequential version my goal is now to port i= t to Hadoop.

=
T= hanks in advance, 
Regards,
<= br>
Chris MacKenz= ie
= --B_3486796569_464427--