hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Runping Qi" <runp...@yahoo-inc.com>
Subject RE: Question about valueaggregators in 0.14.1...
Date Sat, 13 Oct 2007 03:28:21 GMT


I am sorry I overlooked something.
Try to add the following:

job.setJarByClass(MyValueAggregatorJob.class);
JobClient.runJob(job);

To your main.

Runping

> -----Original Message-----
> From: C G [mailto:parallelguy@yahoo.com]
> Sent: Friday, October 12, 2007 8:05 PM
> To: hadoop-user@lucene.apache.org
> Subject: RE: Question about valueaggregators in 0.14.1...
> 
> Hi Runping and All:
> 
>   I took your suggestion, renamed my original code to testAggregatorPlugin
> and wrote a class testAggregator which contained the main as you show
> here.  It also fails with the same traceback...so it seems something is
> profoundly wrong here.
> 
>   Is anybody else using the aggregation classes in 0.14.1 with success?
> 
>   This is ironic and frustrating...I have several other programs which use
> map/reduce that are quite complex and those are running just fine.  This
> little plugin is all that stands in the way of delivering the system :-(.
> 
>   Any other comments/help, etc. most welcome...
> 
>   Thanks,
>   C G
> 
> Runping Qi <runping@yahoo-inc.com> wrote:
> 
> Hadoop ignored the jar file you specified.
> This is a known problem andhopefully HADOOP-1622
> (https://issues.apache.org/jira/browse/HADOOP-1622) will fix the problem.
> 
> In the meantime, you can create a class like below and add it to your jar:
> 
> package your_package;
> 
> 
> public class MyValueAggregatorJob{
> 
> public static void main(String args[]) throws IOException {
> JobConf job = ValueAggregatorJob.createValueAggregatorJob(args);
> JobClient.runJob(job);
> }
> }
> 
> 
> And use the above class as the main class when you submit hadoop job:
> bin/hadoop jar hadoop-0.14.1-dev-company.jar
> your_package.MyValueAggregatorJob /input/set1 /output/set1 8
> textinputformat
> testAggregator.xml
> 
> 
> Hope this solve your problem.
> 
> Runping
> 
> 
> 
> 
> > -----Original Message-----
> > From: C G [mailto:parallelguy@yahoo.com]
> > Sent: Friday, October 12, 2007 2:43 PM
> > To: hadoop-user@lucene.apache.org
> > Subject: Question about valueaggregators in 0.14.1...
> >
> > Hi All:
> >
> > I am tearing out my hair trying to get a simple valueaggregator to run.
> > This seems like an
> > easy enough thing, but I am consistently getting hit with
> > ClassNotFoundException which
> > makes no sense to me. Any help would be immensely appreciated. Note
> > that I've had
> > valueaggregators running on 0.13.*. I'm a little concerned there is a
> bug
> > in 0.14.1. On the
> > other hand, maybe I'm just blind to something obvious and just need a
> > boot to the head...
> >
> > The environment is hadoop-0.14.1. I am running Java 1.6:
> > $ java -version
> > java version "1.6.0_02"
> > Java(TM) SE Runtime Environment (build 1.6.0_02-b05)
> > Java HotSpot(TM) Server VM (build 1.6.0_02-b05, mixed mode)
> >
> > The class name is called testAggregator, and exists in my jar file:
> > $ jar -tvf hadoop-0.14.1-dev-company.jar | grep testAggregator
> > 1826 Fri Oct 12 16:30:14 EDT 2007
> > com/company/hadoop/metrics/testAggregator.class
> >
> > My XML file is testAggregator.xml and looks like:
> >
> >
> >
> >
> 
> > aggregator.descriptor.num
> > 1
> >
> 
> >
> 
> > aggregator.descriptor.0
> > UserDefined,com.company.hadoop.metrics.testAggregator
> >
> 
> >
> >
> > My code couldn't be simpler:
> >
> > package com.company.hadoop.metrics;
> > // snip a bunch of imports
> > public class testAggregator extends ValueAggregatorBaseDescriptor {
> > public ArrayList generateKeyValPairs(Object key, Object val) {
> > String fields[] = val.toString().split("\t");
> > ArrayList retv = new ArrayList();
> > retv.add(generateEntry(LONG_VALUE_SUM, fields[ 6 ], ONE));
> > retv.add(generateEntry(UNIQ_VALUE_COUNT, fields[ 6 ], new Text(
> > fields[ 2 ])));
> > return retv;
> > }
> > public void configure(JobConf job) {}
> > }
> >
> > I invoke things this way:
> >
> > bin/hadoop jar hadoop-0.14.1-dev-company.jar
> > org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorJob /input/set1
> > /output/set1 8 textinputformat testAggregator.xml
> >
> > All my jobs fail with a ClassNotFoundException like the following
> > traceback:
> >
> > 2007-10-12 16:56:14,227 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
> > Initializing JVM Metrics with processName=MAP, sessionId=
> > 2007-10-12 16:56:14,277 INFO org.apache.hadoop.mapred.MapTask:
> > numReduceTasks: 8
> > 2007-10-12 16:56:14,299 WARN org.apache.hadoop.mapred.TaskTracker: Error
> > running child
> > java.lang.RuntimeException: java.lang.ClassNotFoundException:
> > com.company.hadoop.metrics.testAggregator
> > at
> >
> >
> org.apache.hadoop.mapred.lib.aggregate.UserDefinedValueAggregatorDescripto
> > r.createInstance
> > (UserDefinedValueAggregatorDescriptor.java:56)
> > at
> >
> org.apache.hadoop.mapred.lib.aggregate.UserDefinedValueAggregatorDescripto
> > r.createAggregator(UserDefinedValueAggregatorDescriptor.java:63)
> > at
> >
> org.apache.hadoop.mapred.lib.aggregate.UserDefinedValueAggregatorDescripto
> > r.(UserDefinedValueAggregatorDescriptor.java:75)
> > at
> >
> org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorJobBase.getValueAggr
> > egatorDescriptor(ValueAggregatorJobBase.java:49)
> > at
> >
> org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorJobBase.getAggregato
> > rDescriptors(ValueAggregatorJobBase.java:60)
> > at
> >
> org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorJobBase.initializeMy
> > Spec(ValueAggregatorJobBase.java:69)
> > at
> >
> org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorJobBase.configure(Va
> > lueAggregatorJobBase.java:37)
> > at
> > org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58)
> > at
> >
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:82
> > )
> > at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:32)
> > at
> > org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58)
> > at
> >
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:82
> > )
> > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:185)
> > at
> > org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1777)
> > Caused by: java.lang.ClassNotFoundException:
> > com.company.hadoop.metrics.testAggregator
> > at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
> > at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:276)
> > at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
> > at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
> > at java.lang.Class.forName0(Native Method)
> > at java.lang.Class.forName(Class.java:247)
> > at
> >
> >
> org.apache.hadoop.mapred.lib.aggregate.UserDefinedValueAggregatorDescripto
> > r.createInstance
> > (UserDefinedValueAggregatorDescriptor.java:51)
> > ... 13 more
> >
> > As near as I can tell I am doing everything correctly. Other code
> > existing in this jar file is
> > running just fine - in fact it was used to produce the input to this
> > test. I've checked my
> > systems to make I've got the jar file (and XML file) onto each of the 4
> > compute nodes in the
> > system.
> >
> > Can anybody spot my problem or tell me what to do to get this going?
> >
> > Thanks,
> > C G
> >
> >
> > ---------------------------------
> > Tonight's top picks. What will you watch tonight? Preview the hottest
> > shows on Yahoo! TV.
> 
> 
> 
> 
> ---------------------------------
> Be a better Heartthrob. Get better relationship answers from someone who
> knows.
> Yahoo! Answers - Check it out.


Mime
View raw message