hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Runping Qi" <runp...@yahoo-inc.com>
Subject RE: Question about valueaggregators in 0.14.1...
Date Sat, 13 Oct 2007 04:01:50 GMT

Glad it works.

HADOOP-1622 should fix the problem properly.
Until then, the users have to use this kind of hacky obscure workaround:)

Runping

> -----Original Message-----
> From: C G [mailto:parallelguy@yahoo.com]
> Sent: Friday, October 12, 2007 8:48 PM
> To: hadoop-user@lucene.apache.org
> Subject: RE: Question about valueaggregators in 0.14.1...
> 
> Hi Runping and All:
> 
>   That fixed the problem.  Of course my aggregator is now failing for a
> different reason, but that's an error in my code that I can fix.
> 
>   I am extremely grateful for your assistance!
> 
>   Thanks,
>   C G
> 
> Runping Qi <runping@yahoo-inc.com> wrote:
> 
> 
> I am sorry I overlooked something.
> Try to add the following:
> 
> job.setJarByClass(MyValueAggregatorJob.class);
> JobClient.runJob(job);
> 
> To your main.
> 
> Runping
> 
> > -----Original Message-----
> > From: C G [mailto:parallelguy@yahoo.com]
> > Sent: Friday, October 12, 2007 8:05 PM
> > To: hadoop-user@lucene.apache.org
> > Subject: RE: Question about valueaggregators in 0.14.1...
> >
> > Hi Runping and All:
> >
> > I took your suggestion, renamed my original code to testAggregatorPlugin
> > and wrote a class testAggregator which contained the main as you show
> > here. It also fails with the same traceback...so it seems something is
> > profoundly wrong here.
> >
> > Is anybody else using the aggregation classes in 0.14.1 with success?
> >
> > This is ironic and frustrating...I have several other programs which use
> > map/reduce that are quite complex and those are running just fine. This
> > little plugin is all that stands in the way of delivering the system :-
> (.
> >
> > Any other comments/help, etc. most welcome...
> >
> > Thanks,
> > C G
> >
> > Runping Qi wrote:
> >
> > Hadoop ignored the jar file you specified.
> > This is a known problem andhopefully HADOOP-1622
> > (https://issues.apache.org/jira/browse/HADOOP-1622) will fix the
> problem.
> >
> > In the meantime, you can create a class like below and add it to your
> jar:
> >
> > package your_package;
> >
> >
> > public class MyValueAggregatorJob{
> >
> > public static void main(String args[]) throws IOException {
> > JobConf job = ValueAggregatorJob.createValueAggregatorJob(args);
> > JobClient.runJob(job);
> > }
> > }
> >
> >
> > And use the above class as the main class when you submit hadoop job:
> > bin/hadoop jar hadoop-0.14.1-dev-company.jar
> > your_package.MyValueAggregatorJob /input/set1 /output/set1 8
> > textinputformat
> > testAggregator.xml
> >
> >
> > Hope this solve your problem.
> >
> > Runping
> >
> >
> >
> >
> > > -----Original Message-----
> > > From: C G [mailto:parallelguy@yahoo.com]
> > > Sent: Friday, October 12, 2007 2:43 PM
> > > To: hadoop-user@lucene.apache.org
> > > Subject: Question about valueaggregators in 0.14.1...
> > >
> > > Hi All:
> > >
> > > I am tearing out my hair trying to get a simple valueaggregator to
> run.
> > > This seems like an
> > > easy enough thing, but I am consistently getting hit with
> > > ClassNotFoundException which
> > > makes no sense to me. Any help would be immensely appreciated. Note
> > > that I've had
> > > valueaggregators running on 0.13.*. I'm a little concerned there is a
> > bug
> > > in 0.14.1. On the
> > > other hand, maybe I'm just blind to something obvious and just need a
> > > boot to the head...
> > >
> > > The environment is hadoop-0.14.1. I am running Java 1.6:
> > > $ java -version
> > > java version "1.6.0_02"
> > > Java(TM) SE Runtime Environment (build 1.6.0_02-b05)
> > > Java HotSpot(TM) Server VM (build 1.6.0_02-b05, mixed mode)
> > >
> > > The class name is called testAggregator, and exists in my jar file:
> > > $ jar -tvf hadoop-0.14.1-dev-company.jar | grep testAggregator
> > > 1826 Fri Oct 12 16:30:14 EDT 2007
> > > com/company/hadoop/metrics/testAggregator.class
> > >
> > > My XML file is testAggregator.xml and looks like:
> > >
> > >
> > >
> > >
> >
> > > aggregator.descriptor.num
> > > 1
> > >
> >
> > >
> >
> > > aggregator.descriptor.0
> > > UserDefined,com.company.hadoop.metrics.testAggregator
> > >
> >
> > >
> > >
> > > My code couldn't be simpler:
> > >
> > > package com.company.hadoop.metrics;
> > > // snip a bunch of imports
> > > public class testAggregator extends ValueAggregatorBaseDescriptor {
> > > public ArrayList generateKeyValPairs(Object key, Object val) {
> > > String fields[] = val.toString().split("\t");
> > > ArrayList retv = new ArrayList();
> > > retv.add(generateEntry(LONG_VALUE_SUM, fields[ 6 ], ONE));
> > > retv.add(generateEntry(UNIQ_VALUE_COUNT, fields[ 6 ], new Text(
> > > fields[ 2 ])));
> > > return retv;
> > > }
> > > public void configure(JobConf job) {}
> > > }
> > >
> > > I invoke things this way:
> > >
> > > bin/hadoop jar hadoop-0.14.1-dev-company.jar
> > > org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorJob /input/set1
> > > /output/set1 8 textinputformat testAggregator.xml
> > >
> > > All my jobs fail with a ClassNotFoundException like the following
> > > traceback:
> > >
> > > 2007-10-12 16:56:14,227 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
> > > Initializing JVM Metrics with processName=MAP, sessionId=
> > > 2007-10-12 16:56:14,277 INFO org.apache.hadoop.mapred.MapTask:
> > > numReduceTasks: 8
> > > 2007-10-12 16:56:14,299 WARN org.apache.hadoop.mapred.TaskTracker:
> Error
> > > running child
> > > java.lang.RuntimeException: java.lang.ClassNotFoundException:
> > > com.company.hadoop.metrics.testAggregator
> > > at
> > >
> > >
> >
> org.apache.hadoop.mapred.lib.aggregate.UserDefinedValueAggregatorDescripto
> > > r.createInstance
> > > (UserDefinedValueAggregatorDescriptor.java:56)
> > > at
> > >
> >
> org.apache.hadoop.mapred.lib.aggregate.UserDefinedValueAggregatorDescripto
> > > r.createAggregator(UserDefinedValueAggregatorDescriptor.java:63)
> > > at
> > >
> >
> org.apache.hadoop.mapred.lib.aggregate.UserDefinedValueAggregatorDescripto
> > > r.(UserDefinedValueAggregatorDescriptor.java:75)
> > > at
> > >
> >
> org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorJobBase.getValueAggr
> > > egatorDescriptor(ValueAggregatorJobBase.java:49)
> > > at
> > >
> >
> org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorJobBase.getAggregato
> > > rDescriptors(ValueAggregatorJobBase.java:60)
> > > at
> > >
> >
> org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorJobBase.initializeMy
> > > Spec(ValueAggregatorJobBase.java:69)
> > > at
> > >
> >
> org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorJobBase.configure(Va
> > > lueAggregatorJobBase.java:37)
> > > at
> > >
> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58)
> > > at
> > >
> >
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:82
> > > )
> > > at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:32)
> > > at
> > >
> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58)
> > > at
> > >
> >
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:82
> > > )
> > > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:185)
> > > at
> > > org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1777)
> > > Caused by: java.lang.ClassNotFoundException:
> > > com.company.hadoop.metrics.testAggregator
> > > at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
> > > at java.security.AccessController.doPrivileged(Native Method)
> > > at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
> > > at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> > > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:276)
> > > at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
> > > at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
> > > at java.lang.Class.forName0(Native Method)
> > > at java.lang.Class.forName(Class.java:247)
> > > at
> > >
> > >
> >
> org.apache.hadoop.mapred.lib.aggregate.UserDefinedValueAggregatorDescripto
> > > r.createInstance
> > > (UserDefinedValueAggregatorDescriptor.java:51)
> > > ... 13 more
> > >
> > > As near as I can tell I am doing everything correctly. Other code
> > > existing in this jar file is
> > > running just fine - in fact it was used to produce the input to this
> > > test. I've checked my
> > > systems to make I've got the jar file (and XML file) onto each of the
> 4
> > > compute nodes in the
> > > system.
> > >
> > > Can anybody spot my problem or tell me what to do to get this going?
> > >
> > > Thanks,
> > > C G
> > >
> > >
> > > ---------------------------------
> > > Tonight's top picks. What will you watch tonight? Preview the hottest
> > > shows on Yahoo! TV.
> >
> >
> >
> >
> > ---------------------------------
> > Be a better Heartthrob. Get better relationship answers from someone who
> > knows.
> > Yahoo! Answers - Check it out.
> 
> 
> 
> 
> ---------------------------------
> Catch up on fall's hot new shows on Yahoo! TV.  Watch previews, get
> listings, and more!


Mime
View raw message