hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From C G <parallel...@yahoo.com>
Subject RE: Question about valueaggregators in 0.14.1...
Date Sat, 13 Oct 2007 03:05:03 GMT
Hi Runping and All:
   
  I took your suggestion, renamed my original code to testAggregatorPlugin and wrote a class
testAggregator which contained the main as you show here.  It also fails with the same traceback...so
it seems something is profoundly wrong here.
   
  Is anybody else using the aggregation classes in 0.14.1 with success?
   
  This is ironic and frustrating...I have several other programs which use map/reduce that
are quite complex and those are running just fine.  This little plugin is all that stands
in the way of delivering the system :-(.
   
  Any other comments/help, etc. most welcome...
   
  Thanks,
  C G

Runping Qi <runping@yahoo-inc.com> wrote:
  
Hadoop ignored the jar file you specified.
This is a known problem andhopefully HADOOP-1622
(https://issues.apache.org/jira/browse/HADOOP-1622) will fix the problem.

In the meantime, you can create a class like below and add it to your jar:

package your_package;


public class MyValueAggregatorJob{

public static void main(String args[]) throws IOException {
JobConf job = ValueAggregatorJob.createValueAggregatorJob(args);
JobClient.runJob(job);
}
}


And use the above class as the main class when you submit hadoop job:
bin/hadoop jar hadoop-0.14.1-dev-company.jar
your_package.MyValueAggregatorJob /input/set1 /output/set1 8 textinputformat
testAggregator.xml


Hope this solve your problem.

Runping




> -----Original Message-----
> From: C G [mailto:parallelguy@yahoo.com]
> Sent: Friday, October 12, 2007 2:43 PM
> To: hadoop-user@lucene.apache.org
> Subject: Question about valueaggregators in 0.14.1...
> 
> Hi All:
> 
> I am tearing out my hair trying to get a simple valueaggregator to run.
> This seems like an
> easy enough thing, but I am consistently getting hit with
> ClassNotFoundException which
> makes no sense to me. Any help would be immensely appreciated. Note
> that I've had
> valueaggregators running on 0.13.*. I'm a little concerned there is a bug
> in 0.14.1. On the
> other hand, maybe I'm just blind to something obvious and just need a
> boot to the head...
> 
> The environment is hadoop-0.14.1. I am running Java 1.6:
> $ java -version
> java version "1.6.0_02"
> Java(TM) SE Runtime Environment (build 1.6.0_02-b05)
> Java HotSpot(TM) Server VM (build 1.6.0_02-b05, mixed mode)
> 
> The class name is called testAggregator, and exists in my jar file:
> $ jar -tvf hadoop-0.14.1-dev-company.jar | grep testAggregator
> 1826 Fri Oct 12 16:30:14 EDT 2007
> com/company/hadoop/metrics/testAggregator.class
> 
> My XML file is testAggregator.xml and looks like:
> 
> 
> 
> 

> aggregator.descriptor.num
> 1
> 

> 

> aggregator.descriptor.0
> UserDefined,com.company.hadoop.metrics.testAggregator
> 

> 
> 
> My code couldn't be simpler:
> 
> package com.company.hadoop.metrics;
> // snip a bunch of imports
> public class testAggregator extends ValueAggregatorBaseDescriptor {
> public ArrayList generateKeyValPairs(Object key, Object val) {
> String fields[] = val.toString().split("\t");
> ArrayList retv = new ArrayList();
> retv.add(generateEntry(LONG_VALUE_SUM, fields[ 6 ], ONE));
> retv.add(generateEntry(UNIQ_VALUE_COUNT, fields[ 6 ], new Text(
> fields[ 2 ])));
> return retv;
> }
> public void configure(JobConf job) {}
> }
> 
> I invoke things this way:
> 
> bin/hadoop jar hadoop-0.14.1-dev-company.jar
> org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorJob /input/set1
> /output/set1 8 textinputformat testAggregator.xml
> 
> All my jobs fail with a ClassNotFoundException like the following
> traceback:
> 
> 2007-10-12 16:56:14,227 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
> Initializing JVM Metrics with processName=MAP, sessionId=
> 2007-10-12 16:56:14,277 INFO org.apache.hadoop.mapred.MapTask:
> numReduceTasks: 8
> 2007-10-12 16:56:14,299 WARN org.apache.hadoop.mapred.TaskTracker: Error
> running child
> java.lang.RuntimeException: java.lang.ClassNotFoundException:
> com.company.hadoop.metrics.testAggregator
> at
> 
> org.apache.hadoop.mapred.lib.aggregate.UserDefinedValueAggregatorDescripto
> r.createInstance
> (UserDefinedValueAggregatorDescriptor.java:56)
> at
> org.apache.hadoop.mapred.lib.aggregate.UserDefinedValueAggregatorDescripto
> r.createAggregator(UserDefinedValueAggregatorDescriptor.java:63)
> at
> org.apache.hadoop.mapred.lib.aggregate.UserDefinedValueAggregatorDescripto
> r.(UserDefinedValueAggregatorDescriptor.java:75)
> at
> org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorJobBase.getValueAggr
> egatorDescriptor(ValueAggregatorJobBase.java:49)
> at
> org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorJobBase.getAggregato
> rDescriptors(ValueAggregatorJobBase.java:60)
> at
> org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorJobBase.initializeMy
> Spec(ValueAggregatorJobBase.java:69)
> at
> org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorJobBase.configure(Va
> lueAggregatorJobBase.java:37)
> at
> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58)
> at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:82
> )
> at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:32)
> at
> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58)
> at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:82
> )
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:185)
> at
> org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1777)
> Caused by: java.lang.ClassNotFoundException:
> com.company.hadoop.metrics.testAggregator
> at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:276)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
> at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:247)
> at
> 
> org.apache.hadoop.mapred.lib.aggregate.UserDefinedValueAggregatorDescripto
> r.createInstance
> (UserDefinedValueAggregatorDescriptor.java:51)
> ... 13 more
> 
> As near as I can tell I am doing everything correctly. Other code
> existing in this jar file is
> running just fine - in fact it was used to produce the input to this
> test. I've checked my
> systems to make I've got the jar file (and XML file) onto each of the 4
> compute nodes in the
> system.
> 
> Can anybody spot my problem or tell me what to do to get this going?
> 
> Thanks,
> C G
> 
> 
> ---------------------------------
> Tonight's top picks. What will you watch tonight? Preview the hottest
> shows on Yahoo! TV.



       
---------------------------------
Be a better Heartthrob. Get better relationship answers from someone who knows.
Yahoo! Answers - Check it out. 
Mime
  • Unnamed multipart/alternative (inline, 8-Bit, 0 bytes)
View raw message