hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Runping Qi" <runp...@yahoo-inc.com>
Subject RE: Question about valueaggregators in 0.14.1...
Date Fri, 12 Oct 2007 22:14:19 GMT

Hadoop ignored the jar file you specified.
This is a known problem andhopefully HADOOP-1622
(https://issues.apache.org/jira/browse/HADOOP-1622) will fix the problem.

In the meantime, you can create a class like below and add it to your jar:

package your_package;


public class MyValueAggregatorJob{

public static void main(String args[]) throws IOException {
  JobConf job = ValueAggregatorJob.createValueAggregatorJob(args);
  JobClient.runJob(job);
}
}


And use the above class as the main class when you submit hadoop job:
bin/hadoop jar hadoop-0.14.1-dev-company.jar
your_package.MyValueAggregatorJob /input/set1 /output/set1 8 textinputformat
testAggregator.xml


Hope this solve your problem.

Runping




> -----Original Message-----
> From: C G [mailto:parallelguy@yahoo.com]
> Sent: Friday, October 12, 2007 2:43 PM
> To: hadoop-user@lucene.apache.org
> Subject: Question about valueaggregators in 0.14.1...
> 
> Hi All:
> 
>   I am tearing out my hair trying to get a simple valueaggregator to run.
> This seems like an
>   easy enough thing, but I am consistently getting hit with
> ClassNotFoundException which
>   makes no sense to me.  Any help would be immensely appreciated.  Note
> that I've had
> valueaggregators running on 0.13.*.  I'm a little concerned there is a bug
> in 0.14.1.  On the
>   other hand, maybe I'm just blind to  something obvious and just need a
> boot to the head...
> 
>   The environment is hadoop-0.14.1.  I am running Java 1.6:
>   $ java -version
>   java version "1.6.0_02"
> Java(TM) SE Runtime Environment (build 1.6.0_02-b05)
> Java HotSpot(TM) Server VM (build 1.6.0_02-b05, mixed mode)
> 
>   The class name is called testAggregator, and exists in my jar file:
>   $ jar -tvf hadoop-0.14.1-dev-company.jar | grep testAggregator
>     1826 Fri Oct 12 16:30:14 EDT 2007
> com/company/hadoop/metrics/testAggregator.class
> 
> My XML file is testAggregator.xml and looks like:
> <?xml version="1.0"?>
> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
>   <configuration>
>   <property>
>     <name>aggregator.descriptor.num</name>
>     <value>1</value>
> </property>
> <property>
>     <name>aggregator.descriptor.0</name>
>    <value>UserDefined,com.company.hadoop.metrics.testAggregator</value>
> </property>
>   </configuration>
> 
>   My code couldn't be simpler:
> 
>   package com.company.hadoop.metrics;
> // snip a bunch of imports
> public class testAggregator extends ValueAggregatorBaseDescriptor {
>     public ArrayList<Entry> generateKeyValPairs(Object key, Object val) {
>         String fields[] = val.toString().split("\t");
>         ArrayList<Entry> retv = new ArrayList<Entry>();
>       retv.add(generateEntry(LONG_VALUE_SUM, fields[ 6 ], ONE));
>     retv.add(generateEntry(UNIQ_VALUE_COUNT, fields[ 6 ], new Text(
> fields[ 2 ])));
>     return retv;
>    }
>   public void configure(JobConf job) {}
> }
> 
>   I invoke things this way:
> 
>   bin/hadoop jar hadoop-0.14.1-dev-company.jar
> org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorJob /input/set1
> /output/set1 8 textinputformat testAggregator.xml
> 
>   All my jobs fail with a ClassNotFoundException like the following
> traceback:
> 
>   2007-10-12 16:56:14,227 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
> Initializing JVM Metrics with processName=MAP, sessionId=
> 2007-10-12 16:56:14,277 INFO org.apache.hadoop.mapred.MapTask:
> numReduceTasks: 8
> 2007-10-12 16:56:14,299 WARN org.apache.hadoop.mapred.TaskTracker: Error
> running child
> java.lang.RuntimeException: java.lang.ClassNotFoundException:
>   com.company.hadoop.metrics.testAggregator
>         at
> 
> org.apache.hadoop.mapred.lib.aggregate.UserDefinedValueAggregatorDescripto
> r.createInstance
>   (UserDefinedValueAggregatorDescriptor.java:56)
>         at
> org.apache.hadoop.mapred.lib.aggregate.UserDefinedValueAggregatorDescripto
> r.createAggregator(UserDefinedValueAggregatorDescriptor.java:63)
>         at
> org.apache.hadoop.mapred.lib.aggregate.UserDefinedValueAggregatorDescripto
> r.<init>(UserDefinedValueAggregatorDescriptor.java:75)
>         at
> org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorJobBase.getValueAggr
> egatorDescriptor(ValueAggregatorJobBase.java:49)
>         at
> org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorJobBase.getAggregato
> rDescriptors(ValueAggregatorJobBase.java:60)
>         at
> org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorJobBase.initializeMy
> Spec(ValueAggregatorJobBase.java:69)
>         at
> org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorJobBase.configure(Va
> lueAggregatorJobBase.java:37)
>         at
> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58)
>         at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:82
> )
>         at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:32)
>         at
> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58)
>         at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:82
> )
>         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:185)
>         at
> org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1777)
> Caused by: java.lang.ClassNotFoundException:
> com.company.hadoop.metrics.testAggregator
>         at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>         at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:276)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
>         at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
>         at java.lang.Class.forName0(Native Method)
>         at java.lang.Class.forName(Class.java:247)
>         at
> 
> org.apache.hadoop.mapred.lib.aggregate.UserDefinedValueAggregatorDescripto
> r.createInstance
>   (UserDefinedValueAggregatorDescriptor.java:51)
>         ... 13 more
> 
>   As near as I can tell I am doing everything correctly.  Other code
> existing in this jar file is
>   running just fine - in fact it was used to produce the input to this
> test.  I've checked my
>   systems to make I've got the jar file (and XML file) onto each  of the 4
> compute nodes in the
>   system.
> 
>   Can anybody spot my problem or tell me what to do to get this going?
> 
>   Thanks,
> C G
> 
> 
> ---------------------------------
> Tonight's top picks. What will you watch tonight? Preview the hottest
> shows on Yahoo! TV.


Mime
View raw message