hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "11 Nov." <nov.eleve...@gmail.com>
Subject Re: Java inputformat for pipes job
Date Tue, 08 Apr 2008 10:08:59 GMT
You should use the -pipes option in the command.
For the input format, you can pack it into the hadoop core class jar file,
or put it into the cache file.

2008/4/8, Rahul Sood <rsood@yahoo-inc.com>:
>
> Hi,
>
> I implemented a customized input format in Java for a Map Reduce job.
> The mapper and reducer classes are implemented in C++, using the Hadoop
> Pipes API.
>
> The package documentation for org.apache.hadoop.mapred.pipes states that
> "The job may consist of any combination of Java and C++ RecordReaders,
> Mappers, Paritioner, Combiner, Reducer, and RecordWriter"
>
> I packaged the input format class in a jar file and ran the job
> invocation command:
>
> hadoop pipes -jar mytest.jar -inputformat mytest.PriceInputFormat -conf
> conf/mytest.xml -input mgr/in -output mgr/out -program mgr/bin/TestMgr
>
> It keeps failing with error ClassNotFoundException
> Although I've specified the jar file name with the -jar parameter, the
> input format class still cannot be located. Is there any other means to
> specify the input format class, or the job jar file, for a Pipes job ?
>
> Stack trace:
>
> Exception in thread "main" java.lang.ClassNotFoundException:
> mytest.PriceInputFormat
>         at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>         at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:276)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
>         at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
>         at java.lang.Class.forName0(Native Method)
>         at java.lang.Class.forName(Class.java:247)
>         at
>
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:524)
>         at
> org.apache.hadoop.mapred.pipes.Submitter.getClass(Submitter.java:309)
>         at
> org.apache.hadoop.mapred.pipes.Submitter.main(Submitter.java:357)
>
> Thanks,
>
>
> Rahul Sood
> rsood@yahoo-inc.com
>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message