hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shahab Yunus <shahab.yu...@gmail.com>
Subject Re: ClassCastException on running map-reduce jobs + tests on Windows (mongo-hadoop)
Date Wed, 17 Sep 2014 15:20:00 GMT
You String as the outputKey (). java.lang.String is not Writable. Change it
to Text just like you did for the Mapper.

Regards,
Shahab

On Wed, Sep 17, 2014 at 10:43 AM, Blanca Hernandez <
Blanca.Hernandez@willhaben.at> wrote:

>  Thanks for answering:
>
>
>
> hadoop jar /tmp/hadoop-test.jar
> at.willhaben.hadoop.AveragePriceCalculationJob
>
>
>
> In the AveragePriceCalculationJob I have my configuration:
>
>
>
>
>
> private static class AveragePriceCalculationJob extends MongoTool {
>
>         private AveragePriceCalculationJob(AveragePriceNode currentNode,
> String currentId, int nodeNumber) {
>
>             Configuration conf = new Configuration();
>
>             MongoConfig config = new MongoConfig(conf);
>
>             setConf(conf);
>
>             // change for my values
>
>             config.setInputFormat(MongoInputFormat.class);
>
>             config.setOutputFormat(MongoOutputFormat.class);
>
>
>
>             config.setMapperOutputKey(Text.class);
>
>             config.setMapperOutputValue(BSONObject.class);
>
>             config.setOutputKey(String.class);
>
>             config.setOutputValue(BSONWritable.class);
>
>
>
>             config.setInputURI("myUrl");
>
>             config.setOutputURI("myUrl");
>
>             config.setMapper(AveragePriceMapper.class);
>
>             config.setReducer(AveragePriceReducer.class);
>
>
>
>         }
>
>     }
>
>
>
>
>
> And the main method:
>
>
>
>
>
> public static void main(String [] args) throws InterruptedException,
> IOException, ClassNotFoundException {
>
>         // … some code
>
>
>
>             try {
>
>                 ToolRunner.run(new AveragePriceCalculationJob(currentNode,
> currentId, nodeNumber), args);
>
>             } catch (Exception e) {
>
>                 e.printStackTrace();  //To change body of catch statement
> use File | Settings | File Templates.
>
>             }
>
>     }
>
>
>
>
>
> Best regards,
>
>
>
> Blanca
>
>
>
> *Von:* Shahab Yunus [mailto:shahab.yunus@gmail.com]
> *Gesendet:* Mittwoch, 17. September 2014 16:37
> *An:* user@hadoop.apache.org
> *Betreff:* Re: ClassCastException on running map-reduce jobs + tests on
> Windows (mongo-hadoop)
>
>
>
> Can you provide the driver code for this job?
>
>
>
> Regards,
>
> Shahab
>
>
>
> On Wed, Sep 17, 2014 at 10:28 AM, Blanca Hernandez <
> Blanca.Hernandez@willhaben.at> wrote:
>
> Hi again, I changed the String objects with org.apache.hadoop.io.Text
> objects (why is String not accepted?), and now I get another exception, so
> I don´t really know if I solved something or I broke something:
>
>
>
>
>
> java.lang.Exception: java.lang.NullPointerException
>
>         at
> org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462)
>
>         at
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:522)
>
> Caused by: java.lang.NullPointerException
>
>         at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.init(MapTask.java:988)
>
>         at
> org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:391)
>
>         at org.apache.hadoop.mapred.MapTask.access$100(MapTask.java:80)
>
>         at
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:675)
>
>         at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:747)
>
>         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
>
>         at
> org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243)
>
>         at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>
>         at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>
>         at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>
>         at java.lang.Thread.run(Thread.java:744)
>
>
>
> If I could debug it in my IDE, I think I could work faster, but I have the
> problems already exposed. How am I testing now? Building a jar, copying it
> on the server and running a Hadoop jar command (not very performance
> approach…).
>
>
>
> Could you give me a hand on this? Any Windows + IntelliJ IDEa there?
> Maaaany thanks!
>
>
>
>
>
>
>
> *Von:* Blanca Hernandez [mailto:Blanca.Hernandez@willhaben.at]
> *Gesendet:* Mittwoch, 17. September 2014 15:27
> *An:* user@hadoop.apache.org
> *Betreff:* ClassCastException on running map-reduce jobs + tests on
> Windows (mongo-hadoop)
>
>
>
> Hi!
>
>
>
> I am getting some CCE and don´t really understand why…
>
>
>
> Here my mapper:
>
>
>
> public class AveragePriceMapper extends Mapper<String, BSONObject, String,
> BSONObject>{
>
>     @Override
>
>     public void map(final String key, final BSONObject val, final Context
> context) throws IOException, InterruptedException {
>
>         String id = “result_of_making_some_operations”;
>
>         context.write(id, val);
>
>     }
>
> }
>
>
>
> And in my configuration:
>
>
>
> config.setMapperOutputKey(String.class);
>
> config.setMapperOutputValue(BSONObject.class);
>
>
>
>
>
> On running my generated jar on the server, seems to work everything ok
> until:
>
>
>
> 14/09/17 15:20:36 INFO mapred.MapTask: Processing split:
> MongoInputSplit{URI=mongodb://user:pass@host:27017/my_db.my_collection,
> authURI=null, min={ "_id" : { "$oid" : "541666d8e4b07265e257a42e"}}, max={
> }, query={ }, sort={ }, fields={ }, notimeout=false}
>
> 14/09/17 15:20:36 INFO mapred.MapTask: Map output collector class =
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer
>
> 14/09/17 15:20:36 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
>
> 14/09/17 15:20:36 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
>
> 14/09/17 15:20:36 INFO mapred.MapTask: soft limit at 83886080
>
> 14/09/17 15:20:36 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
>
> 14/09/17 15:20:36 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
>
> 14/09/17 15:20:36 INFO mapred.LocalJobRunner: map task executor complete.
>
> 14/09/17 15:20:36 WARN mapred.LocalJobRunner:
> job_local1701078621_0001java.lang.Exception: java.lang.ClassCastException:
> class java.lang.String
>
>         at
> org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462)
>
>         at
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:522)
>
> Caused by: java.lang.ClassCastException: class java.lang.String
>
>         at java.lang.Class.asSubclass(Class.java:3126)
>
>         at
> org.apache.hadoop.mapred.JobConf.getOutputKeyComparator(JobConf.java:885)
>
>         at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.init(MapTask.java:981)
>
>         at
> org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:391)
>
>         at org.apache.hadoop.mapred.MapTask.access$100(MapTask.java:80)
>
>         at
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:675)
>
>         at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:747)
>
>         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
>
>         at
> org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243)
>
>         at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>
>         at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>
>         at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>
>         at java.lang.Thread.run(Thread.java:744)
>
>
>
>
>
> Did I miss something??
>
>
>
>
>
> Another issue I am worry about: working on a Windows system makes
> everything quite complicated with Hadoop. I have it installed and running,
> the same as my mongoDB database (I am using the connector provided by
> them). Running the same main class I am using in the hadooop jar call on
> the server (in the example before), but from my IDE, I get this exception:
>
>
>
> PriviledgedActionException as:hernanbl cause:java.io.IOException: Failed
> to set permissions of path:
> \tmp\hadoop-hernanbl\mapred\staging\hernanbl1600842219\.staging to 0700
>
>
>
> How could I make it run?
>
>
>
>
>
> Many thanks!!
>
>
>
> Best regards,
>
>
>
> Blanca
>
>
>

Mime
View raw message