hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rishav <superis...@gmail.com>
Subject Re: Passing objects to Mapper
Date Tue, 20 Apr 2010 19:18:12 GMT
So My mapper has:
public static LexicalizedParser lp;

    public void configure(JobConf job) {
        lp = new LexicalizedParser("/user/me/project4/englishPCFG.ser.gz");
    }


I get a runtimeException of error in configuration object. If you notice
later on in the exception thread, the parser actually gets loaded. My hunch
is it is happening in the master node which can access it. The
englishPCFG.ser.gz trained model is on hdfs.



10/04/20 22:14:24 INFO mapred.JobClient: Task Id :
attempt_201003091315_0078_m_000000_0, Status : FAILED
java.lang.RuntimeException: Error in configuring object
    at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
    at
org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
    at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:354)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
    at org.apache.hadoop.mapred.Child.main(Child.java:170)
Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
    ... 5 more
Caused by: java.lang.RuntimeException: Error in configuring object
    at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
    at
org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
    at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
    at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34)
    ... 10 more
Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
    ... 13 more
Caused by: java.lang.RuntimeException: File not found:
/user/me/project4/englishPCFG.ser.gz
    at
edu.stanford.nlp.parser.lexparser.LexicalizedParser.getParserDataFromSerializedFile(LexicalizedParser.java:829)
    at
edu.stanford.nlp.parser.lexparser.LexicalizedParser.getParserDataFromFile(LexicalizedParser.java:606)
    at
edu.stanford.nlp.parser.lexparser.LexicalizedParser.<init>(LexicalizedParser.java:172)
    at
edu.stanford.nlp.parser.lexparser.LexicalizedParser.<init>(LexicalizedParser.java:158)
    at ParseMapper.configure(ParseMapper.java:22)
    ... 18 more
Caused by: java.io.FileNotFoundException:
/user/me/project4/englishPCFG.ser.gz (No such file or directory)
    at java.io.FileInputStream.open(Native Method)
    at java.io.FileInputStream.<init>(FileInputStream.java:106)
    at java.io.FileInputStream.<init>(FileInputStream.java:66)
    at edu.stanford.nlp.io.IOUtils.readStreamFromString(IOUtils.java:231)
    at
edu.stanford.nlp.parser.lexparser.LexicalizedParser.getParserDataFromSerializedFile(LexicalizedParser.java:816)
    ... 22 more

attempt_201003091315_0078_m_000000_0: Loading parser from serialized file
/user/me/project4/englishPCFG.ser.gz ...
10/04/20 22:14:24 INFO mapred.JobClient: Task Id :
attempt_201003091315_0078_m_000001_0, Status : FAILED
java.lang.RuntimeException: Error in configuring object
    at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
    at
org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
    at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:354)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
    at org.apache.hadoop.mapred.Child.main(Child.java:170)
Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
    ... 5 more
Caused by: java.lang.RuntimeException: Error in configuring object
    at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
    at
org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
    at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
    at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34)
    ... 10 more
Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
    ... 13 more
Caused by: java.lang.RuntimeException: File not found:
/user/me/project4/englishPCFG.ser.gz
    at
edu.stanford.nlp.parser.lexparser.LexicalizedParser.getParserDataFromSerializedFile(LexicalizedParser.java:829)
    at
edu.stanford.nlp.parser.lexparser.LexicalizedParser.getParserDataFromFile(LexicalizedParser.java:606)
    at
edu.stanford.nlp.parser.lexparser.LexicalizedParser.<init>(LexicalizedParser.java:172)
    at
edu.stanford.nlp.parser.lexparser.LexicalizedParser.<init>(LexicalizedParser.java:158)
    at ParseMapper.configure(ParseMapper.java:22)
    ... 18 more
Caused by: java.io.FileNotFoundException:
/user/me/project4/englishPCFG.ser.gz (No such file or directory)
    at java.io.FileInputStream.open(Native Method)
    at java.io.FileInputStream.<init>(FileInputStream.java:106)
    at java.io.FileInputStream.<init>(FileInputStream.java:66)
    at edu.stanford.nlp.io.IOUtils.readStreamFromString(IOUtils.java:231)
    at
edu.stanford.nlp.parser.lexparser.LexicalizedParser.getParserDataFromSerializedFile(LexicalizedParser.java:816)
    ... 22 more

attempt_201003091315_0078_m_000001_0: Loading parser from serialized file
/user/me/project4/englishPCFG.ser.gz ...
10/04/20 22:14:30 INFO mapred.JobClient: Task Id :
attempt_201003091315_0078_m_000000_1, Status : FAILED
java.lang.RuntimeException: Error in configuring object

On Tue, Apr 20, 2010 at 9:21 PM, J.G.Konrad <konradjg@gmail.com> wrote:

> I would suggest adding a configure() method to your Map class and then load
> the parser and save it in a class object.
>
>
> On Tue, Apr 20, 2010 at 4:16 AM, Rishav <superishav@gmail.com> wrote:
>
>> Hello
>>
>> I am trying to pass an object of type LexicalizedParser (which is from an
>> imported jar file stanford-parser) from the main method to the Mapper Class.
>> This is to load the trained model first and then apply on every file offset
>> in the map function in the Mapper Class.
>> Could you tell me the best way to pass an object of any type to the
>> Mapper?
>>
>> Thanking you
>> Rishav
>>
>>
>> --
>> Rishav Bhowmick
>> Computer Science -2010
>> Carnegie Mellon University
>> Tel: +974-561-4371
>> e-mail: rishavb@cmu.edu
>>
>>
>


-- 
Rishav Bhowmick
Computer Science -2010
Carnegie Mellon University
Tel: +974-561-4371
e-mail: rishavb@cmu.edu

Mime
View raw message