hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sonal Goyal <sonalgoy...@gmail.com>
Subject Re: Setting java.library.path for map-reduce job
Date Mon, 28 Feb 2011 13:50:13 GMT
Adarsh,

Are you trying to distribute both the native library and the jcuda.jar?
Could you please explain your job's dependencies?
Thanks and Regards,
Sonal
<https://github.com/sonalgoyal/hiho>Hadoop ETL and Data
Integration<https://github.com/sonalgoyal/hiho>
Nube Technologies <http://www.nubetech.co>

<http://in.linkedin.com/in/sonalgoyal>





On Mon, Feb 28, 2011 at 6:54 PM, Adarsh Sharma <adarsh.sharma@orkash.com>wrote:

> Sonal Goyal wrote:
>
>> Hi Adarsh,
>>
>> I think your mapred.cache.files property has an extra space at the end.
>> Try
>> removing that and let us know how it goes.
>> Thanks and Regards,
>> Sonal
>> <https://github.com/sonalgoyal/hiho>Hadoop ETL and Data
>> Integration<https://github.com/sonalgoyal/hiho>
>> Nube Technologies <http://www.nubetech.co>
>>
>> <http://in.linkedin.com/in/sonalgoyal>
>>
>>
>>
>>
>
> Thanks a Lot Sonal but it doesn't succeed.
> Please if possible tell me the proper steps that are need to be followed
> after Configuring Hadoop Cluster.
>
> I don't believe that a simple commands succeeded as
>
> [root@cuda1 hadoop-0.20.2]# javac EnumDevices.java
> [root@cuda1 hadoop-0.20.2]# java EnumDevices
> Total number of devices: 1
> Name: Tesla C1060
> Version: 1.3
> Clock rate: 1296000 MHz
> Threads per block: 512
>
>
> but in Map-reduce job it fails :
>
> 11/02/28 18:42:47 INFO mapred.JobClient: Task Id :
> attempt_201102281834_0001_m_000001_2, Status : FAILED
> java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
>       at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:115)
>       at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:569)
>       at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
>       at org.apache.hadoop.mapred.Child.main(Child.java:170)
> Caused by: java.lang.reflect.InvocationTargetException
>       at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
>       at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
>       at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>       at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>       at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:113)
>       ... 3 more
> Caused by: java.lang.UnsatisfiedLinkError: no jcuda in java.library.path
>       at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1734)
>       at java.lang.Runtime.loadLibrary0(Runtime.java:823)
>       at java.lang.System.loadLibrary(System.java:1028)
>       at jcuda.driver.CUDADriver.<clinit>(CUDADriver.java:909)
>       at jcuda.CUDA.init(CUDA.java:62)
>       at jcuda.CUDA.<init>(CUDA.java:42)
>
>
>
>
> Thanks & best Regards,
>
> Adarsh Sharma
>
>>
>>
>> On Mon, Feb 28, 2011 at 5:06 PM, Adarsh Sharma <adarsh.sharma@orkash.com
>> >wrote:
>>
>>
>>
>>> Thanks Sanjay, it seems i found the root cause.
>>>
>>> But I result in following error:
>>>
>>> [hadoop@ws37-mah-lin hadoop-0.20.2]$ bin/hadoop jar wordcount1.jar
>>> org.myorg.WordCount /user/hadoop/gutenberg /user/hadoop/output1
>>> Exception in specified URI's java.net.URISyntaxException: Illegal
>>> character
>>> in path at index 36: hdfs://192.168.0.131:54310/jcuda.jar
>>>      at java.net.URI$Parser.fail(URI.java:2809)
>>>      at java.net.URI$Parser.checkChars(URI.java:2982)
>>>      at java.net.URI$Parser.parseHierarchical(URI.java:3066)
>>>      at java.net.URI$Parser.parse(URI.java:3014)
>>>      at java.net.URI.<init>(URI.java:578)
>>>      at
>>> org.apache.hadoop.util.StringUtils.stringToURI(StringUtils.java:204)
>>>      at
>>>
>>> org.apache.hadoop.filecache.DistributedCache.getCacheFiles(DistributedCache.java:593)
>>>      at
>>>
>>> org.apache.hadoop.mapred.JobClient.configureCommandLineOptions(JobClient.java:638)
>>>      at
>>> org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:761)
>>>      at org.apache.hadoop.mapreduce.Job.submit(Job.java:432)
>>>      at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:447)
>>>      at org.myorg.WordCount.main(WordCount.java:59)
>>>      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>      at
>>>
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>      at
>>>
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>      at java.lang.reflect.Method.invoke(Method.java:597)
>>>      at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
>>>
>>> Exception in thread "main" java.lang.NullPointerException
>>>      at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:176)
>>>      at
>>>
>>> org.apache.hadoop.filecache.DistributedCache.getTimestamp(DistributedCache.java:506)
>>>      at
>>>
>>> org.apache.hadoop.mapred.JobClient.configureCommandLineOptions(JobClient.java:640)
>>>      at
>>> org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:761)
>>>      at org.apache.hadoop.mapreduce.Job.submit(Job.java:432)
>>>      at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:447)
>>>      at org.myorg.WordCount.main(WordCount.java:59)
>>>      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>      at
>>>
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>      at
>>>
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>      at java.lang.reflect.Method.invoke(Method.java:597)
>>>      at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
>>>
>>> Please check my attached mapred-site.xml
>>>
>>>
>>> Thanks & best regards,
>>>
>>> Adarsh Sharma
>>>
>>>
>>>
>>> Kaluskar, Sanjay wrote:
>>>
>>>
>>>
>>>> You will probably have to use distcache to distribute your jar to all
>>>> the nodes too. Read the distcache documentation; Then on each node you
>>>> can add the new jar to the java.library.path through
>>>> mapred.child.java.opts.
>>>>
>>>> You need to do something like the following in mapred-site.xml, where
>>>> fs-uri is the URI of the file system (something like
>>>> host.mycompany.com:54310).
>>>>
>>>> <property>
>>>>  <name>mapred.cache.files</name>
>>>>  <value>hdfs://fs-uri/jcuda/jcuda.jar#jcuda.jar </value>
>>>> </property>
>>>> <property>
>>>>  <name>mapred.create.symlink</name>
>>>>  <value>yes</value>
>>>> </property>
>>>> <property>
>>>>  <name>mapred.child.java.opts</name>
>>>>  <value>-Djava.library.path=jcuda.jar</value>
>>>> </property>
>>>>
>>>>
>>>> -----Original Message-----
>>>> From: Adarsh Sharma [mailto:adarsh.sharma@orkash.com] Sent: 28 February
>>>> 2011 16:03
>>>> To: common-user@hadoop.apache.org
>>>> Subject: Setting java.library.path for map-reduce job
>>>>
>>>> Dear all,
>>>>
>>>> I want to set some extra jars in java.library.path , used while running
>>>> map-reduce program in Hadoop Cluster.
>>>>
>>>> I got a exception entitled "no jcuda in java.library.path" in each map
>>>> task.
>>>>
>>>> I run my map-reduce code by below commands :
>>>>
>>>> javac -classpath
>>>> /home/hadoop/project/hadoop-0.20.2/hadoop-0.20.2-core.jar://home/hadoop/
>>>> project/hadoop-0.20.2/jcuda_1.1_linux64/jcuda.jar:/home/hadoop/project/h
>>>> adoop-0.20.2/lib/commons-cli-1.2.jar
>>>> -d wordcount_classes1/ WordCount.java
>>>>
>>>> jar -cvf wordcount1.jar -C wordcount_classes1/ .
>>>>
>>>> bin/hadoop jar wordcount1.jar org.myorg.WordCount /user/hadoop/gutenberg
>>>> /user/hadoop/output1
>>>>
>>>>
>>>> Please guide how to achieve this.
>>>>
>>>>
>>>>
>>>> Thanks & best Regards,
>>>>
>>>> Adarsh Sharma
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message