hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alex Kozlov <ale...@cloudera.com>
Subject Re: JNI native library loading problem in standalone mode
Date Tue, 01 Jun 2010 15:07:35 GMT
Hi Edward,  Can you provide the output of 'ps -aef -u hadoop | grep Child'
on a tasktracker node while your job is running? -- Alex K

On Mon, May 31, 2010 at 8:47 PM, Hemanth Yamijala <yhemanth@gmail.com>wrote:

> Edward,
>
> If it's an option to copy the libraries to a fixed location on all the
> cluster nodes, you could do that and configure them in the library
> path via mapred.child.java.opts. Please look at http://bit.ly/ab93Z8
> (MapReduce tutorial on Hadoop site) to see how to use this config
> option for setting paths to native libraries.
>
> Thanks
> Hemanth
>
> On Tue, Jun 1, 2010 at 6:50 AM, edward choi <mp2893@gmail.com> wrote:
> > Alex,
> >
> > Thanks for your advice.
> > I tried both of them and neither works.
> > I am still getting the UnsatisfiedLinkError when I try to run my
> application
> > in StandAlone mode.
> >
> > I also made LangAnal.java belong to the package of "org.knu.freeparser"
> just
> > in case. But still no luck.
> > I put libetri_lmi.so to "/usr/lib" and still no luck.
> >
> > 2010/6/1 Alex Kozlov <alexvk@cloudera.com>
> >
> >> Try
> >>
> >>   1. export JAVA_LIBRARY_PATH=/home/qa/lang/dll4jni/
> >>   2. mkdir lib; mv LangAnal.jar lib; jar -cvef org.etri.mp2893.LangAnal
> >>   ./lib ./org
> >>
> >> In a distributed mode you'll have to copy your dynamic libs to some
> >> directory thoughout the cluster and point JAVA_LIBRARY_PATH to it (or
> you
> >> can use distributed cache).
> >>
> >> Let me know if you have other config problems.
> >>
> >> Alex K
> >>
> >> On Mon, May 31, 2010 at 9:49 AM, edward choi <mp2893@gmail.com> wrote:
> >>
> >> > Hi, I am trying to run a JNI application on StandAlone mode and I have
> >> been
> >> > getting this result ever since.
> >> > I've looked up every possible webs and sites but never could get the
> >> > solution to it.
> >> > Please help me find out what's wrong.
> >> >
> >> > I have 2 java files (LangAnal.java, freeparser.java)
> >> > LangAnal.java belongs to a package "org.etri.mp2893"
> >> > freeparser.java belongs to a package "org.knu.freeparser"
> >> >
> >> > freeparser.java loads "libetri_lmi.so" which is in
> >> "/home/qa/lang/dll4jni/"
> >> > and I believe "libetri_lmi.so" uses another library "libparser.so"
> which
> >> is
> >> > in "/home/qa/lang/lib".
> >> >
> >> > I have compiled LangAnal.java and freeparser.java. The resulting
> classes
> >> > are
> >> > placed respectively at "/home/qa/lang/dll4jni/org/etri/mp2893/" and
> >> > "/home/qa/lang/dll4jni/org/knu/freeparser/".
> >> >
> >> > Then I have archived those classes by
> >> >
> >> > "jar -cvef org.etri.mp2893.LangAnal ./LangAnal.jar ./org/*"
> >> >
> >> > from the directory of "/home/qa/lang/dll4jni/".
> >> >
> >> > Then I tried to run
> >> >
> >> > hadoop jar LangAnal.jar ./news.txt ./output
> >> >
> >> > from the same directory and I get the result down below.
> >> >
> >> > I tried to set LD_LIBRARY_PATH to
> >> > ".:/home/qa/lang/dll4jni:/home/qa/lang/lib"
> >> > I tried java.setProperty("java.library.path",
> "/home/qa/lang/dll4jni");
> >> in
> >> > freeparser.java.
> >> > So far nothing has worked.
> >> >
> >> > But the funny thing is, if I run
> >> >
> >> > hadoop -Djava.library.path=/home/qa/lang/dll4jni
> org.etri.mp2893.LangAnal
> >> > ./news.txt ./output
> >> >
> >> > from "/home/qa/lang/dll4jni/" it works just fine. So I guess the
> problem
> >> is
> >> > just about jar file.
> >> > And I intend to run this application in Fully-Distributed mode soon, I
> >> have
> >> > to figure out how to run jar files.
> >> >
> >> > Please someone help me.
> >> >
> >> >
> >> >
> >>
> -------------------------------------------------------------------------------------------------------------------
> >> >
> >> > [qa@qa128 dll4jni]$ hadoop jar LangAnal.jar ./news.txt ./output
> >> > 10/06/01 01:19:39 INFO jvm.JvmMetrics: Initializing JVM Metrics with
> >> > processName=JobTracker, sessionId=
> >> > 10/06/01 01:19:39 WARN mapred.JobClient: Use GenericOptionsParser for
> >> > parsing the arguments. Applications should implement Tool for the
> same.
> >> > 10/06/01 01:19:39 WARN mapred.JobClient: No job jar file set.  User
> >> classes
> >> > may not be found. See JobConf(Class) or JobConf#setJar(String).
> >> > 10/06/01 01:19:39 INFO input.FileInputFormat: Total input paths to
> >> process
> >> > :
> >> > 1
> >> > 10/06/01 01:19:39 INFO mapred.JobClient: Running job: job_local_0001
> >> > 10/06/01 01:19:39 INFO input.FileInputFormat: Total input paths to
> >> process
> >> > :
> >> > 1
> >> > 10/06/01 01:19:39 WARN mapred.LocalJobRunner: job_local_0001
> >> > java.lang.RuntimeException:
> java.lang.reflect.InvocationTargetException
> >> > at
> >> >
> >> >
> >>
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:115)
> >> > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:569)
> >> > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
> >> > at
> >> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177)
> >> > Caused by: java.lang.reflect.InvocationTargetException
> >> > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> >> > at
> >> >
> >> >
> >>
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> >> > at
> >> >
> >> >
> >>
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> >> > at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> >> > at
> >> >
> >> >
> >>
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:113)
> >> > ... 3 more
> >> > Caused by: java.lang.UnsatisfiedLinkError: no etri_lmi in
> >> java.library.path
> >> > at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1709)
> >> > at java.lang.Runtime.loadLibrary0(Runtime.java:823)
> >> > at java.lang.System.loadLibrary(System.java:1030)
> >> > at org.knu.freeparser.freeparser.<clinit>(freeparser.java:9)
> >> > at org.etri.mp2893.LangAnal$LangAnalMapper.<init>(LangAnal.java:26)
> >> > ... 8 more
> >> > 10/06/01 01:19:40 INFO mapred.JobClient:  map 0% reduce 0%
> >> > 10/06/01 01:19:40 INFO mapred.JobClient: Job complete: job_local_0001
> >> > 10/06/01 01:19:40 INFO mapred.JobClient: Counters: 0
> >> >
> >>
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message