Mailing-List: contact user-help@spark.apache.org; run by ezmlm
Precedence: bulk
MIME-Version: 1.0
Date: Wed, 14 Oct 2015 01:44:51 -0300
Message-ID: 
 <CA+CehZVK1j61GPutONibb4EWDe32x8jVxZM5sBpx=pQdWBrxYQ@mail.gmail.com>
Subject: Running in cluster mode causes native library linking to fail
From: Bernardo Vecchia Stein <bernardovstein@gmail.com>
To: user@spark.apache.org
Content-Type: multipart/alternative; boundary=047d7b41890b184934052209392d

--047d7b41890b184934052209392d
Content-Type: text/plain; charset=UTF-8

Hello,

I am trying to run some scala code in cluster mode using spark-submit. This
code uses addLibrary to link with a .so that exists in the machine, and
this library has a function to be called natively (there's a native
definition as needed in the code).

The problem I'm facing is: whenever I try to run this code in cluster mode,
spark fails with the following message when trying to execute the native
function:
java.lang.UnsatisfiedLinkError:
org.name.othername.ClassName.nativeMethod([B[B)[B

Apparently, the library is being found by spark, but the required function
isn't found.

When trying to run in client mode, however, this doesn't fail and
everything works as expected.

Does anybody have any idea of what might be the problem here? Is there any
bug that could be related to this when running in cluster mode?

I appreciate any help.
Thanks,
Bernardo

--047d7b41890b184934052209392d
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div><div><div><div><div><div><div>Hello,<br><br></div>I a=
m trying to run some scala code in cluster mode using spark-submit. This co=
de uses addLibrary to link with a .so that exists in the machine, and this =
library has a function to be called natively (there&#39;s a native definiti=
on as needed in the code).<br><br></div>The problem I&#39;m facing is: when=
ever I try to run this code in cluster mode, spark fails with the following=
 message when trying to execute the native function:<br></div><div>java.lan=
g.UnsatisfiedLinkError: org.name.othername.ClassName.nativeMethod([B[B)[B<b=
r></div><div><br>Apparently, the library is being found by spark, but the r=
equired function isn&#39;t found.<br><br></div>When trying to run in client=
 mode, however, this doesn&#39;t fail and everything works as expected.<br>=
<br></div>Does anybody have any idea of what might be the problem here? Is =
there any bug that could be related to this when running in cluster mode?<b=
r><br></div>I appreciate any help.<br></div>Thanks,<br></div>Bernardo<br></=
div>

--047d7b41890b184934052209392d--