Mailing-List: contact dev-help@spark.apache.org; run by ezmlm
Precedence: bulk
MIME-Version: 1.0
References: <CAMhVi-Qv0ncgatTWELoaYdZ3ftYHSoSypJz8JTGfjA00m9zP0g@mail.gmail.com>
 <6a995af4-ed42-8966-9617-84f04ac27cb1@gmail.com>
In-Reply-To: <6a995af4-ed42-8966-9617-84f04ac27cb1@gmail.com>
From: Georg Heiler <georg.kf.heiler@gmail.com>
Date: Sat, 03 Jun 2017 06:27:07 +0000
Message-ID: <CAMhVi-Skw3OCk7_9i2s4G4y0OQ9FuLEeryNFZ6uLO7k71WjRGg@mail.gmail.com>
Subject: Re: spark messing up handling of native dependency code?
To: Maciej Szymkiewicz <mszymkiewicz@gmail.com>, dev@spark.apache.org
Content-Type: multipart/alternative; boundary="001a114e737c9780c30551085c8f"
archived-at: Sat, 03 Jun 2017 06:27:29 -0000

--001a114e737c9780c30551085c8f
Content-Type: text/plain; charset="UTF-8"

When tested without any parallelism the same problem persists. Actually,
NiFi shows the same issues. So probably it is not related to spark.

Maciej Szymkiewicz <mszymkiewicz@gmail.com> schrieb am Sa., 3. Juni 2017 um
01:37 Uhr:

> Maybe not related, but in general geotools are not thread safe,so using
> from workers is most likely a gamble.
> On 06/03/2017 01:26 AM, Georg Heiler wrote:
>
> Hi,
>
> There is a weird problem with spark when handling native dependency code:
> I want to use a library (JAI) with spark to parse some spatial raster
> files. Unfortunately, there are some strange issues. JAI only works when
> running via the build tool i.e. `sbt run` when executed in spark.
>
> When executed via spark-submit the error is:
>
>     java.lang.IllegalArgumentException: The input argument(s) may not be
> null.
>     at
> javax.media.jai.ParameterBlockJAI.getDefaultMode(ParameterBlockJAI.java:136)
>     at
> javax.media.jai.ParameterBlockJAI.<init>(ParameterBlockJAI.java:157)
>     at
> javax.media.jai.ParameterBlockJAI.<init>(ParameterBlockJAI.java:178)
>     at
> org.geotools.process.raster.PolygonExtractionProcess.execute(PolygonExtractionProcess.java:171)
>
> Which looks like some native dependency (I think GEOS is running in the
> background) is not there correctly.
>
> Assuming something is wrong with the class path I tried to run a plain
> java/scala function. but this one works just fine.
>
> Is spark messing with the class paths?
>
> I created a minimal example here:
> https://github.com/geoHeil/jai-packaging-problem
>
>
> Hope someone can shed some light on this problem,
> Regards,
> Georg
>
>
>

--001a114e737c9780c30551085c8f
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">When tested without any parallelism the same problem persi=
sts. Actually, NiFi shows the same issues. So probably it is not related to=
 spark.</div><br><div class=3D"gmail_quote"><div dir=3D"ltr">Maciej Szymkie=
wicz &lt;<a href=3D"mailto:mszymkiewicz@gmail.com">mszymkiewicz@gmail.com</=
a>&gt; schrieb am Sa., 3. Juni 2017 um 01:37=C2=A0Uhr:<br></div><blockquote=
 class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc soli=
d;padding-left:1ex">
 =20
   =20
 =20
  <div bgcolor=3D"#FFFFFF" text=3D"#000000">
    <p>Maybe not related, but in general geotools are not thread safe,so
      using from workers is most likely a gamble.<br>
    </p></div><div bgcolor=3D"#FFFFFF" text=3D"#000000">
    <div class=3D"m_-4283188811904423688moz-cite-prefix">On 06/03/2017 01:2=
6 AM, Georg Heiler
      wrote:<br>
    </div>
    <blockquote type=3D"cite">
      <div dir=3D"ltr">Hi,
        <div><br>
        </div>
        <div>There is a weird problem with spark when handling native
          dependency code:</div>
        <div>
          <div>I want to use a library (JAI) with spark to parse some
            spatial raster files. Unfortunately, there are some strange
            issues. JAI only works when running via the build tool i.e.
            `sbt run` when executed in spark.</div>
          <div><br>
          </div>
          <div>When executed via spark-submit the error is:</div>
          <div><br>
          </div>
          <div>=C2=A0 =C2=A0 java.lang.IllegalArgumentException: The input
            argument(s) may not be null.</div>
          <div>=C2=A0 =C2=A0 <span class=3D"m_-4283188811904423688Apple-tab=
-span" style=3D"white-space:pre-wrap">	</span>at
javax.media.jai.ParameterBlockJAI.getDefaultMode(ParameterBlockJAI.java:136=
)</div>
          <div>=C2=A0 =C2=A0 <span class=3D"m_-4283188811904423688Apple-tab=
-span" style=3D"white-space:pre-wrap">	</span>at
javax.media.jai.ParameterBlockJAI.&lt;init&gt;(ParameterBlockJAI.java:157)<=
/div>
          <div>=C2=A0 =C2=A0 <span class=3D"m_-4283188811904423688Apple-tab=
-span" style=3D"white-space:pre-wrap">	</span>at
javax.media.jai.ParameterBlockJAI.&lt;init&gt;(ParameterBlockJAI.java:178)<=
/div>
          <div>=C2=A0 =C2=A0 <span class=3D"m_-4283188811904423688Apple-tab=
-span" style=3D"white-space:pre-wrap">	</span>at
org.geotools.process.raster.PolygonExtractionProcess.execute(PolygonExtract=
ionProcess.java:171)</div>
          <div><br>
          </div>
          <div>Which looks like some native dependency (I think GEOS is
            running in the background) is not there correctly.</div>
          <div><br>
          </div>
          <div>Assuming something is wrong with the class path I tried
            to run a plain java/scala function. but this one works just
            fine.</div>
          <div><br>
          </div>
          <div>Is spark messing with the class paths?</div>
          <div><br>
          </div>
          <div>I created a minimal example here:</div>
          <div><a href=3D"https://github.com/geoHeil/jai-packaging-problem"=
 target=3D"_blank">https://github.com/geoHeil/jai-packaging-problem</a></di=
v>
          <div><br>
          </div>
          <div><br>
          </div>
          <div>Hope someone can shed some light on this problem,</div>
          <div>Regards,</div>
          <div>Georg=C2=A0</div>
        </div>
      </div>
    </blockquote>
    <br>
  </div></blockquote></div>

--001a114e737c9780c30551085c8f--