livy-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kyunam Kim <kim.kyu...@gmail.com>
Subject Re: Not understanding "jars" & "conf": {"spark.jars":" options in POST /sessions
Date Sat, 03 Feb 2018 04:38:28 GMT
It turns out I need to prefix with file:// in either "jars" or "spark.jars".

Thanks,
Q

On Thu, Feb 1, 2018 at 2:10 PM, Kyunam Kim <kim.kyunam@gmail.com> wrote:

> Here are my findings.
> If you could explain why I'm seeing these behaviors, that'll be great.
>
> A. I have to specify BOTH jars and spark.jars
> I have to have jars locally in the Livy server AND same jars in HDFS.
> For example, {"jars": ["/tmp/myLib0.jar,/tmp/myUber.jar"], "conf":
> {"spark.jars": "/tmp/myLib0.jar,/tmp/myUber.jar","spark.files.overwrite":"
> true"}}
>
> Why do I need to specify both "jars" and "spark.jars"? Is this as designed?
>
> B. error: not found: value Source
> 'myUber.jar' contains 'Source' class. When {"jars": ["/tmp/myUber.jar"],
> "conf": {"spark.jars": "/tmp/myUber.jar","spark.files.overwrite":"true"}}
> is used, it doesn't see 'Source' class. Notice only 'myUber.jar' is
> included.
>
> I had to supply the original library jar that contains 'Source' class in
> addition to 'myUber.jar' as follows.
> {"jars": ["/tmp/myLib0.jar,/tmp/myUber.jar"], "conf": {"spark.jars":
> "/tmp/myLib0.jar,/tmp/myUber.jar","spark.files.overwrite":"true"}}
> So, 'Source' class exists in 'myLib0.jar' and 'myUber.jar'.
>
> I also tried {"jars": ["/tmp/myLib0.jar,/tmp/myApp.jar"], "conf":
> {"spark.jars": "/tmp/myLib0.jar,/tmp/myApp.jar","spark.files.overwrite":"
> true"}}
> In this case, 'Source' only exists in 'myLib0.jar'. But the same error
> pops up.
>
> Why 'Source' has be to be repeated through multiple jars?
>
> Thanks,
> Q
>



-- 

Thanks,
Q

Mime
View raw message