livy-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeff Zhang <zjf...@gmail.com>
Subject Re: How to use multiple programming languages in the same Spark context in Livy?
Date Mon, 11 Dec 2017 03:07:14 GMT
You can use dataframe in scala if this dataframe is registered in python.
Because they share the same sparkcontext.

I believe livy can meet your requirement. If you know zeppelin, the
behavior of livy now is very similar as zeppelin where you can run one
paragraph via scala and another paragraph via python or R. And they run in
the same spark application and be able to share data via sparkcontext.






David Hu <hoodavy@gmail.com>于2017年12月11日周一 上午10:44写道:

> Hi Jeff & Saisai,
>
> Thank you so much for the explanation and they are very helpful, also
> sorry for not replying in time.
>
> I had read all the links you provided and the impression I got is that,
> correct me if I am wrong, this feature would not allow different
> session-kind interacting with each other? What I mean is, if I ran one
> Scala kind and one Python kind in the same context, unless some kind of
> persistence it won't be possible to refer a dataframe variable in Python
> code that was defined in Scala right?
>
> The goal I want to achieve is to mix different languages together and run
> as one integrated spark job within which vars/methods defined in one
> language can be referred/used in other, because our users might have
> different programming background. It might sound silly but I am keen to
> know if that's possible under the current Livy infrastructure. Appreciate
> it if anyone could answer. Thanks in advance!
>
> Regards, Dawei
>
> 2017-12-04 8:30 GMT+08:00 Saisai Shao <sai.sai.shao@gmail.com>:
>
>> This feature is targeted for Livy 0.5.0 community version. But we already
>> back-ported this in HDP 2.6.3, so you can try this feature in HDP 2.6.3.
>>
>> You can check this doc (
>> https://github.com/apache/incubator-livy/blob/master/docs/rest-api.md)
>> to see the API difference for this feature.
>>
>> 2017-12-03 9:55 GMT+08:00 Jeff Zhang <zjffdu@gmail.com>:
>>
>>>
>>> It is implemented in https://issues.apache.org/jira/browse/LIVY-194
>>>
>>> But not release in apache version, HDP backport it in their distribution
>>>
>>>
>>>
>>> 胡大为(David) <hoodavy@gmail.com>于2017年12月2日周六 上午10:58写道:
>>>
>>>> I forgot to add the link reference and here it is.
>>>>
>>>> https://hortonworks.com/blog/hdp-2-6-3-dataplane-service/
>>>>
>>>> Regards, Dawei
>>>>
>>>> On 2 Dec 2017, at 8:24 AM, 胡大为(David) <hoodavy@gmail.com>
wrote:
>>>>
>>>>
>>>> Hi all,
>>>>
>>>> I was reading the HDP 2.6.3 release notes and it mentions that Livy
>>>> service is able to multiple programming languages in the same Spark
>>>> context, but I went through all the Livy document and examples I can find
>>>> but so far haven’t found out how to get it work. Currently I am using the
>>>> latest Livy 0.4 to submit Scala code only and it would be awesome to mix
it
>>>> with Python or R code in the same session. Much appreciate it anyone could
>>>> give me some clue about this.
>>>>
>>>> Thanks in advance and have a good day :)
>>>>
>>>> Regards, Dawei
>>>>
>>>>
>>>>
>>
>

Mime
View raw message