livy-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 王峰 <wangfengfight...@gmail.com>
Subject Re: Livy POST Sessions can not work with conf
Date Thu, 30 Nov 2017 02:32:59 GMT
No, that`s not the point. I was just have no idea which value (false or
"false") should be selected. Now , you have met this question again?

Best wishes

2017-11-30 1:42 GMT+08:00 Meisam Fathi <meisam.fathi@gmail.com>:

> I am curious to know if this is resolved yet. I see that in the original
> email you used "conf" : {"spark.dynamicAllocation.enabled":false
> ,"spark.shuffle.service.enabled":false}. But in the second email, you
> used "conf" : {"spark.dynamicAllocation.enabled":"false", "spark.
> shuffle.service.enabled":"false"}.  Was that the issue?
>
> Thanks,
> Meisam
>
> On Fri, Nov 3, 2017 at 3:18 AM 王峰 <wangfengfighting@gmail.com> wrote:
>
>> I have try to use livy 0.4 with POST requst
>> POST :livyurl/sessions
>> Content-Type: application/json; charset=utf-8
>> {
>>   "kind" : "spark",
>>   "proxyUser" : "root",
>>   "executorMemory" : "4G",
>>   "executorCores": 4,
>>   "numExecutors" : 4,
>>    "conf" : {"spark.dynamicAllocation.enabled":"false","spark.
>> shuffle.service.enabled":"false"}
>>
>> }
>>
>> But it doesn`t come into force...maybe I should try another way...
>>
>> 2017-11-03 17:54 GMT+08:00 Saisai Shao <sai.sai.shao@gmail.com>:
>>
>>> I think it should be worked, can you please test with 0.4 version of
>>> Livy. Also "conf" should be a map of string key to string value.
>>>
>>> "conf" : {"spark.dynamicAllocation.enabled":"false","spark.shuffle.se
>>> rvice.enabled":"false"}
>>>
>>> Besides, please be aware in the current Livy we only tested on local and
>>> yarn mode, we don't guarantee the correct behavior using Mesos cluster
>>> manager.
>>>
>>> On Fri, Nov 3, 2017 at 5:36 PM, 王峰 <wangfengfighting@gmail.com> wrote:
>>>
>>>> Hello everyone , I have meet a problem about Livy-0.3 when I run `POST
>>>> /sessions` to create a new interactive spark session
>>>>
>>>>
>>>> here is the post body
>>>> ```Content-Type: application/json; charset=utf-8
>>>> {
>>>>   "kind" : "spark",
>>>>   "proxyUser" : "root",
>>>>   "executorMemory" : "4G",
>>>>   "executorCores": 4,
>>>>   "numExecutors" : 4,
>>>>    "conf" : {"spark.dynamicAllocation.enabled":false,"spark.shuffle.
>>>> service.enabled":false}
>>>> }
>>>> ```
>>>>
>>>> However, I found that this Livy session  allocated all of resources in
>>>> Mesos UI as Pic shows
>>>>
>>>> [image: 内嵌图片 1]
>>>>
>>>> It seem like that `conf` did not worked but numExecutors ,
>>>> executorCores and executorMemory worked well..
>>>>
>>>> please help me  thanks...
>>>>
>>>
>>>

Mime
View raw message