livy-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Meisam Fathi <meisam.fa...@gmail.com>
Subject Re: How do I set spark conf parameters?
Date Wed, 31 Jan 2018 15:45:57 GMT
On a more general note, with or without Livy, Spark configurations should
be set before creating the Spark contex (sc). Setting configs after sc is
created has no effect.

Thanks,
Meisam

On Wed, Jan 31, 2018 at 3:14 AM Stefan Miklosovic <miklosovic@gmail.com>
wrote:

> livyClient = new LivyClientBuilder()
>   .setAll(withSparkProperties())
>   .setURI(new URI(LIVY_URI))
>   .build()
>
>
>
> On Wed, Jan 31, 2018 at 12:13 PM, Stefan Miklosovic <miklosovic@gmail.com>
> wrote:
>
>> Yes, I am using setAll(Properties).
>>
>> On Wed, Jan 31, 2018 at 11:45 AM, kant kodali <kanth909@gmail.com> wrote:
>>
>>> Sorry. I found an answer online.It should be something like this
>>>
>>> new LivyClientBuilder().setConf("spark.es.index.auto.create", "true").setConf("spark.cassandra.connection.host",
"127.0.0.01").build();
>>>
>>>
>>> On Wed, Jan 31, 2018 at 1:16 AM, kant kodali <kanth909@gmail.com> wrote:
>>>
>>>> Hi All,
>>>>
>>>>
>>>> How do I set Spark Conf Parameters ? The below doesnt seem to get
>>>> picked up? If so, how can I change my program such that it can pick it up.
>>>> I am not seeing a way if sparkcontext is already created?
>>>>
>>>> public String call(JobContext ctx) throws Exception {
>>>>     ctx.sc().setLogLevel("INFO");
>>>>     ctx.sc().getConf()
>>>>         .set("spark.cassandra.connection.host", config.getString("cassandra.host"))
>>>>         .set("spark.cassandra.auth.username", config.getString("cassandra.user"))
>>>>         .set("spark.cassandra.auth.password", config.getString("cassandra.pass"))
>>>>         .set("es.index.auto.create", "true");
>>>>
>>>> }
>>>>
>>>>
>>>> Thanks!
>>>>
>>>>
>>>
>>
>>
>> --
>> Stefan Miklosovic
>>
>
>
>
> --
> Stefan Miklosovic
>

Mime
View raw message