livy-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stefan Miklosovic <mikloso...@gmail.com>
Subject Re: How do I set spark conf parameters?
Date Wed, 31 Jan 2018 11:13:05 GMT
Yes, I am using setAll(Properties).

On Wed, Jan 31, 2018 at 11:45 AM, kant kodali <kanth909@gmail.com> wrote:

> Sorry. I found an answer online.It should be something like this
>
> new LivyClientBuilder().setConf("spark.es.index.auto.create", "true").setConf("spark.cassandra.connection.host",
"127.0.0.01").build();
>
>
> On Wed, Jan 31, 2018 at 1:16 AM, kant kodali <kanth909@gmail.com> wrote:
>
>> Hi All,
>>
>>
>> How do I set Spark Conf Parameters ? The below doesnt seem to get picked
>> up? If so, how can I change my program such that it can pick it up. I am
>> not seeing a way if sparkcontext is already created?
>>
>> public String call(JobContext ctx) throws Exception {
>>     ctx.sc().setLogLevel("INFO");
>>     ctx.sc().getConf()
>>         .set("spark.cassandra.connection.host", config.getString("cassandra.host"))
>>         .set("spark.cassandra.auth.username", config.getString("cassandra.user"))
>>         .set("spark.cassandra.auth.password", config.getString("cassandra.pass"))
>>         .set("es.index.auto.create", "true");
>>
>> }
>>
>>
>> Thanks!
>>
>>
>


-- 
Stefan Miklosovic

Mime
View raw message