From user-return-217-archive-asf-public=cust-asf.ponee.io@livy.incubator.apache.org Wed Jan 31 12:14:55 2018 Return-Path: X-Original-To: archive-asf-public@eu.ponee.io Delivered-To: archive-asf-public@eu.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by mx-eu-01.ponee.io (Postfix) with ESMTP id AF9FF180662 for ; Wed, 31 Jan 2018 12:14:55 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 9F586160C35; Wed, 31 Jan 2018 11:14:55 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id C0F06160C25 for ; Wed, 31 Jan 2018 12:14:54 +0100 (CET) Received: (qmail 85231 invoked by uid 500); 31 Jan 2018 11:14:54 -0000 Mailing-List: contact user-help@livy.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@livy.incubator.apache.org Delivered-To: mailing list user@livy.incubator.apache.org Received: (qmail 85221 invoked by uid 99); 31 Jan 2018 11:14:53 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 31 Jan 2018 11:14:53 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 2334AD9299 for ; Wed, 31 Jan 2018 11:14:53 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.629 X-Spam-Level: ** X-Spam-Status: No, score=2.629 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, KAM_INFOUSMEBIZ=0.75, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id JonMbrZERQS6 for ; Wed, 31 Jan 2018 11:14:52 +0000 (UTC) Received: from mail-qt0-f174.google.com (mail-qt0-f174.google.com [209.85.216.174]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 5FBCC6125E for ; Wed, 31 Jan 2018 11:13:30 +0000 (UTC) Received: by mail-qt0-f174.google.com with SMTP id l20so21246492qtj.11 for ; Wed, 31 Jan 2018 03:13:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=b0v+yixfeYSKauwKAwuFvVivYumUyRmDqiHGHkswLD4=; b=F6++FPA/C+1qFJSKbaahstmKD2G5dYVHQvFg05z/px0lDs2+163eSQmkUoANVFW/rJ OOp89zdlGqmK6faCIqB2Y/FKthYvEIIwR7szmMjo9wu+E7vY4w/InM5uMLQWQLRPnWXK UV3Ig2k/L4pIsuOvz1kWJU+KIZDHYqgv8GwZfg4xUYd5kkRQAjRstEi1o7qjDvMmh+PH 3SM/lALmZLXJj1mhlTvrSHjZgEkIVXZB682ocPIoKSPrx5v5UWB4k7PKnjnE5envJchI ievDgOX25gdtUAJYX3fHO9aOvL13n7XbAmHfAaVqsqsBcKjvhTDD1HvJIHQSKYpH794R M5Gw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=b0v+yixfeYSKauwKAwuFvVivYumUyRmDqiHGHkswLD4=; b=n1149BIgdhZ6Mb7c+0p8dcM0b0X7dahwTWh2Kh0QuYz45CESRboPHzN+bT+qEeiYHa dBRFZRI7dRgXDBXwPW06bCfM9oYuBXfbOiBWs3yQldvYYkx7WlSRTJRTK2NhgNmVWhCJ CAPPtOh0D9mDDVQAL+E5wvlSk+KI08nInf/Nsv546bIO1tEhd94W+yvFK8ax1MbILG7H +0cayOimFbkiRdLC76IvnjgCJ40HPK4u3tQt9wIDOOG/G3P5A9jOx2k6gdOfo6B60aCx oBEk+jt7BzRELJMq3PmwliHmBHoFe7GWJxwSrIOEu9IYetJuuLmMd1S7VHsrnE1VzmJu dTJA== X-Gm-Message-State: AKwxytc1NcYMJZkHmBXlcxQwlZjGOxAVf2mmdTuriajBzHGEk5goEvbO yHrV7s1I4f2PiEVBCHPDykGnsH7nz5h1GFciGFT4xg== X-Google-Smtp-Source: AH8x226cFQbX8/bMcWo9uWXhnnvj23gwrniMDj6S5yDVZilpjpfEmP+02rYMDKHRS7idlORloLdqnCG7imqHvFb75yo= X-Received: by 10.237.43.39 with SMTP id p36mr4493744qtd.92.1517397209126; Wed, 31 Jan 2018 03:13:29 -0800 (PST) MIME-Version: 1.0 Received: by 10.55.156.206 with HTTP; Wed, 31 Jan 2018 03:13:28 -0800 (PST) In-Reply-To: References: From: Stefan Miklosovic Date: Wed, 31 Jan 2018 12:13:28 +0100 Message-ID: Subject: Re: How do I set spark conf parameters? To: user@livy.incubator.apache.org Content-Type: multipart/alternative; boundary="94eb2c0566c4a21c840564109116" --94eb2c0566c4a21c840564109116 Content-Type: text/plain; charset="UTF-8" livyClient = new LivyClientBuilder() .setAll(withSparkProperties()) .setURI(new URI(LIVY_URI)) .build() On Wed, Jan 31, 2018 at 12:13 PM, Stefan Miklosovic wrote: > Yes, I am using setAll(Properties). > > On Wed, Jan 31, 2018 at 11:45 AM, kant kodali wrote: > >> Sorry. I found an answer online.It should be something like this >> >> new LivyClientBuilder().setConf("spark.es.index.auto.create", "true").setConf("spark.cassandra.connection.host", "127.0.0.01").build(); >> >> >> On Wed, Jan 31, 2018 at 1:16 AM, kant kodali wrote: >> >>> Hi All, >>> >>> >>> How do I set Spark Conf Parameters ? The below doesnt seem to get picked >>> up? If so, how can I change my program such that it can pick it up. I am >>> not seeing a way if sparkcontext is already created? >>> >>> public String call(JobContext ctx) throws Exception { >>> ctx.sc().setLogLevel("INFO"); >>> ctx.sc().getConf() >>> .set("spark.cassandra.connection.host", config.getString("cassandra.host")) >>> .set("spark.cassandra.auth.username", config.getString("cassandra.user")) >>> .set("spark.cassandra.auth.password", config.getString("cassandra.pass")) >>> .set("es.index.auto.create", "true"); >>> >>> } >>> >>> >>> Thanks! >>> >>> >> > > > -- > Stefan Miklosovic > -- Stefan Miklosovic --94eb2c0566c4a21c840564109116 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable 
livyClient =3D new LivyClientBuilder(=
)
  .setAll(withSparkProperties())
  .setURI(new URI(LIVY_URI))
  .build()


On Wed, Jan 31, 2018 = at 12:13 PM, Stefan Miklosovic <miklosovic@gmail.com> wro= te:
Yes, I am using setA= ll(Properties).

=
On Wed, Jan 31, 2018 at 11:45 AM, kant kodali <kanth909@gmail.com> wrote:
Sorry. I found an answer online.It should be something li= ke this

new LivyCl= ientBuilder().setConf("spark.es.index.auto.create", "tr= ue").setConf("spark.cassandra.connection.host", "127.0.0.01").build();

On Wed, Jan = 31, 2018 at 1:16 AM, kant kodali <kanth909@gmail.com> wrote= :
Hi All,

=

How do I set Spark Conf Parameters ? The below doesnt s= eem to get picked up? If so, how can I change my program such that it can p= ick it up. I am not seeing a way if sparkcontext is already created?
<= div>
public S=
tring call(JobContext ctx) throws Exception {
    ctx.sc().setLogLevel("INFO");
    ctx.sc().getConf()
        .set("spark.cassandra.connection.host"=
;, config.getString("ca=
ssandra.host"))
        .set("spark.cassandra.auth.username", config=
.getString("cassandra.user"))
        .set("spark.cassandra.auth.password", config.getString("cassandra.pass"))
        .set(<=
span style=3D"color:rgb(0,128,0);font-weight:bold">"es.index.auto.crea=
te", "=
true");
}

Thanks!




<= /div>--
Stef= an Miklosovic


--
Stefan Miklosovic
--94eb2c0566c4a21c840564109116--