Return-Path: X-Original-To: apmail-spark-user-archive@minotaur.apache.org Delivered-To: apmail-spark-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 2D21617513 for ; Mon, 20 Apr 2015 15:32:42 +0000 (UTC) Received: (qmail 41894 invoked by uid 500); 20 Apr 2015 15:32:39 -0000 Delivered-To: apmail-spark-user-archive@spark.apache.org Received: (qmail 41811 invoked by uid 500); 20 Apr 2015 15:32:39 -0000 Mailing-List: contact user-help@spark.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@spark.apache.org Received: (qmail 41801 invoked by uid 99); 20 Apr 2015 15:32:39 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Apr 2015 15:32:39 +0000 X-ASF-Spam-Status: No, hits=4.8 required=5.0 tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_REPLY,HTML_MESSAGE,SPF_PASS,URI_HEX X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: message received from 54.164.171.186 which is an MX secondary for user@spark.apache.org) Received: from [54.164.171.186] (HELO mx1-us-east.apache.org) (54.164.171.186) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Apr 2015 15:32:33 +0000 Received: from mail-ie0-f172.google.com (mail-ie0-f172.google.com [209.85.223.172]) by mx1-us-east.apache.org (ASF Mail Server at mx1-us-east.apache.org) with ESMTPS id CB77543E56 for ; Mon, 20 Apr 2015 15:32:12 +0000 (UTC) Received: by iejt8 with SMTP id t8so118923364iej.2 for ; Mon, 20 Apr 2015 08:31:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=content-type:mime-version:subject:from:in-reply-to:date:cc :message-id:references:to; bh=HEVtrCt8olZrf+6d5q1+HO5yTjTBwuObjvAclf28zmE=; b=JCqpEK2zKDi3ji4C7vq4SGFHQpqKU45TzP9bJJsiSRH7PkC588xMiAuIPkUML74T86 +bcjGg1eVzjAUWBHjIRIxe4XZDkekNnk+rjbxKJ1+pekzm32iT6td06DOaTjgXeOv5Om eGPYsp7cuBKUPZOWOsEU1Qi9LxAxsP1ACpmHQRwG7SWu/nJ2C6S7anrsnkfA/kEjETTh U+DBCluM6dGfCjhz6bOZQy28dIBA+RAWBJM2lIc+/GZ5iU9yuxTABZTfYj5d3TjJWnnq BaGURNNZz4Z6rqt4L1lHqAuz3z0oPGhvs+Ol2ZLKkHiAbNy0QM2eMgy0dZSbi0gKTRRu YBHg== X-Received: by 10.43.169.137 with SMTP id nm9mr18819235icc.82.1429543887299; Mon, 20 Apr 2015 08:31:27 -0700 (PDT) Received: from [192.168.1.106] (c-50-179-151-18.hsd1.il.comcast.net. [50.179.151.18]) by mx.google.com with ESMTPSA id e69sm11594876ioe.11.2015.04.20.08.31.26 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Mon, 20 Apr 2015 08:31:26 -0700 (PDT) Content-Type: multipart/alternative; boundary="Apple-Mail=_B9EC1CEF-2698-48F3-A390-D21A250000A2" Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2070.6\)) Subject: Re: Configuring logging properties for executor From: Lan Jiang In-Reply-To: Date: Mon, 20 Apr 2015 10:31:27 -0500 Cc: user@spark.apache.org Message-Id: References: <1429539974930-22572.post@n3.nabble.com> <505AC66C-4F8F-4E9C-9773-2B638983854A@gmail.com> To: Michael Ryabtsev X-Mailer: Apple Mail (2.2070.6) X-Virus-Checked: Checked by ClamAV on apache.org --Apple-Mail=_B9EC1CEF-2698-48F3-A390-D21A250000A2 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Each application gets its own executor processes, so there should be no = problem running them in parallel.=20 Lan > On Apr 20, 2015, at 10:25 AM, Michael Ryabtsev = wrote: >=20 > Hi Lan,=20 >=20 > Thanks for fast response. It could be a solution if it works. I have = more than one log4 properties file, for different run modes like = debug/production, for executor and for application core. I think I would = like to keep them separate. Then, I suppose I should give all other = properties files a special names and keep the executor configuration = with the default name? Can I conclude that going this way I will not be = able to run several applications on the same cluster in parallel? >=20 > Regarding submit, I am not using it now, I submit from the code, but I = think I should consider this option. >=20 > Thanks. >=20 > On Mon, Apr 20, 2015 at 5:59 PM, Lan Jiang > wrote: > Rename your log4j_special.properties file as log4j.properties and = place it under the root of your jar file, you should be fine. >=20 > If you are using Maven to build your jar, please the log4j.properties = in the src/main/resources folder. >=20 > However, please note that if you have other dependency jar file in the = classpath that contains another log4j.properties file this way, it might = not work since the first log4j.properties file that is loaded will be = used. >=20 > You can also do spark-submit =E2=80=94file log4j_special.properties = =E2=80=A6 ,which should transfer your log4j property file to the worker = nodes automatically without you copying them manually. >=20 > Lan >=20 >=20 > > On Apr 20, 2015, at 9:26 AM, Michael Ryabtsev > wrote: > > > > Hi all, > > > > I need to configure spark executor log4j.properties on a standalone = cluster. > > It looks like placing the relevant properties file in the spark > > configuration folder and setting the = spark.executor.extraJavaOptions from > > my application code: > > sparkConf.set("spark.executor.extraJavaOptions", > > "-Dlog4j.configuration=3Dlog4j_special.properties"); > > does the work, and the executor logs are written in the required = place and > > level. As far as I understand, it works, because the spark = configuration > > folder is on the class path, and passing parameter without path = works here. > > However, I would like to avoid deploying these properties to each = worker > > spark configuration folder. > > I wonder, if I put the properties in my application jar, is there = any way of > > telling executor to load them? > > > > Thanks, > > > > > > > > -- > > View this message in context: = http://apache-spark-user-list.1001560.n3.nabble.com/Configuring-logging-pr= operties-for-executor-tp22572.html = > > Sent from the Apache Spark User List mailing list archive at = Nabble.com. > > > > = --------------------------------------------------------------------- > > To unsubscribe, e-mail: user-unsubscribe@spark.apache.org = > > For additional commands, e-mail: user-help@spark.apache.org = > > >=20 >=20 --Apple-Mail=_B9EC1CEF-2698-48F3-A390-D21A250000A2 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 Each application gets its own executor processes,  so = there should be no problem running them in parallel. 

Lan


On = Apr 20, 2015, at 10:25 AM, Michael Ryabtsev <michael.ry@gmail.com> wrote:

Hi Lan, 

Thanks for fast response. It could be a solution if it works. = I have more than one log4 properties file, for different run modes like = debug/production, for executor and for application core. I think I would = like to keep them separate. Then, I suppose I should give all other = properties files a special names and keep the executor configuration = with the default name? Can I conclude that going this way I will not be = able to run several applications on the same cluster in = parallel?

Regarding submit, I am not using it now, I submit from the = code, but I think I should consider this option.

Thanks.

On Mon, = Apr 20, 2015 at 5:59 PM, Lan Jiang <ljiang2@gmail.com> wrote:
Rename your = log4j_special.properties file as log4j.properties and place it under the = root of your jar file, you should be fine.

If you are using Maven to build your jar, please the log4j.properties in = the src/main/resources folder.

However, please note that if you have other dependency jar file in the = classpath that contains another log4j.properties file this way, it might = not work since the first log4j.properties file that is loaded will be = used.

You can also do spark-submit =E2=80=94file log4j_special.properties =E2=80= =A6 ,which should transfer your log4j property file to the worker nodes = automatically without you copying them manually.

Lan


> On Apr 20, 2015, at 9:26 AM, Michael Ryabtsev <michael.ry@gmail.com> wrote:
>
> Hi all,
>
> I need to configure spark executor log4j.properties on a standalone = cluster.
> It looks like placing the relevant properties file in the spark
> configuration folder and  setting the = spark.executor.extraJavaOptions from
> my application code:
> sparkConf.set("spark.executor.extraJavaOptions",
> "-Dlog4j.configuration=3Dlog4j_special.properties");
> does the work, and the executor logs are written in the required = place and
> level. As far as I understand, it works, because the spark = configuration
> folder is on the class path, and passing parameter without path = works here.
> However, I would like to avoid deploying these properties to each = worker
> spark configuration folder.
> I wonder, if I put the properties in my application jar, is there = any way of
> telling executor to load them?
>
> Thanks,
>
>
>
> --
> View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Configuring= -logging-properties-for-executor-tp22572.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> = ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>



= --Apple-Mail=_B9EC1CEF-2698-48F3-A390-D21A250000A2--