Return-Path: X-Original-To: apmail-spark-user-archive@minotaur.apache.org Delivered-To: apmail-spark-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id E72041794B for ; Thu, 15 Jan 2015 21:23:59 +0000 (UTC) Received: (qmail 93258 invoked by uid 500); 15 Jan 2015 21:23:59 -0000 Delivered-To: apmail-spark-user-archive@spark.apache.org Received: (qmail 93184 invoked by uid 500); 15 Jan 2015 21:23:59 -0000 Mailing-List: contact user-help@spark.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@spark.apache.org Received: (qmail 93174 invoked by uid 99); 15 Jan 2015 21:23:59 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Jan 2015 21:23:59 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,WEIRD_PORT X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of szheng.code@gmail.com designates 209.85.223.173 as permitted sender) Received: from [209.85.223.173] (HELO mail-ie0-f173.google.com) (209.85.223.173) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Jan 2015 21:23:31 +0000 Received: by mail-ie0-f173.google.com with SMTP id y20so17343965ier.4 for ; Thu, 15 Jan 2015 13:23:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=from:to:references:in-reply-to:subject:date:message-id:mime-version :content-type:thread-index:content-language; bh=y5GOBnJ2HqHP1pw5Losh/7Vaho46xzALxpemjhhVOZE=; b=Id2GCgJ8u2LI5B99GhFo1ZD3QZ27Epk8ijMpEYMfnmZPNvmZ38xv4nKQ4dLN7bDiB4 B4IH83vhwyUwvbulm6X+9ZE1SMo2NFDlM3GCqpERZiw5KcUvKmGoJMDsMZ+6GBqjjbtX Ol4L+WgM7Yp120ONyERtbOStunH+DSMxhy/HqPUuenc3vkJNBGZcb++Ib9k/RrNaqjsv 39CnvbSvnCbhEAoaDmLuFB9IjNNd4dPBWI0b+XFwpR8Dj4FBAGBE92p7L25HJWRwJRK0 oCkOpnJVgPxIolgYnjwKOanbpTVA+w4k94md1A89YRiXEQRyM/oxw+RixJ8tTSMlKGcl P5uQ== X-Received: by 10.51.17.1 with SMTP id ga1mr12664850igd.33.1421357009762; Thu, 15 Jan 2015 13:23:29 -0800 (PST) Received: from NY02913D ([38.104.189.222]) by mx.google.com with ESMTPSA id i20sm237041igh.16.2015.01.15.13.23.28 for (version=TLSv1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Thu, 15 Jan 2015 13:23:28 -0800 (PST) From: "Shuai Zheng" To: References: <1a4201d03106$b04ac7c0$10e05740$@gmail.com> In-Reply-To: <1a4201d03106$b04ac7c0$10e05740$@gmail.com> Subject: RE: Executor parameter doesn't work for Spark-shell on EMR Yarn Date: Thu, 15 Jan 2015 16:23:26 -0500 Message-ID: <1a7601d03109$8115a350$8340e9f0$@gmail.com> MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_NextPart_000_1A77_01D030DF.983FC260" X-Mailer: Microsoft Outlook 14.0 Thread-Index: AQIH0gCPuAr5btZG1LBLfYIukLOTTJxSbMFw Content-Language: en-us X-Virus-Checked: Checked by ClamAV on apache.org ------=_NextPart_000_1A77_01D030DF.983FC260 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit I figure out the second question, because if I don't pass in the num of partition for the test data, it will by default assume has max executors (although I don't know what is this default max num). val lines = sc.parallelize(List("-240990|161327,9051480,0,2,30.48,75", "-240990|161324,9051480,0,2,30.48,75"),2) will only trigger 2 executors. So I think the default executors num will be decided by the first RDD operation need to send to executors. This give me a weird way to control the num of executors (a fake/test code piece run to kick off the executors first, then run the real behavior - because executor will run the whole lifecycle of the applications? Although this may not have any real value in practice J But I still need help for my first question. Thanks a lot. Regards, Shuai From: Shuai Zheng [mailto:szheng.code@gmail.com] Sent: Thursday, January 15, 2015 4:03 PM To: user@spark.apache.org Subject: RE: Executor parameter doesn't work for Spark-shell on EMR Yarn Forget to mention, I use EMR AMI 3.3.1, Spark 1.2.0. Yarn 2.4. The spark is setup by the standard script: s3://support.elasticmapreduce/spark/install-spark From: Shuai Zheng [mailto:szheng.code@gmail.com] Sent: Thursday, January 15, 2015 3:52 PM To: user@spark.apache.org Subject: Executor parameter doesn't work for Spark-shell on EMR Yarn Hi All, I am testing Spark on EMR cluster. Env is a one node cluster r3.8xlarge. Has 32 vCore and 244G memory. But the command line I use to start up spark-shell, it can't work. For example: ~/spark/bin/spark-shell --jars /home/hadoop/vrisc-lib/aws-java-sdk-1.9.14/lib/*.jar --num-executors 6 --executor-memory 10G Neither num-executors nor memory setup works. And more interesting, if I use test code: val lines = sc.parallelize(List("-240990|161327,9051480,0,2,30.48,75", "-240990|161324,9051480,0,2,30.48,75")) var count = lines.mapPartitions(dynamoDBBatchWriteFunc).collect.sum It will start 32 executors (then I assume it try to start all executors for every vCore). But if I use some real data to do it (the file size is 200M): val lines = sc.textFile("s3://.../part-r-00000") var count = lines.mapPartitions(dynamoDBBatchWriteFunc).collect.sum It will only start 4 executors, which map to the number of HDFS split (200M will have 4 splits). So I have two questions: 1, Why the setup parameter is ignored by Yarn? How can I limit the number of executors I can run? 2, Why my much smaller test data set will trigger 32 executors but my real 200M data set will only have 4 executors? So how should I control the executor setup on the spark-shell? And I print the sparkConf, it looks like much less than I expect, and I don't see my pass in parameter show there. scala> sc.getConf.getAll.foreach(println) (spark.tachyonStore.folderName,spark-af0c4d42-fe4d-40b0-a3cf-25b6a9e16fa0) (spark.app.id,local-1421353031552) (spark.eventLog.enabled,true) (spark.executor.id,driver) (spark.repl.class.uri,http://10.181.82.38:58415) (spark.driver.host,ip-10-181-82-38.ec2.internal) (spark.executor.extraJavaOptions,-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:MaxHeapFreeRatio=70) (spark.app.name,Spark shell) (spark.fileserver.uri,http://10.181.82.38:54666) (spark.jars,file:/home/hadoop/vrisc-lib/aws-java-sdk-1.9.14/lib/aws-java-sdk -1.9.14.jar) (spark.eventLog.dir,hdfs:///spark-logs) (spark.executor.extraClassPath,/home/hadoop/spark/classpath/emr/*:/home/hado op/spark/classpath/emrfs/*:/home/hadoop/share/hadoop/common/lib/*:/home/hado op/.versions/2.4.0/share/hadoop/common/lib/hadoop-lzo.jar) (spark.master,local[*]) (spark.driver.port,54191) (spark.driver.extraClassPath,/home/hadoop/spark/classpath/emr/*:/home/hadoop /spark/classpath/emrfs/*:/home/hadoop/share/hadoop/common/lib/*:/home/hadoop /.versions/2.4.0/share/hadoop/common/lib/hadoop-lzo.jar) I search the old threads, attached email answer the question about why vCore setup doesn't work. But I think this is not same issue as me. Otherwise then default Yarn Spark setup can't do any adjustment? Regards, Shuai ------=_NextPart_000_1A77_01D030DF.983FC260 Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

I figure out the second question, because if I = don’t pass in the num of partition for the test data, it will by = default assume has max executors (although I don’t know what is = this default max num).

 

val lines =3D = sc.parallelize(List("-240990|161327,9051480,0,2,30.48,75", = "-240990|161324,9051480,0,2,30.48,75"),2)

will only trigger 2 executors.

 

So I think the default = executors num will be decided by the first RDD operation need to send to = executors. This give me a weird way to control the num of executors (a = fake/test code piece run to kick off the executors first, then run the = real behavior – because executor will run the whole lifecycle of = the applications? Although this may not have any real value in practice = J

 

But I still need help = for my first question.

 

Thanks a = lot.

 

Regards,

 

Shuai

 

From:= = Shuai Zheng [mailto:szheng.code@gmail.com]
Sent: Thursday, = January 15, 2015 4:03 PM
To: = user@spark.apache.org
Subject: RE: Executor parameter doesn't = work for Spark-shell on EMR Yarn

 

Forget to mention, I use EMR AMI 3.3.1, Spark = 1.2.0. Yarn 2.4. The spark is setup by the standard script: s3://support.elasticmapreduce/spark/install-spark<= /o:p>

 

 

From:= = Shuai Zheng [mailto:szheng.code@gmail.com] =
Sent: Thursday, January 15, 2015 3:52 PM
To: user@spark.apache.org
Sub= ject: Executor parameter doesn't work for Spark-shell on EMR = Yarn

 

Hi = All,

 

I am testing Spark on EMR cluster. Env is a one node = cluster r3.8xlarge. Has 32 vCore and 244G memory.

 

But the = command line I use to start up spark-shell, it can’t work. For = example:

 

~/spark/bin/spark-shell --jars = /home/hadoop/vrisc-lib/aws-java-sdk-1.9.14/lib/*.jar --num-executors 6 = --executor-memory 10G

 

Neither = num-executors nor memory setup works.

 

And more = interesting, if I use test code:

val = lines =3D = sc.parallelize(List("-240990|161327,9051480,0,2,30.48,75", = "-240990|161324,9051480,0,2,30.48,75"))

var count =3D = lines.mapPartitions(dynamoDBBatchWriteFunc).collect.sum

 

It will = start 32 executors (then I assume it try to start all executors for = every vCore).

 

But if I use some real data to do it (the file size is = 200M):

val lines =3D = sc.textFile("s3://.../part-r-00000")

var count =3D = lines.mapPartitions(dynamoDBBatchWriteFunc).collect.sum

It will only start 4 executors, which map to the = number of HDFS split (200M will have 4 splits).

 

So I have = two questions:

1, Why the setup = parameter is ignored by Yarn? How can I limit the number of executors I = can run?

2, Why my much smaller test = data set will trigger 32 executors but my real 200M data set will only = have 4 executors?

 

So how = should I control the executor setup on the spark-shell? And I print the = sparkConf, it looks like much less than I expect, and I don’t see = my pass in parameter show there.

 

scala> = sc.getConf.getAll.foreach(println)

(spark.tachyonStore.folderName,spark-af0c4d42-fe4d-40b0-a3cf-25= b6a9e16fa0)

(spark.app.id,local-1421353031552)

(spark.eventLog.enabled,true)

(spark.executor.id,driver)

(spark.repl.class.uri,http://10.181.82.38:58415)

(spark.driver.host,ip-10-181-82-38.ec2.internal)

(spark.executor.extraJavaOptions,-verbose:gc = -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+UseConcMarkSweepGC = -XX:CMSInitiatingOccupancyFraction=3D70 = -XX:MaxHeapFreeRatio=3D70)

(spark.app.name,Spark shell)

(spark.fileserver.uri,http://10.181.82.38:54666)

(spark.jars,file:/home/hadoop/vrisc-lib/aws-java-sdk-1.9.14/lib= /aws-java-sdk-1.9.14.jar)

(spark.eventLog.dir,hdfs:///spark-logs)

(spark.executor.extraClassPath,/home/hadoop/spark/classpath/emr= /*:/home/hadoop/spark/classpath/emrfs/*:/home/hadoop/share/hadoop/common/= lib/*:/home/hadoop/.versions/2.4.0/share/hadoop/common/lib/hadoop-lzo.jar= )

(spark.master,local[*])

(spark.driver.port,54191)

(spark.driver.extraClassPath,/home/hadoop/spark/classpath/emr/*= :/home/hadoop/spark/classpath/emrfs/*:/home/hadoop/share/hadoop/common/li= b/*:/home/hadoop/.versions/2.4.0/share/hadoop/common/lib/hadoop-lzo.jar)<= o:p>

 

I search the old threads, attached email answer the = question about why vCore setup doesn’t work. But I think this is = not same issue as me. Otherwise then default Yarn Spark setup = can’t do any adjustment?

 

Regards,

 

Shuai

 

 

 

 

------=_NextPart_000_1A77_01D030DF.983FC260--