Mailing-List: contact dev-help@spark.apache.org; run by ezmlm
Precedence: bulk
MIME-Version: 1.0
From: Prabhu Joseph <prabhujose.gates@gmail.com>
Date: Thu, 23 Jun 2016 17:51:52 +0530
Message-ID: <CAKRKJ1Psa-Mv827-howuWezeEDR8kYWYRpFMCMiUJWerBmkUig@mail.gmail.com>
Subject: Spark Thrift Server Concurrency
To: Spark dev list <dev@spark.apache.org>, user <user@spark.apache.org>
Content-Type: multipart/alternative; boundary=001a114f865086f2f60535f11966
archived-at: Thu, 23 Jun 2016 12:22:10 -0000

--001a114f865086f2f60535f11966
Content-Type: text/plain; charset=UTF-8

Hi All,

   On submitting 20 parallel same SQL query to Spark Thrift Server, the
query execution time for some queries are less than a second and some are
more than 2seconds. The Spark Thrift Server logs shows all 20 queries are
submitted at same time 16/06/23 12:12:01 but the result schema are at
different times.

16/06/23 12:12:01 INFO SparkExecuteStatementOperation: Running query
'select distinct val2 from philips1 where key>=1000 and key<=1500

16/06/23 12:12:*02* INFO SparkExecuteStatementOperation: Result Schema:
ArrayBuffer(val2#2110)
16/06/23 12:12:*03* INFO SparkExecuteStatementOperation: Result Schema:
ArrayBuffer(val2#2182)
16/06/23 12:12:*04* INFO SparkExecuteStatementOperation: Result Schema:
ArrayBuffer(val2#2344)
16/06/23 12:12:*05* INFO SparkExecuteStatementOperation: Result Schema:
ArrayBuffer(val2#2362)

There are sufficient executors running on YARN. The concurrency is affected
by Single Driver. How to improve the concurrency and what are the best
practices.

Thanks,
Prabhu Joseph

--001a114f865086f2f60535f11966
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div><div><div><div>Hi All,<br><br></div>=C2=A0=C2=A0 On s=
ubmitting 20 parallel same SQL query to Spark Thrift Server, the query exec=
ution time for some queries are less than a second and some are more than 2=
seconds. The Spark Thrift Server logs shows all 20 queries are submitted at=
 same time 16/06/23 12:12:01 but the result schema are at different times.<=
br><br>16/06/23 12:12:01 INFO SparkExecuteStatementOperation: Running query=
 &#39;select distinct val2 from philips1 where key&gt;=3D1000 and key&lt;=
=3D1500<br><br>16/06/23 12:12:<b>02</b> INFO SparkExecuteStatementOperation=
: Result Schema: ArrayBuffer(val2#2110)<br>16/06/23 12:12:<b>03</b> INFO Sp=
arkExecuteStatementOperation: Result Schema: ArrayBuffer(val2#2182)<br>16/0=
6/23 12:12:<b>04</b> INFO SparkExecuteStatementOperation: Result Schema: Ar=
rayBuffer(val2#2344)<br>16/06/23 12:12:<b>05</b> INFO SparkExecuteStatement=
Operation: Result Schema: ArrayBuffer(val2#2362)<br><br></div>There are suf=
ficient executors running on YARN. The concurrency is affected by Single Dr=
iver. How to improve the concurrency and what are the best practices.<br><b=
r></div>Thanks,<br></div>Prabhu Joseph<br></div>

--001a114f865086f2f60535f11966--