Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 21CA5200C45 for ; Tue, 28 Mar 2017 18:19:12 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 20329160B89; Tue, 28 Mar 2017 16:19:12 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 15F90160B7E for ; Tue, 28 Mar 2017 18:19:10 +0200 (CEST) Received: (qmail 9999 invoked by uid 500); 28 Mar 2017 16:19:09 -0000 Mailing-List: contact user-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hive.apache.org Delivered-To: mailing list user@hive.apache.org Received: (qmail 9989 invoked by uid 99); 28 Mar 2017 16:19:09 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Mar 2017 16:19:09 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 5EB2EC0D5B for ; Tue, 28 Mar 2017 16:19:09 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.679 X-Spam-Level: * X-Spam-Status: No, score=1.679 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RCVD_IN_SORBS_SPAM=0.5, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id iUFbdvdnhhIz for ; Tue, 28 Mar 2017 16:19:07 +0000 (UTC) Received: from mail-qt0-f182.google.com (mail-qt0-f182.google.com [209.85.216.182]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id 738305F29A for ; Tue, 28 Mar 2017 16:19:07 +0000 (UTC) Received: by mail-qt0-f182.google.com with SMTP id n21so68464030qta.1 for ; Tue, 28 Mar 2017 09:19:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=lyGUcSjrDXDJExumEBgUyuLlPJ8eSWbc/cGuq6Pshvw=; b=Fel7nX9CazqIjysxBmzh75bZgJll+nFZnUiUFPHzU/kWXv24Tx7W7qkKalAzdLvWzO 9hbpLHl+ixaZbq1Vk5RdlvXLOzX3pnU3K7Fj33YRC/qCPkYwyVy17pw9FSPn0utGL4oI vDT0k8O0FRmtquKdAa2pgCOQdgjI810rv8RZRhlXT65N50l84vDXjH8yltxB3h0obuhV 6laFIsqt//j3iZaJwqRKfrCYceERg50ZIq5VpqbW5NGzrGi3qP5xbXVfZp3uZtKE66GU D3vg5l1uYGaJAs5Px0Z3Hj52yOplQR+RQrAwe2ThCs/K4fiYE7FeipFxOSOLSN94uVZE fCvQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=lyGUcSjrDXDJExumEBgUyuLlPJ8eSWbc/cGuq6Pshvw=; b=L9Oo8NMc0Wl1VmsG1aNNKb06TADGgs7lgZMuDnHVi8rjnxZMMXBYKNP37FQT7bLpIX CSxh2yCHq83QjGI6x0clFD8bZld62iajD5Z0CF1UD156RiiXk1iDdgClRiBty86ly7t/ c0RZaoz9hAuZS3AWUkk29xdG4wY+MQkELzCgfDM3VtVfyhZG9vMl97/Q6qbX14Xtoakb 0Cxx0fEXk3htc5CDTgiJGXHnW2ZRpJDaaO1TGX511p/bZ5Fb+4UTujFr64y4CyCTbmfL YGXUFnD7RSEUJTcpAypzPUzOBjBOvPIF3AUjAiBsRBnUdEtzcl5/x15hus8Sfz8dcq0E 6JJA== X-Gm-Message-State: AFeK/H2sEDqnONRu3wuAkMNQ6WX6kp+CUlmlTsrZ9nBDhchtbDXh6bt+gA6irHnACI2olKuWpCU2oaD0NmPA8A== X-Received: by 10.237.53.124 with SMTP id b57mr29284427qte.113.1490717946851; Tue, 28 Mar 2017 09:19:06 -0700 (PDT) MIME-Version: 1.0 Received: by 10.55.164.72 with HTTP; Tue, 28 Mar 2017 09:19:06 -0700 (PDT) In-Reply-To: References: From: Premal Shah Date: Tue, 28 Mar 2017 09:19:06 -0700 Message-ID: Subject: Re: LLAP queries create a yarn app per query To: user@hive.apache.org Content-Type: multipart/alternative; boundary=001a11c103c8aec811054bccd1bb archived-at: Tue, 28 Mar 2017 16:19:12 -0000 --001a11c103c8aec811054bccd1bb Content-Type: text/plain; charset=UTF-8 My bad. Looks like the thrift server is cycling through various AMs it started when the thrift server was started. I think this is different from either Hive 2.0.1 or LLAP. On Mon, Mar 27, 2017 at 11:38 PM, Premal Shah wrote: > Hi, > I have a thrift server to run queries using LLAP. There is script > connecting to this thrift server and running queries back to back. I am > tailing the thrift server logs and looks like every query is being run by a > new application master, which is adding overhead. Is there a reason, the > same App Master is not to run multiple queries. This does not happen when > not using LLAP ie. the same AM runs multiple queries. Is there a setting > I'm missing or is this expected behavior. > > Hive 2.1.1 and Hadoop 2.7.2 > > > Also, the application ids are out of order as you can see from the bolded > text. > > > > Query ID = hadoop_20170328052835_09ba338e-cc63-4d6f-a5d0-9c8e43548d1a > Total jobs = 1 > Launching Job 1 out of 1 > > > Status: Running (Executing on YARN cluster with App id > *application_1490656022550_0893*) > > ------------------------------------------------------------ > ---------------------------------- > VERTICES MODE STATUS TOTAL COMPLETED RUNNING > PENDING FAILED KILLED > ------------------------------------------------------------ > ---------------------------------- > Map 1 .......... llap SUCCEEDED 26 26 0 > 0 0 0 > Reducer 2 ...... llap SUCCEEDED 1 1 0 > 0 0 0 > ------------------------------------------------------------ > ---------------------------------- > VERTICES: 02/02 [==========================>>] 100% ELAPSED TIME: 8.76 > s > ------------------------------------------------------------ > ---------------------------------- > OK > Query ID = hadoop_20170328053107_b769a605-08ae-48d1-baa8-39096fdad17d > Total jobs = 1 > Launching Job 1 out of 1 > > Status: Running (Executing on YARN cluster with App id > *application_1490656022550_0863*) > > ------------------------------------------------------------ > ---------------------------------- > VERTICES MODE STATUS TOTAL COMPLETED RUNNING > PENDING FAILED KILLED > ------------------------------------------------------------ > ---------------------------------- > Map 1 .......... llap SUCCEEDED 26 26 0 > 0 0 0 > Reducer 2 ...... llap SUCCEEDED 1 1 0 > 0 0 0 > ------------------------------------------------------------ > ---------------------------------- > VERTICES: 02/02 [==========================>>] 100% ELAPSED TIME: 33.33 > s > ------------------------------------------------------------ > ---------------------------------- > OK > Query ID = hadoop_20170328053142_a23de218-2812-41ef-8419-02aaaf2b9dab > Total jobs = 1 > Launching Job 1 out of 1 > > > Status: Running (Executing on YARN cluster with App id > *application_1490656022550_0865*) > > ------------------------------------------------------------ > ---------------------------------- > VERTICES MODE STATUS TOTAL COMPLETED RUNNING > PENDING FAILED KILLED > ------------------------------------------------------------ > ---------------------------------- > Map 1 .......... llap SUCCEEDED 26 26 0 > 0 0 0 > Reducer 2 ...... llap SUCCEEDED 1 1 0 > 0 0 0 > ------------------------------------------------------------ > ---------------------------------- > VERTICES: 02/02 [==========================>>] 100% ELAPSED TIME: 8.49 > s > ------------------------------------------------------------ > ---------------------------------- > OK > Query ID = hadoop_20170328053153_8677d9d6-e748-4eb7-bfeb-1f1abdbb367c > Total jobs = 1 > Launching Job 1 out of 1 > > > -- > Regards, > Premal Shah. > -- Regards, Premal Shah. --001a11c103c8aec811054bccd1bb Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
My bad. Looks like the thrift server is cycling through va= rious AMs it started when the thrift server was started. I think this is di= fferent from either Hive 2.0.1 or LLAP.=C2=A0

On Mon, Mar 27, 2017 at 11:38 PM, Premal = Shah <premal.j.shah@gmail.com> wrote:
Hi,
I have a thrift server to run qu= eries using LLAP. There is script connecting to this thrift server and runn= ing queries back to back. I am tailing the thrift server logs and looks lik= e every query is being run by a new application master, which is adding ove= rhead. Is there a reason, the same App Master is not to run multiple querie= s. This does not happen when not using LLAP ie. the same AM runs multiple q= ueries. Is there a setting I'm missing or is this expected behavior.

Hive 2.1.1 and Hadoop 2.7.2


Also, the application ids are out of order as you can see f= rom the bolded text.



Query ID =3D hadoop_20170328052835_09ba338e-cc63-4d6f-a5= d0-9c8e43548d1a
Total jobs =3D 1
Launching Job 1 o= ut of 1


Status: Running (Executing = on YARN cluster with App id application_1490656022550_0893)

------------------------------------------------= ----------------------------------------------
=C2=A0 = =C2=A0 =C2=A0 =C2=A0 VERTICES =C2=A0 =C2=A0 =C2=A0MODE =C2=A0 =C2=A0 =C2=A0= =C2=A0STATUS =C2=A0TOTAL =C2=A0COMPLETED =C2=A0RUNNING =C2=A0PENDING =C2= =A0FAILED =C2=A0KILLED =C2=A0
------------------------------= ----------------------------------------------------------------<= /div>
Map 1 .......... =C2=A0 =C2=A0 =C2=A0llap =C2=A0 =C2=A0 SUCCEEDED= =C2=A0 =C2=A0 26 =C2=A0 =C2=A0 =C2=A0 =C2=A0 26 =C2=A0 =C2=A0 =C2=A0 =C2= =A00 =C2=A0 =C2=A0 =C2=A0 =C2=A00 =C2=A0 =C2=A0 =C2=A0 0 =C2=A0 =C2=A0 =C2= =A0 0
Reducer 2 ...... =C2=A0 =C2=A0 =C2=A0llap =C2=A0 =C2=A0 SUC= CEEDED =C2=A0 =C2=A0 =C2=A01 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A01 =C2=A0 =C2= =A0 =C2=A0 =C2=A00 =C2=A0 =C2=A0 =C2=A0 =C2=A00 =C2=A0 =C2=A0 =C2=A0 0 =C2= =A0 =C2=A0 =C2=A0 0
----------------------------------------= ------------------------------------------------------
= VERTICES: 02/02 =C2=A0[=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D>>] 100% =C2=A0ELAPSED TIME: 8.76 s = =C2=A0 =C2=A0=C2=A0
----------------------------------------= ------------------------------------------------------
= OK
Query ID =3D hadoop_20170328053107_b769a605-08ae-48d1= -baa8-39096fdad17d
Total jobs =3D 1
Launching Job = 1 out of 1

Status: Running (Executing on YARN clus= ter with App id application_1490656022550_0863)

<= /div>
------------------------------------------------------------= ----------------------------------
=C2=A0 =C2=A0 =C2=A0= =C2=A0 VERTICES =C2=A0 =C2=A0 =C2=A0MODE =C2=A0 =C2=A0 =C2=A0 =C2=A0STATUS= =C2=A0TOTAL =C2=A0COMPLETED =C2=A0RUNNING =C2=A0PENDING =C2=A0FAILED =C2= =A0KILLED =C2=A0
-------------------------------------------= ---------------------------------------------------
Map= 1 .......... =C2=A0 =C2=A0 =C2=A0llap =C2=A0 =C2=A0 SUCCEEDED =C2=A0 =C2= =A0 26 =C2=A0 =C2=A0 =C2=A0 =C2=A0 26 =C2=A0 =C2=A0 =C2=A0 =C2=A00 =C2=A0 = =C2=A0 =C2=A0 =C2=A00 =C2=A0 =C2=A0 =C2=A0 0 =C2=A0 =C2=A0 =C2=A0 0 =C2=A0<= /div>
Reducer 2 ...... =C2=A0 =C2=A0 =C2=A0llap =C2=A0 =C2=A0 SUCCEEDED= =C2=A0 =C2=A0 =C2=A01 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A01 =C2=A0 =C2=A0 = =C2=A0 =C2=A00 =C2=A0 =C2=A0 =C2=A0 =C2=A00 =C2=A0 =C2=A0 =C2=A0 0 =C2=A0 = =C2=A0 =C2=A0 0 =C2=A0
-------------------------------------= ---------------------------------------------------------
VERTICES: 02/02 =C2=A0[=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D>>] 100% =C2=A0ELAPSED TIME: 33.33= s =C2=A0 =C2=A0
-------------------------------------------= ---------------------------------------------------
OK<= /div>
Query ID =3D hadoop_20170328053142_a23de218-2812-41ef-8419-<= wbr>02aaaf2b9dab
Total jobs =3D 1
Launching Job 1 out o= f 1


Status: Running (Executing on Y= ARN cluster with App id application_1490656022550_0865)

----------------------------------------------------= ------------------------------------------
=C2=A0 =C2= =A0 =C2=A0 =C2=A0 VERTICES =C2=A0 =C2=A0 =C2=A0MODE =C2=A0 =C2=A0 =C2=A0 = =C2=A0STATUS =C2=A0TOTAL =C2=A0COMPLETED =C2=A0RUNNING =C2=A0PENDING =C2=A0= FAILED =C2=A0KILLED =C2=A0
---------------------------------= -------------------------------------------------------------
Map 1 .......... =C2=A0 =C2=A0 =C2=A0llap =C2=A0 =C2=A0 SUCCEEDED = =C2=A0 =C2=A0 26 =C2=A0 =C2=A0 =C2=A0 =C2=A0 26 =C2=A0 =C2=A0 =C2=A0 =C2=A0= 0 =C2=A0 =C2=A0 =C2=A0 =C2=A00 =C2=A0 =C2=A0 =C2=A0 0 =C2=A0 =C2=A0 =C2=A0 = 0 =C2=A0
Reducer 2 ...... =C2=A0 =C2=A0 =C2=A0llap =C2=A0 =C2=A0 = SUCCEEDED =C2=A0 =C2=A0 =C2=A01 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A01 =C2=A0 = =C2=A0 =C2=A0 =C2=A00 =C2=A0 =C2=A0 =C2=A0 =C2=A00 =C2=A0 =C2=A0 =C2=A0 0 = =C2=A0 =C2=A0 =C2=A0 0 =C2=A0
------------------------------= ----------------------------------------------------------------<= /div>
VERTICES: 02/02 =C2=A0[=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D>>] 100% =C2=A0ELAPSED TIME:= 8.49 s =C2=A0 =C2=A0=C2=A0
--------------------------------= --------------------------------------------------------------
OK
Query ID =3D hadoop_20170328053153_8677d9d6-e748-= 4eb7-bfeb-1f1abdbb367c
Total jobs =3D 1
Launching = Job 1 out of 1


--
Regards,
Premal Shah.



--
Regards,
Premal = Shah.
--001a11c103c8aec811054bccd1bb--