Return-Path: X-Original-To: apmail-hive-user-archive@www.apache.org Delivered-To: apmail-hive-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 5086918DE6 for ; Tue, 25 Aug 2015 22:34:58 +0000 (UTC) Received: (qmail 10024 invoked by uid 500); 25 Aug 2015 22:34:56 -0000 Delivered-To: apmail-hive-user-archive@hive.apache.org Received: (qmail 9954 invoked by uid 500); 25 Aug 2015 22:34:56 -0000 Mailing-List: contact user-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hive.apache.org Delivered-To: mailing list user@hive.apache.org Received: (qmail 9944 invoked by uid 99); 25 Aug 2015 22:34:56 -0000 Received: from Unknown (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 Aug 2015 22:34:56 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id F2CA3182348 for ; Tue, 25 Aug 2015 22:34:55 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.899 X-Spam-Level: ** X-Spam-Status: No, score=2.899 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=3, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-eu-west.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id mRf5oWRAGpcb for ; Tue, 25 Aug 2015 22:34:49 +0000 (UTC) Received: from mail-ig0-f172.google.com (mail-ig0-f172.google.com [209.85.213.172]) by mx1-eu-west.apache.org (ASF Mail Server at mx1-eu-west.apache.org) with ESMTPS id 5E279212C5 for ; Tue, 25 Aug 2015 22:34:48 +0000 (UTC) Received: by igui7 with SMTP id i7so23016183igu.0 for ; Tue, 25 Aug 2015 15:34:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=gQUo/QhEYHxVxvGhmClOSruviBnM0/zX48RrsW4l60g=; b=uGqRPD4ahIvT9+UkdaY2W3l06syjde6h5bWq2nv2ExZuLfc4vIcJtiNpo71xMOnKS9 nuPRVBEKmB8uRfOE4yXfL0kw8goH6dcB1jX7YVGGLnylL1Ut9ejECUCxtSkPOFOVZA8T dr3TV+hEWtYcwuBw9bc4ID1g9Tdnit95i4Sp7CzEAZue8tcVLlG8LLAsddNDu4RWsqCn VmRen2kOJIDS/fn/O7gZn8P9rwMvz5XcfJzPOkmptmnfCLL/BulV29QH2aaN9+UGv1is 3SjqRLF0HbxTwHDrfKrweMoUqEuuOJ86W3hRoQtTELbb/X0dDRHvJEwjhmcjr+gJU5pZ Wafw== MIME-Version: 1.0 X-Received: by 10.50.79.167 with SMTP id k7mr6307756igx.67.1440542087345; Tue, 25 Aug 2015 15:34:47 -0700 (PDT) Received: by 10.107.33.211 with HTTP; Tue, 25 Aug 2015 15:34:47 -0700 (PDT) In-Reply-To: References: Date: Tue, 25 Aug 2015 17:34:47 -0500 Message-ID: Subject: Re: Run multiple queries simultaneously From: Raajay To: user@hive.apache.org Content-Type: multipart/alternative; boundary=089e013a1a38667ec4051e2a57a1 --089e013a1a38667ec4051e2a57a1 Content-Type: text/plain; charset=UTF-8 The back-end execution engine is Tez, and I use YARN for resource management. I completely agree with your deduction that the impact on the run time will be dependent on the nature of the queries. I would like to conduct some experiments (for a given workload, cluster configuration) to quantify the impact. For this, I need to be able to run queries simultaneously and measure the running times. What I glean from other threads is that, it should be good enough to fire up 2 CLI's and issue the queries. Raajay On Tue, Aug 25, 2015 at 4:17 PM, Ryan Harris wrote: > You need to be a bit more clear with your environment and objective > here.... > > What is your back-end execution engine? MapReduce, Spark, or Tez? > > What are you using for resource management? YARN or MapReduce? > > > > The running time of one query in the presence of other queries will > entirely depend on the cost/complexity of the queries. If each query is > able to fully utilize your allocated resources on the cluster then they > will be slower when run at the same time. However, many times in hive, > depending on the query and depending on the cluster resources, a single > hive query will only utilize a fraction of the cluster resources, in this > case multiple queries could be run at the same time with no detrimental > impact to performance as long as those queries aren't updating the same > hive table. > > > > *From:* Raajay [mailto:raajay.v@gmail.com] > *Sent:* Tuesday, August 25, 2015 7:21 AM > *To:* user@hive.apache.org > *Subject:* Run multiple queries simultaneously > > > > Hello, > > I want to compare the running time of an query when run alone against the > run time in presence of other queries. > > What is the ideal setup required to run this experiment ? Should I have > two Hive CLI's open and issue queries simultaneously ? How to script such > experiment in Hive ? > > Raajay > ------------------------------ > THIS ELECTRONIC MESSAGE, INCLUDING ANY ACCOMPANYING DOCUMENTS, IS > CONFIDENTIAL and may contain information that is privileged and exempt from > disclosure under applicable law. If you are neither the intended recipient > nor responsible for delivering the message to the intended recipient, > please note that any dissemination, distribution, copying or the taking of > any action in reliance upon the message is strictly prohibited. If you have > received this communication in error, please notify the sender immediately. > Thank you. > --089e013a1a38667ec4051e2a57a1 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
The back-end execution engine is Tez, and I= use YARN for resource management.

I completely agree with yo= ur deduction that the impact on the run time will be dependent on the natur= e of the queries. I would like to conduct some experiments (for a given wor= kload, cluster configuration) to quantify the impact.

For thi= s, I need to be able to run queries simultaneously and measure the running = times. What I glean from other threads is that, it should be good enough to= fire up 2 CLI's and issue the queries.

Raajay

On Tue, Aug 25, 201= 5 at 4:17 PM, Ryan Harris <Ryan.Harris@zionsbancorp.com>= wrote:

You need to be a bit more= clear with your environment and objective here....

What is your back-end exe= cution engine?=C2=A0 MapReduce, Spark, or Tez?

What are you using for re= source management? YARN or MapReduce?

=C2=A0

The running time of one q= uery in the presence of other queries will entirely depend on the cost/comp= lexity of the queries.=C2=A0 If each query is able to fully utilize your allocated resources on the cluster then they will be slower w= hen run at the same time.=C2=A0 However, many times in hive, depending on t= he query and depending on the cluster resources, a single hive query will o= nly utilize a fraction of the cluster resources, in this case multiple queries could be run at the same time wit= h no detrimental impact to performance as long as those queries aren't = updating the same hive table.

=C2=A0

From: Raajay [= mailto:raajay.v@gma= il.com]
Sent: Tuesday, August 25, 2015 7:21 AM
To: user@h= ive.apache.org
Subject: Run multiple queries simultaneously

=C2=A0

Hello,<= /p>

I want to compare the= running time of an query when run alone against the run time in presence o= f other queries.

What is the ideal set= up required to run this experiment ? Should I have two Hive CLI's open = and issue queries simultaneously ? How to script such experiment in Hive ?<= u>

Raajay

THIS ELECTRONIC MESSAGE, INCLUDING ANY ACCOMPANYING DOCUMENTS, IS CONFI= DENTIAL and may contain information that is privileged and exempt from disc= losure under applicable law. If you are neither the intended recipient nor = responsible for delivering the message to the intended recipient, please no= te that any dissemination, distribution, copying or the taking of any actio= n in reliance upon the message is strictly prohibited. If you have received= this communication in error, please notify the sender immediately. Thank = you.

--089e013a1a38667ec4051e2a57a1--