Mailing-List: contact user-help@hive.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hive.apache.org
MIME-Version: 1.0
Date: Fri, 11 Sep 2015 17:41:17 -0700
Message-ID: 
 <CAK18kC-Nv_-QVKbZRoxPJQO3FFTbS=zABWbBW4t1VraHgvc03A@mail.gmail.com>
Subject: Checking the number of Readers
From: James Pirz <james.pirz@gmail.com>
To: user@hive.apache.org
Content-Type: multipart/alternative; boundary=089e013cc168235137051f821732

--089e013cc168235137051f821732
Content-Type: text/plain; charset=UTF-8

I am using Hive 1.2.0 on Hadoop 2.6 (on a cluster with 10 machines) and I
am trying to understand the performance of a full-table scan. I am running
the following query:

SELECT * FROM LINEITEM
WHERE L_LINENUMBER < 0;

and I am measuring its performance in different scenarios: using "MR vs.
Tez" and  with different table types/formats (an external table on text
data, or ORC).

My question is:
What is the best way to check the number of readers (scanners) that Hive
uses in parallel to read the data ?

My data is in HDFS and on each node I have 1 datanode process running which
writes its blocks into 3 separate paths (each path persists its data on a
separate disk).

I tried to get this info using "explain" or from the available consoles,
but I could not find that. Checking the number of established connections
to the data transfer port for datanode (using the command below) gives me
12, but I am not sure If I am looking at the correct metric:

netstat -anp | grep -w 50010 | grep ESTABLISHED | wc -l


Any help would be appreciated.

Thnx

--089e013cc168235137051f821732
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">I am using Hive 1.2.0 on Hadoop 2.6 (on a cluster with 10 =
machines) and I am trying to understand the performance of a full-table sca=
n. I am running the following query:<div><br></div><div>SELECT * FROM LINEI=
TEM=C2=A0</div><div>WHERE L_LINENUMBER &lt; 0;</div><div><br></div><div>and=
 I am measuring its performance in different scenarios: using &quot;MR vs. =
Tez&quot; and =C2=A0with different table types/formats (an external table o=
n text data, or ORC).</div><div><br></div><div>My question is:</div><div>Wh=
at is the best way to check the number of readers (scanners) that Hive uses=
 in parallel to read the data ?=C2=A0</div><div><br></div><div>My data is i=
n HDFS and on each node I have 1 datanode process running which writes its =
blocks into 3 separate paths (each path persists its data on a separate dis=
k).</div><div><br></div><div>I tried to get this info using &quot;explain&q=
uot; or from the available consoles, but I could not find that. Checking th=
e number of established connections to the data transfer port for datanode =
(using the command below) gives me 12, but I am not sure If I am looking at=
 the correct metric:<br><div><br></div><div>netstat -anp | grep -w 50010 | =
grep ESTABLISHED | wc -l<br></div></div><div><br></div><div><br></div><div>=
Any help would be appreciated.</div><div><br></div><div>Thnx</div></div>

--089e013cc168235137051f821732--