Mailing-List: contact user-help@hive.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hive.apache.org
MIME-Version: 1.0
Date: Thu, 30 Jul 2015 12:27:47 -0500
Message-ID: 
 <CAMP0dUEVdwXvZ5oK0n5a4AhuFGH9SN5+sny3k6RGN7fk-Yyouw@mail.gmail.com>
Subject: Semantic Analysis Run Through
From: Raajay <raajay.v@gmail.com>
To: user@hive.apache.org
Content-Type: multipart/alternative; boundary=047d7b10c889a17ed8051c1b056a

--047d7b10c889a17ed8051c1b056a
Content-Type: text/plain; charset=UTF-8

Hello,

I am currently playing around with Hive Semantic Analysis code, to
understand how DAGs or Map Reduce plans are generated from Abstract Syntax
Trees. The idea is to explore various possible DAGs and compare their
performance based on execution run time.

The function "analyzeInternal" seems to be handling the entire the plan
generation process. The different steps (at a high level) as described in
the comment section are:

1. Get Resolved Parse Tree from Syntax Tree

2. Get OP tree (Operator tree?) from Resolved parse tree

3. Deduce Result Set schema

4. Generate Parse Context

5. Do View creation

6. Collect Table Access stats

7. Perform Logical Optimization

8. Get Column Access Stats

9. Optimize Physical OP tree.

10. Translate to target execution engine.


I understand that step 7 (Logical Optimization) applies multiple transforms
( e.g. Join Reordering, Constant Propagation, Predicate pushdown) to alter
the AST and thus, different DAGs can be obtained by choosing whether to
apply or not apply certain transformations.

Can changes to the code in Steps 1-2 and 9 also possibly affect the
resulting DAGs ? How does the AST get affected in these steps ? Any
pointers / explanations will be helpful.

Thanks,
Raajay

--047d7b10c889a17ed8051c1b056a
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Hello,<div><br></div><div>I am currently playing around wi=
th Hive Semantic Analysis code, to understand how DAGs or Map Reduce plans =
are generated from Abstract Syntax Trees. The idea is to explore various po=
ssible DAGs and compare their performance based on execution run time.</div=
><div><br></div><div>The function &quot;analyzeInternal&quot; seems to be h=
andling the entire the plan generation process. The different steps (at a h=
igh level) as described in the comment section are:</div><div><br></div><di=
v>1. Get Resolved Parse Tree from Syntax Tree</div><div><br></div><div>2. G=
et OP tree (Operator tree?) from Resolved parse tree</div><div><br></div><d=
iv>3. Deduce Result Set schema</div><div><br></div><div>4. Generate Parse C=
ontext</div><div><br></div><div>5. Do View creation</div><div><br></div><di=
v>6. Collect Table Access stats</div><div><br></div><div>7. Perform Logical=
 Optimization</div><div><br></div><div>8. Get Column Access Stats</div><div=
><br></div><div>9. Optimize Physical OP tree.</div><div><br></div><div>10. =
Translate to target execution engine.</div><div><br></div><div><br></div><d=
iv>I understand that step 7 (Logical Optimization) applies multiple transfo=
rms ( e.g. Join Reordering, Constant Propagation, Predicate pushdown) to al=
ter the AST and thus, different DAGs can be obtained by choosing whether to=
 apply or not apply certain transformations.</div><div><br></div><div>Can c=
hanges to the code in Steps 1-2 and 9 also possibly affect the resulting DA=
Gs ? How does the AST get affected in these steps ? Any pointers / explanat=
ions will be helpful.</div><div><br></div><div>Thanks,</div><div>Raajay</di=
v></div>

--047d7b10c889a17ed8051c1b056a--