impala-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael Brown (Code Review)" <ger...@cloudera.org>
Subject [Impala-CR](cdh5-trunk) IMPALA-3864: qgen: reduce likelihood of create query() exceptions
Date Thu, 21 Jul 2016 23:47:23 GMT
Michael Brown has uploaded a new patch set (#4).

Change subject: IMPALA-3864: qgen: reduce likelihood of create_query() exceptions
......................................................................

IMPALA-3864: qgen: reduce likelihood of create_query() exceptions

1. Fix a bug in which the computation to produce the string for an
   exception was raising a TypeError. We fix the bug by changing how the
   string is built.

2. Fix a bug in which we tried to choose a relational function (defined
   as taking in more than one argument and returning a Boolean) and were
   looking for its weight in QueryProfile.weights.RELATIONAL_FUNCS, but
   the function wasn't defined in that dictionary. We fix the bug by
   defining weights for those functions.

3. Fix a bug in which QueryProfile.choose_func_signatures() was choosing
   a function without taking into account the set of functions in the
   signatures given to it. We fix the bug by pruning off the weights of
   functions that aren't included in provided signatures. We also add a
   note explaining how the weights defined are "best effort", since
   sometimes functions will be pruned.

4. Add Char signatures to LessThan, GreaterThan, LessThanOrEquals,
   GreaterThanOrEquals. Debugging #3 above brought this to my attention.

5. Make changes to aid in debugging or testing:

   a. Add funcs.Signature representation.
   b. Move the code in query_generator.__main__ to
      generate_queries_for_manual_inspection(); call it from __main__.
   c. Increase the number of fake columns when calling
      generate_queries_for_manual_inspection(), which is useful for
      testing.
   d. Rename a few variables. For some reason in the query_generator
      module there are a lot of overwritten variables, which makes
      debugging difficult.

Testing:

1. impala-python tests/comparison/query_generator.py produces far fewer
   exceptions. The ones for this bug are fixed, but see IMPALA-3890.

2. Full 3 and 6 hour runs of the query generator system don't show any
   obvious regressions in behavior.

Change-Id: Idd9434a92973176aefb99e11e039209cac3cea65
---
M tests/comparison/funcs.py
M tests/comparison/query_generator.py
M tests/comparison/query_profile.py
3 files changed, 55 insertions(+), 20 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala refs/changes/20/3720/4
-- 
To view, visit http://gerrit.cloudera.org:8080/3720
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Idd9434a92973176aefb99e11e039209cac3cea65
Gerrit-PatchSet: 4
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Michael Brown <mikeb@cloudera.com>
Gerrit-Reviewer: David Knupp <dknupp@cloudera.com>
Gerrit-Reviewer: Michael Brown <mikeb@cloudera.com>
Gerrit-Reviewer: Taras Bobrovytsky <tbobrovytsky@cloudera.com>

Mime
View raw message