impala-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael Brown (Code Review)" <ger...@cloudera.org>
Subject [Impala-CR](cdh5-trunk) IMPALA-3864: qgen: reduce likelihood of create query() exceptions
Date Fri, 22 Jul 2016 17:57:23 GMT
Michael Brown has submitted this change and it was merged.

Change subject: IMPALA-3864: qgen: reduce likelihood of create_query() exceptions
......................................................................


IMPALA-3864: qgen: reduce likelihood of create_query() exceptions

1. Fix a bug in which the computation to produce the string for an
   exception was raising a TypeError. We fix the bug by changing how the
   string is built.

2. Fix a bug in which we tried to choose a relational function (defined
   as taking in more than one argument and returning a Boolean) and were
   looking for its weight in QueryProfile.weights.RELATIONAL_FUNCS, but
   the function wasn't defined in that dictionary. We fix the bug by
   defining weights for those functions.

3. Fix a bug in which QueryProfile.choose_func_signatures() was choosing
   a function without taking into account the set of functions in the
   signatures given to it. We fix the bug by pruning off the weights of
   functions that aren't included in provided signatures. We also add a
   note explaining how the weights defined are "best effort", since
   sometimes functions will be pruned.

4. Add Char signatures to LessThan, GreaterThan, LessThanOrEquals,
   GreaterThanOrEquals. Debugging #3 above brought this to my attention.

5. Make changes to aid in debugging or testing:

   a. Add funcs.Signature representation.
   b. Move the code in query_generator.__main__ to
      generate_queries_for_manual_inspection(); call it from __main__.
   c. Increase the number of fake columns when calling
      generate_queries_for_manual_inspection(), which is useful for
      testing.
   d. Rename a few variables. For some reason in the query_generator
      module there are a lot of overwritten variables, which makes
      debugging difficult.

Testing:

1. impala-python tests/comparison/query_generator.py produces far fewer
   exceptions. The ones for this bug are fixed, but see IMPALA-3890.

2. Full 3 and 6 hour runs of the query generator system don't show any
   obvious regressions in behavior.

Change-Id: Idd9434a92973176aefb99e11e039209cac3cea65
Reviewed-on: http://gerrit.cloudera.org:8080/3720
Tested-by: Michael Brown <mikeb@cloudera.com>
Reviewed-by: Michael Brown <mikeb@cloudera.com>
---
M tests/comparison/funcs.py
M tests/comparison/query_generator.py
M tests/comparison/query_profile.py
3 files changed, 65 insertions(+), 20 deletions(-)

Approvals:
  Michael Brown: Looks good to me, approved; Verified



-- 
To view, visit http://gerrit.cloudera.org:8080/3720
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Idd9434a92973176aefb99e11e039209cac3cea65
Gerrit-PatchSet: 9
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Michael Brown <mikeb@cloudera.com>
Gerrit-Reviewer: David Knupp <dknupp@cloudera.com>
Gerrit-Reviewer: Michael Brown <mikeb@cloudera.com>
Gerrit-Reviewer: Taras Bobrovytsky <tbobrovytsky@cloudera.com>

Mime
View raw message