impala-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From tarmstr...@apache.org
Subject [1/3] impala git commit: IMPALA-6318: Adjustment for hanging query cancellation test
Date Thu, 04 Jan 2018 06:48:10 GMT
Repository: impala
Updated Branches:
  refs/heads/master 6f2ebadf8 -> 4f4912c53


IMPALA-6318: Adjustment for hanging query cancellation test

Apparently test_query_cancellation_during_fetch hangs occasionally
in Jenkins builds. The Impala debug page shows the query being
cancelled, however, on the host the ImpalaShell process related to
that query is still running.

Since I had no luck in reproducing the issue locally I only have a
theory what might be going on here: The query is cancelled
successfully on Impala backend and when the test tries to get the
stdout and stderr from the ImpalaShell it gets stuck. It might be
the case that ImpalaShell process fetching the query results holds
the stdout. According to the documentation of subprocess.communicate()
it may cause issues to fetch data when the data size is large or
unlimited, that we can consider to be the case here.
As a workaround there is a new optional parameter to
util.ImpalaShell to omit the stdout because this test wouldn't use
it anyway and we get rid of fetching the large result from
ImpalaShell.

Change-Id: I082c83b91b6d0c527de92c7992f0dc9d1b290433
Reviewed-on: http://gerrit.cloudera.org:8080/8852
Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com>
Tested-by: Impala Public Jenkins


Project: http://git-wip-us.apache.org/repos/asf/impala/repo
Commit: http://git-wip-us.apache.org/repos/asf/impala/commit/7810d1f9
Tree: http://git-wip-us.apache.org/repos/asf/impala/tree/7810d1f9
Diff: http://git-wip-us.apache.org/repos/asf/impala/diff/7810d1f9

Branch: refs/heads/master
Commit: 7810d1f9a2c7d59b4b916d4d1793672cd8c33143
Parents: 6f2ebad
Author: Gabor Kaszab <gaborkaszab@cloudera.com>
Authored: Fri Dec 15 09:19:08 2017 +0100
Committer: Impala Public Jenkins <impala-public-jenkins@gerrit.cloudera.org>
Committed: Wed Jan 3 20:32:24 2018 +0000

----------------------------------------------------------------------
 tests/shell/test_shell_commandline.py |  2 +-
 tests/shell/util.py                   | 20 ++++++++++++--------
 2 files changed, 13 insertions(+), 9 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/impala/blob/7810d1f9/tests/shell/test_shell_commandline.py
----------------------------------------------------------------------
diff --git a/tests/shell/test_shell_commandline.py b/tests/shell/test_shell_commandline.py
index 9bd6b8c..e4caa99 100644
--- a/tests/shell/test_shell_commandline.py
+++ b/tests/shell/test_shell_commandline.py
@@ -373,7 +373,7 @@ class TestImpalaShell(ImpalaTestSuite):
     execution in fact starts and then cancels it. Expects the query
     cancellation to succeed."""
     args = "-q \"" + stmt + ";\""
-    p = ImpalaShell(args)
+    p = ImpalaShell(args, omit_stdout=True)
 
     self.wait_for_query_state(stmt, cancel_at_state)
 

http://git-wip-us.apache.org/repos/asf/impala/blob/7810d1f9/tests/shell/util.py
----------------------------------------------------------------------
diff --git a/tests/shell/util.py b/tests/shell/util.py
index 22ffa1a..1fd7601 100755
--- a/tests/shell/util.py
+++ b/tests/shell/util.py
@@ -130,11 +130,14 @@ class ImpalaShellResult(object):
     self.stderr = str()
 
 class ImpalaShell(object):
-  """A single instance of the Impala shell. The proces is started when this object is
+  """A single instance of the Impala shell. The process is started when this object is
      constructed, and then users should repeatedly call send_cmd(), followed eventually by
      get_result() to retrieve the process output."""
-  def __init__(self, args=None, env=None):
-    self.shell_process = self._start_new_shell_process(args, env=env)
+  def __init__(self, args=None, env=None, omit_stdout=False):
+    self.args = args
+    self.env = env
+    self.omit_stdout = omit_stdout
+    self.shell_process = self._start_new_shell_process()
 
   def pid(self):
     return self.shell_process.pid
@@ -158,11 +161,12 @@ class ImpalaShell(object):
     result.rc = self.shell_process.returncode
     return result
 
-  def _start_new_shell_process(self, args=None, env=None):
+  def _start_new_shell_process(self):
     """Starts a shell process and returns the process handle"""
     shell_args = SHELL_CMD
-    if args is not None: shell_args = "%s %s" % (SHELL_CMD, args)
+    if self.args is not None: shell_args = "%s %s" % (SHELL_CMD, self.args)
     lex = shlex.split(shell_args)
-    if not env: env = os.environ
-    return Popen(lex, shell=False, stdout=PIPE, stdin=PIPE, stderr=PIPE,
-                 env=env)
+    if not self.env: self.env = os.environ
+    stdout = open(os.devnull, 'w') if self.omit_stdout else PIPE
+    return Popen(lex, shell=False, stdout=stdout, stdin=PIPE, stderr=PIPE,
+                 env=self.env)


Mime
View raw message