impala-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jin Chul Kim <jinc...@gmail.com>
Subject Re: A test failure at Jenkins build is not reproducible on my local
Date Thu, 16 Nov 2017 04:10:46 GMT
Hi Michael,

As I described in the previous email, I am using debug binary. I could
reproduce the issue using either impala-py.test or the last option which
restart impala cluster with enablement of stress_fn_ctx_alloc. My change
replaces rand_r in C library with std::mt19937 in C++ library, so
Allocate<std::mt19937>() is introduced. FYI, sizeof(std::mt19937) is 5000
bytes.

By the way, I still have a curiosity. The command below with "run-tests.py"
looks fine to me because it shows tests are finished successfully. I guess
it shows false positive error. If "tests/custom_cluster/test_alloc_fail.py"
cannot run with "run-tests.py", test should be finished with an error due
to incompatibility case. Would you please clarify two things?
1. The false positive error
2. Does "tests/custom_cluster/test_alloc_fail.py" run with "run-tests.py"?

Regarding guidance of E2E tests in "How to load and run Impala tests
<https://cwiki.apache.org/confluence/display/IMPALA/How+to+load+and+run+Impala+tests>"
wiki page, I wish there are more detailed information. Let me put the
information on the wiki after you answer my question: What is the
difference between "./test/run-tests.py" and "impala-py.test"?

$ ./tests/run-tests.py tests/custom_cluster/test_alloc_fail.py
...
=============================================================================================================================================================================
test session starts
==============================================================================================================================================================================
platform linux2 -- Python 2.7.12, pytest-2.9.2, py-1.4.32, pluggy-0.3.1 --
/home/jinchulkim/Impala/bin/../infra/python/env/bin/python
cachedir: .cache
rootdir: /home/jinchulkim/Impala/tests, inifile: pytest.ini
plugins: xdist-1.15.0, random-0.2
collected 2 items

verifiers/test_verify_metrics.py::TestValidateMetrics::test_metrics_are_zero
PASSED
verifiers/test_verify_metrics.py::TestValidateMetrics::test_num_unused_buffers
PASSED
===========================================================================================================================================================================
2 passed in 0.11 seconds
===========================================================================================================================================================================

Best regards,
Jinchul

2017-11-16 11:23 GMT+09:00 Michael Ho <kwho@cloudera.com>:

> That's odd. Please double check you are running debug builds. Alternately,
> you can try:
>
> impala-py.test tests/custom_cluster/test_alloc_fail.py
>
> Another approach is to try doing what the test is trying to achieve
> manually:
>
> ./bin/start-impala-cluster.py --impalad_args="--stress_fn_ctx_alloc=1"
> ./bin/impala-shell.sh -q "select rand() from functional.alltypes"
>
> My suspicion is that your change modified some states used for rand() and
> it bloated from 4 bytes to 5000 bytes.
> The error string needs to be updated as a result. That said, did anyone ask
> about the change in memory usage during code review ?
>
> On Wed, Nov 15, 2017 at 6:05 PM, Jin Chul Kim <jinchul@gmail.com> wrote:
>
> > Hi,
> >
> > I am trying to look into the build error:
> > https://jenkins.impala.io/job/gerrit-verify-dryrun/1472/
> > (The relevant code change: https://gerrit.cloudera.org/#/c/8355/)
> >
> > There was a test failure at "TestAllocFail.test_alloc_fail_init". I ran
> > the
> > following command but it always passed on my change: ./tests/run-tests.py
> > tests/custom_cluster/test_alloc_fail.py
> >
> > I changed the options below on Impala-shell and then retry it again, but
> > the failure did not happen.
> > exec_option: {'batch_size': 0, 'num_nodes': 0,
> > 'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
> > 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0}
> >
> > I am using Ubuntu 16.04 TLS and debug binary. Is there any differences
> > between Jenkins env. and my env.?
> >
> > ] =========================== short test summary info
> > ============================
> > ] XFAIL
> > custom_cluster/test_alloc_fail.py::TestAllocFail::()::
> > test_alloc_fail_update[exec_option:
> > {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0,
> > 'disable_codegen': False, 'abort_on_error': 1,
> > 'exec_single_node_rows_threshold': 0} | table_format: text/none]
> > ]   IMPALA-2925: the execution is not deterministic so some tests
> sometimes
> > don't fail as expected
> > ] FAIL
> > custom_cluster/test_alloc_fail.py::TestAllocFail::()::
> > test_alloc_fail_init[exec_option:
> > {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0,
> > 'disable_codegen': False, 'abort_on_error': 1,
> > 'exec_single_node_rows_threshold': 0} | table_format: text/none]
> > ] =================================== FAILURES
> > ===================================
> > ]  TestAllocFail.test_alloc_fail_init[exec_option: {'batch_size': 0,
> > 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen':
> > False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} |
> > table_format: text/none]
> > ] custom_cluster/test_alloc_fail.py:34: in test_alloc_fail_init
> > ]     self.run_test_case('QueryTest/alloc-fail-init', vector)
> > ] common/impala_test_suite.py:398: in run_test_case
> > ]     self.__verify_exceptions(test_section['CATCH'], str(e), use_db)
> > ] common/impala_test_suite.py:277: in __verify_exceptions
> > ]     (expected_str, actual_str)
> > ] E   AssertionError: Unexpected exception string. Expected:
> > FunctionContext::Allocate() failed to allocate 4 bytes.
> > ] E   Not found in actual: ImpalaBeeswaxException: Query
> > aborted:FunctionContext::Allocate() failed to allocate 5000 bytes.
> >
> > Best regards,
> > Jinchul
> >
>
>
>
> --
> Thanks,
> Michael
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message