Mailing-List: contact dev-help@impala.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@impala.incubator.apache.org
MIME-Version: 1.0
In-Reply-To: <201604040815.u348FtsY001186@d01av02.pok.ibm.com>
References: 
 <OF0D14C953.CDC4C64E-ON65257F88.0047D5F8-65257F88.0047FD06@LocalDomain>
 <201604040815.u348FtsY001186@d01av02.pok.ibm.com>
From: Jim Apple <jbapple@cloudera.com>
Date: Tue, 5 Apr 2016 08:59:21 -0700
Message-ID: 
 <CAC-pSX1MBdpeD13CtC5AurRLT6KjnmNCdJ8npinDZYzfcM1+3w@mail.gmail.com>
Subject: Re: Fw: Debugging Impala code
To: dev@impala.incubator.apache.org
Cc: Silvius Rus <srus@cloudera.com>, Tim Armstrong <tarmstrong@cloudera.com>,
	Nishidha Panpaliya <nishidha@us.ibm.com>, David Clissold <cliss@us.ibm.com>,
 Manish Patil <patil2@us.ibm.com>
Content-Type: multipart/alternative; boundary=001a113473b60ae9db052fbef0c9

--001a113473b60ae9db052fbef0c9
Content-Type: text/plain; charset=UTF-8

Sudarshan, you have asked, "Please help on list of test issues which
Nishidha has mentioned."

In my personal opinion, it would help to tackle each issue separately, in a
separate thread or JIRA for each. How does that sound?

On Mon, Apr 4, 2016 at 1:15 AM, Sudarshan Jagadale <jagadale@us.ibm.com>
wrote:

> Dear Jim, Silvius and Team,
>
> Its be great to work closely with Impala community, we have been very
> satisfied with all help and collaborative working with you.
> Last Thursday we had call with Silvius and had very much useful call, in
> near future we will be having another call with David, IBM Power solution
> Arch will be joining.
>
> We have some blockers which Nishidha has mentioned in email, We are also
> working on them but we appreciate if could get some help out from dev team.
>
> We will be share another patch, for which we have legal approval.
>
> Please help on list of test issues which Nishidha has mentioned.
>
>
> Thanks and Regards
> Sudarshan Jagadale
> Power Open Source Solutions
>
> [image: Inactive hide details for Nishidha Panpaliya---04/01/2016 06:36:24
> PM---Sorry, attachment size was too big. So, here is the com]Nishidha
> Panpaliya---04/01/2016 06:36:24 PM---Sorry, attachment size was too big.
> So, here is the compressed log and backend test analysis before
>
> From: Nishidha Panpaliya/Austin/Contr/IBM
> To: "Tim Armstrong" <tarmstrong@cloudera.com>, "Jim Apple" <
> jbapple@cloudera.com>, dev@impala.incubator.apache.org
> Cc: "Silvius Rus" <srus@cloudera.com>, Sudarshan
> Jagadale/Austin/Contr/IBM@IBMUS
> Date: 04/01/2016 06:36 PM
> Subject: Fw: Debugging Impala code
> ------------------------------
>
>
> Sorry, attachment size was too big. So, here is the compressed log and
> backend test analysis before and after rebase.
> [attachment "LastTest.zip" deleted by Sudarshan Jagadale/Austin/Contr/IBM]
> [attachment "BackendTestAnalysis.xls" deleted by Sudarshan
> Jagadale/Austin/Contr/IBM]
>
> Thanks,
> Nishidha
>
> ----- Forwarded by Nishidha Panpaliya/Austin/Contr/IBM on 04/01/2016 06:34
> PM -----
>
> From: Nishidha Panpaliya/Austin/Contr/IBM
> To: dev@impala.incubator.apache.org
> Date: 04/01/2016 06:33 PM
> Subject: Fw: Debugging Impala code
> ------------------------------
>
>
> Resending again, as it failed to deliver first time.
>
> ----- Forwarded by Nishidha Panpaliya/Austin/Contr/IBM on 04/01/2016 06:32
> PM -----
>
> From: Nishidha Panpaliya/Austin/Contr/IBM
> To: Jim Apple <jbapple@cloudera.com>, Tim Armstrong <
> tarmstrong@cloudera.com>
> Cc: David Clissold/Austin/IBM@IBMUS, dev@impala.incubator.apache.org,
> Manish Patil/Austin/Contr/IBM@IBMUS, Sudarshan
> Jagadale/Austin/Contr/IBM@IBMUS, Valencia Serrao/Austin/Contr/IBM@IBMUS
> Date: 04/01/2016 04:28 PM
> Subject: Re: Fw: Debugging Impala code
> ------------------------------
>
>
> Hi,
>
> As you've suggested, we've rebased from cdh5-trunk and could compile the
> code on ppc64le with some modifications (ppc specific).
> I also gave a try to backend tests and got the many failures. At first,
> I'd got 41 errors but they all led to a common cause and after fixing it,
> count reduced to 9.
> Please find attached sheet that contains details of test execution before
> and after rebase with their analysis.
>
> [attachment "BackendTestAnalysis.xls" deleted by Nishidha
> Panpaliya/Austin/Contr/IBM]
>
> I've also attached the full log of tests that failed.
> [attachment "LastTest.log" deleted by Nishidha Panpaliya/Austin/Contr/IBM]
>
> We are investigating these failures, but in case, you've any pointers,
> we'll be grateful. Also, other than JIRA, is there any place where we can
> see the existing known issues like Jenkins's log? Can we get read-only
> access to those, just to check which failure is on x86 as well and its
> history like which commit introduced it, etc.?
>
> Thanks,
> Nishidha
>
>
> [image: Inactive hide details for Jim Apple ---03/25/2016 10:21:03
> PM---You can stay on cdh5-trunk. On Fri, Mar 25, 2016 at 9:43 AM, Ni]Jim
> Apple ---03/25/2016 10:21:03 PM---You can stay on cdh5-trunk. On Fri, Mar
> 25, 2016 at 9:43 AM, Nishidha Panpaliya <nishidha@us.ibm.com
>
> From: Jim Apple <jbapple@cloudera.com>
> To: Nishidha Panpaliya/Austin/Contr/IBM@IBMUS
> Cc: Tim Armstrong <tarmstrong@cloudera.com>, David
> Clissold/Austin/IBM@IBMUS, dev@impala.incubator.apache.org, Manish
> Patil/Austin/Contr/IBM@IBMUS, Sudarshan Jagadale/Austin/Contr/IBM@IBMUS
> Date: 03/25/2016 10:21 PM
> Subject: Re: Fw: Debugging Impala code
> ------------------------------
>
>
>
> You can stay on cdh5-trunk.
>
> On Fri, Mar 25, 2016 at 9:43 AM, Nishidha Panpaliya <*nishidha@us.ibm.com*
> <nishidha@us.ibm.com>> wrote:
>
>    Yes, I've started rebasing on cdh5-trunk. As expected, lot of changes
>    and conflicts! Is there any release tag or version to which should I update
>    from cdh5-trunk?
>
>    Will update you the build status after merge.
>
>    Thanks & Regards,
>    Nishidha
>
>    [image: Inactive hide details for Tim Armstrong ---03/24/2016 08:08:06
>    AM---If you haven't already, I'd suggest rebasing on cdh5-trunk]Tim
>    Armstrong ---03/24/2016 08:08:06 AM---If you haven't already, I'd suggest
>    rebasing on cdh5-trunk and taking a look at my patch here that f
>
>    From: Tim Armstrong <*tarmstrong@cloudera.com*
>    <tarmstrong@cloudera.com>>
>    To: Jim Apple <*jbapple@cloudera.com* <jbapple@cloudera.com>>
>    Cc: Nishidha Panpaliya/Austin/Contr/IBM@IBMUS, David
>    Clissold/Austin/IBM@IBMUS, *dev@impala.incubator.apache.org*
>    <dev@impala.incubator.apache.org>, Manish Patil/Austin/Contr/IBM@IBMUS,
>    Sudarshan Jagadale/Austin/Contr/IBM@IBMUS
>    Date: 03/24/2016 08:08 AM
>    Subject: Re: Fw: Debugging Impala code
>    ------------------------------
>
>
>
>    If you haven't already, I'd suggest rebasing on cdh5-trunk and taking
>    a look at my patch here that fixes those tests for later versions of LLVM
>    (on x86): *http://gerrit.cloudera.org/#/c/2486/*
>    <http://gerrit.cloudera.org/#/c/2486/> . There were a lot of subtle
>    issues so it will save you a lot of time.
>
>    - Tim
>
>    On Wed, Mar 23, 2016 at 9:45 AM, Jim Apple <*jbapple@cloudera.com*
>    <jbapple@cloudera.com>> wrote:
>       You might try looking for your broken tests in the bug tracker. For
>          instance, looking for expr-test leads to
>          *https://issues.cloudera.org/browse/IMPALA-2995*
>          <https://issues.cloudera.org/browse/IMPALA-2995>.
>
>          On Wed, Mar 23, 2016 at 2:33 AM, Nishidha Panpaliya <
>          *nishidha@us.ibm.com* <nishidha@us.ibm.com>> wrote:
>          Hi Tim and Jim,
>
>          Once again I thank you for your quick help.
>
>          I ran run-backend-tests.sh and here is the result of all tests -
>             89% tests passed, 8 tests failed out of 71
>
>                      Total Test time (real) = 979.08 sec
>
>                      The following tests FAILED:
>                      1 - llvm-codegen-test (SEGFAULT)
>                      13 - expr-test (OTHER_FAULT)
>                      14 - expr-codegen-test (OTHER_FAULT)
>                      19 - data-stream-test (Failed)
>                      22 - buffered-block-mgr-test (Failed)
>                      32 - tmp-file-mgr-test (Failed)
>                      33 - row-batch-serialize-test (SEGFAULT)
>                      68 - filesystem-util-test (Failed)
>
>          PFA full log.
> * (See attached file: LastTest.log)*
>
>          I also investigated some of the failures like
>             - *In filesystem-util-test* : Cause of this failure is user
>                      running the tests being "root". I could not run these tests with non-root
>                      user but I tested a sample test application that does exactly same thing as
>                      done in Createdirectory test of filesystem-util-test and it passed with
>                      non-root user. Also verified this by linux commands mkdir, chmod, rmdir
>                      with root and non-root user.
>                      - *In tmp-file-mgr-test *: Even this failure looks
>                      same as it also does chmod and the tries allocating space. Since I'm
>                      running these tests using root user, I would not get failure in accessing
>                      the dir/file even after removing write permissions if the user is root.
>                      - *In* *llvm-codegen-test: *I tried debugging this
>                      test and found a crash in llvm::Type::getVoidTy(...).
>                   I'll keep on investigating other crashes and
>          segmentation faults. But in case, you find any of the failures familiar or
>          existing on x86 platforms, please let us know.
>
>          Another news is we have got approval to share our patches. Soon,
>          I'll be uploading a patch with LLVM up-gradation work.
>
>          Thanks,
>          Nishidha
>
>
>          Thanks,
>          Nishidha
>          Tim Armstrong ---03/21/2016 09:10:46 PM---We generally run the
>          full test suite on machines with at least 32GB of memory: it's pretty
>          memory hu
>
>          From: Tim Armstrong <*tarmstrong@cloudera.com*
>          <tarmstrong@cloudera.com>>
>          To: Jim Apple <*jbapple@cloudera.com* <jbapple@cloudera.com>>
>          Cc: *dev@impala.incubator.apache.org*
>          <dev@impala.incubator.apache.org>, Sudarshan
>          Jagadale/Austin/Contr/IBM@IBMUS, Manish
>          Patil/Austin/Contr/IBM@IBMUS, David Clissold/Austin/IBM@IBMUS,
>          Nishidha Panpaliya/Austin/Contr/IBM@IBMUS
>          Date: 03/21/2016 09:10 PM
>          Subject: Re: Fw: Debugging Impala code
>          ------------------------------
>
>
>
>          We generally run the full test suite on machines with at least
>          32GB of memory: it's pretty memory hungry because you have 3 Impalads
>          running side-by-side. I believe we tend to run the full data load on
>          machines with even more memory. You can start the test cluster with a
>          single impalad before running tests (./bin/start-impala-cluster -s1 &&
>          ./tests/run-tests.py). Some tests will fail since they assume 3 Impalads
>          but most should work ok.
>
>          Starting with the backend tests sounds like a good idea - they
>          do exercise some of the codegen and other architecture-dependent parts that
>          will likely be tricky.
>
>          - Tim
>
>          On Mon, Mar 21, 2016 at 5:09 AM, Jim Apple <
>          *jbapple@cloudera.com* <jbapple@cloudera.com>> wrote:
>             I think you should be able to run the backend tests without
>                      data loading:
>
>                      ./bin/run-backend-tests.sh
>                      # or
>                      ctest
>
>                      As in the frontend tests, you can specify which test
>                      you want to run:
>
>                      ctest --output-on-failure -R expr-test # also shows
>                      what broke, if anything
>
>                      To only build the backend test run:
>
>                      make be-test
>
>                      On Mon, Mar 21, 2016 at 4:12 AM, Nishidha Panpaliya <
>                      *nishidha@us.ibm.com* <nishidha@us.ibm.com>> wrote:
>                      Thanks Jim and Tim for your replies. Really
>                      appreciate your co-operation and promptness.
>
>                      I've a few more queries -
>
>                      1. What is the memory requirement of Impala to run
>                      all the tests? Currently, I see test data creation and loading is consuming
>                      almost 7GB of RAM. And after this, it gets stopped with bad_alloc
>                      exception. I've already requested to increase RAM of my VM. But just wanted
>                      to know if 16GB will suffice.
>
>                      2. Can we skip load testing at this stage and simply
>                      run basic unit tests at first? Or is there any setting by means of which we
>                      can lower the volume of test data being generated/loaded? Once basic tests
>                      are working, we can focus on load testing.
>
>                      Also, we wish to have a call with you to discuss all
>                      this. We are located in India.
>
>                      Thanks,
>                      Nishidha
>
>
>                      Sudarshan Jagadale---03/18/2016 11:04:45 AM---Thanks
>                      and Regards Sudarshan Jagadale
>
>                      From: Sudarshan Jagadale/Austin/Contr/IBM
>                      To: Nishidha Panpaliya/Austin/Contr/IBM@IBMUS
>                      Cc: *vserrao@us.ibm.com* <vserrao@us.ibm.com>,
>                      Manish Patil/Austin/Contr/IBM@IBMUS
>                      Date: 03/18/2016 11:04 AM
>                      Subject: Fw: Debugging Impala code
>                      ------------------------------
>
>
>
>                      Thanks and Regards
>                      Sudarshan Jagadale
>                      Power Open Source Solutions
>                      ----- Forwarded by Sudarshan
>                      Jagadale/Austin/Contr/IBM on 03/18/2016 11:04 AM -----
>
>                      From: Tim Armstrong <*tarmstrong@cloudera.com*
>                      <tarmstrong@cloudera.com>>
>                      To: *dev@impala.incubator.apache.org*
>                      <dev@impala.incubator.apache.org>
>                      Cc: Sudarshan Jagadale/Austin/Contr/IBM@IBMUS
>                      Date: 03/17/2016 10:39 PM
>                      Subject: Re: Debugging Impala code
>                      ------------------------------
>
>
>
>                      Was it the impalad process that crashed? If so,
>                      there are a few places you can check:
>                         - Look in /tmp/impalad.ERROR,
>                                                                      /tmp/impalad_node1.ERROR and /tmp/impalad_node2.ERROR for error messages.
>                                                                      If it hit an assertion, you will get the message in there.
>                                                                      - Look
>                                                                      in the equivalent INFO logs for other error messages (for some crashes,
>                                                                      there is info sent to INFO but not ERROR)
>                                                                      - Look
>                                                                      for hs_err_pid*.log files in the directory you ran Impala from. These are
>                                                                      crash reports from the embedded JVM in the impalad process
>                                                                      - Get
>                                                                      impala to produce a core dump (make sure you have ulimit -c unlimited set
>                                                                      when starting the cluster. I have it set in my .bashrc file) then debug
>                                                                      with gdb.
>
>
>                      On Thu, Mar 17, 2016 at 8:59 AM, Jim Apple <
>                      *jbapple@cloudera.com* <jbapple@cloudera.com>> wrote:
>                         I believe Hive is sometimes used for data
>                                              loading, though I'm not sure.
>
>                                              I haven't debugged impala
>                                              during data loading, but when I do need to debug
>                                              the backend, I often do
>
>                                              sudo gdb -p $(ps -C impalad
>                                              -o pid | tail -1 | awk '{print $1}')
>
>
>                                              On Thu, Mar 17, 2016 at 8:50
>                                              AM, Nishidha Panpaliya <
>                                              *nishidha@us.ibm.com*
>                                              <nishidha@us.ibm.com>>
>                                              wrote:
>
>                                              >
>                                              > Hi All,
>                                              >
>                                              > I'm able to build Impala
>                                              on Ubuntu ppc64le but getting crashes while
>                                              > loading test data.
>                                              >
>                                              > I wanted to know how do
>                                              you normally debug Impala code while loading test
>                                              > data before running unit
>                                              tests. Other than core dump, what are the other
>                                              > ways to find out causes of
>                                              crash in Impala?
>                                              >
>                                              >
>                                              > Thanks,
>                                              > Nishidha
>                                              >
>
>
>
>
>

--001a113473b60ae9db052fbef0c9--