impala-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sudarshan Jagadale" <jagad...@us.ibm.com>
Subject Re: Fw: Debugging Impala code
Date Mon, 04 Apr 2016 08:15:46 GMT

Dear Jim, Silvius and Team,

Its be great to work closely with Impala community, we have been very
satisfied with all help and collaborative working with you.
Last Thursday we had call with Silvius and had very much useful call, in
near future we will be having another call with David, IBM Power solution
Arch will be joining.

We have some blockers which Nishidha has mentioned in email, We are also
working on them but  we appreciate if could get some help out from dev
team.

We will be share another patch, for which we have legal approval.

Please help on list of test issues which Nishidha has mentioned.


Thanks and Regards
Sudarshan Jagadale
Power Open Source Solutions



From:	Nishidha Panpaliya/Austin/Contr/IBM
To:	"Tim Armstrong" <tarmstrong@cloudera.com>, "Jim Apple"
            <jbapple@cloudera.com>, dev@impala.incubator.apache.org
Cc:	"Silvius Rus" <srus@cloudera.com>, Sudarshan
            Jagadale/Austin/Contr/IBM@IBMUS
Date:	04/01/2016 06:36 PM
Subject:	Fw: Debugging Impala code


Sorry, attachment size was too big. So, here is the compressed log and
backend test analysis before and after rebase.
[attachment "LastTest.zip" deleted by Sudarshan Jagadale/Austin/Contr/IBM]
[attachment "BackendTestAnalysis.xls" deleted by Sudarshan
Jagadale/Austin/Contr/IBM]

Thanks,
Nishidha

----- Forwarded by Nishidha Panpaliya/Austin/Contr/IBM on 04/01/2016 06:34
PM -----

From:	Nishidha Panpaliya/Austin/Contr/IBM
To:	dev@impala.incubator.apache.org
Date:	04/01/2016 06:33 PM
Subject:	Fw: Debugging Impala code


Resending again, as it failed to deliver first time.

----- Forwarded by Nishidha Panpaliya/Austin/Contr/IBM on 04/01/2016 06:32
PM -----

From:	Nishidha Panpaliya/Austin/Contr/IBM
To:	Jim Apple <jbapple@cloudera.com>, Tim Armstrong
            <tarmstrong@cloudera.com>
Cc:	David Clissold/Austin/IBM@IBMUS,
            dev@impala.incubator.apache.org, Manish
            Patil/Austin/Contr/IBM@IBMUS, Sudarshan
            Jagadale/Austin/Contr/IBM@IBMUS, Valencia
            Serrao/Austin/Contr/IBM@IBMUS
Date:	04/01/2016 04:28 PM
Subject:	Re: Fw: Debugging Impala code


Hi,

As you've suggested, we've rebased from cdh5-trunk and could compile the
code on ppc64le with some modifications (ppc specific).
I also gave a try to backend tests and got the many failures. At first, I'd
got 41 errors but they all led to a common cause and after fixing it, count
reduced to 9.
Please find attached sheet that contains details of test execution before
and after rebase with their analysis.

[attachment "BackendTestAnalysis.xls" deleted by Nishidha
Panpaliya/Austin/Contr/IBM]

I've also attached the full log of tests that failed.
[attachment "LastTest.log" deleted by Nishidha Panpaliya/Austin/Contr/IBM]

We are investigating these failures, but in case, you've any pointers,
we'll be grateful. Also, other than JIRA, is there any place where we can
see the existing known issues like Jenkins's log? Can we get read-only
access to those, just to check which failure is on x86 as well and its
history like which commit introduced it, etc.?

Thanks,
Nishidha




From:	Jim Apple <jbapple@cloudera.com>
To:	Nishidha Panpaliya/Austin/Contr/IBM@IBMUS
Cc:	Tim Armstrong <tarmstrong@cloudera.com>, David
            Clissold/Austin/IBM@IBMUS, dev@impala.incubator.apache.org,
            Manish Patil/Austin/Contr/IBM@IBMUS, Sudarshan
            Jagadale/Austin/Contr/IBM@IBMUS
Date:	03/25/2016 10:21 PM
Subject:	Re: Fw: Debugging Impala code



You can stay on cdh5-trunk.

On Fri, Mar 25, 2016 at 9:43 AM, Nishidha Panpaliya <nishidha@us.ibm.com>
wrote:
  Yes, I've started rebasing on cdh5-trunk. As expected, lot of changes and
  conflicts! Is there any release tag or version to which should I update
  from cdh5-trunk?

  Will update you the build status after merge.

  Thanks & Regards,
  Nishidha

  Inactive hide details for Tim Armstrong ---03/24/2016 08:08:06 AM---If
  you haven't already, I'd suggest rebasing on cdh5-trunk Tim Armstrong
  ---03/24/2016 08:08:06 AM---If you haven't already, I'd suggest rebasing
  on cdh5-trunk and taking a look at my patch here that f

  From: Tim Armstrong <tarmstrong@cloudera.com>
  To: Jim Apple <jbapple@cloudera.com>
  Cc: Nishidha Panpaliya/Austin/Contr/IBM@IBMUS, David
  Clissold/Austin/IBM@IBMUS, dev@impala.incubator.apache.org, Manish
  Patil/Austin/Contr/IBM@IBMUS, Sudarshan Jagadale/Austin/Contr/IBM@IBMUS
  Date: 03/24/2016 08:08 AM
  Subject: Re: Fw: Debugging Impala code



  If you haven't already, I'd suggest rebasing on cdh5-trunk and taking a
  look at my patch here that fixes those tests for later versions of LLVM
  (on x86): http://gerrit.cloudera.org/#/c/2486/ . There were a lot of
  subtle issues so it will save you a lot of time.

  - Tim

  On Wed, Mar 23, 2016 at 9:45 AM, Jim Apple <jbapple@cloudera.com> wrote:
        You might try looking for your broken tests in the bug tracker. For
        instance, looking for expr-test leads to
        https://issues.cloudera.org/browse/IMPALA-2995.

        On Wed, Mar 23, 2016 at 2:33 AM, Nishidha Panpaliya <
        nishidha@us.ibm.com> wrote:
        Hi Tim and Jim,

        Once again I thank you for your quick help.

        I ran run-backend-tests.sh and here is the result of all tests -
                    89% tests passed, 8 tests failed out of 71

                    Total Test time (real) = 979.08 sec

                    The following tests FAILED:
                    1 - llvm-codegen-test (SEGFAULT)
                    13 - expr-test (OTHER_FAULT)
                    14 - expr-codegen-test (OTHER_FAULT)
                    19 - data-stream-test (Failed)
                    22 - buffered-block-mgr-test (Failed)
                    32 - tmp-file-mgr-test (Failed)
                    33 - row-batch-serialize-test (SEGFAULT)
                    68 - filesystem-util-test (Failed)

        PFA full log.
        (See attached file: LastTest.log)

        I also investigated some of the failures like
                    - In filesystem-util-test : Cause of this failure is
                    user running the tests being "root". I could not run
                    these tests with non-root user but I tested a sample
                    test application that does exactly same thing as done
                    in Createdirectory test of filesystem-util-test and it
                    passed with non-root user. Also verified this by linux
                    commands mkdir, chmod, rmdir with root and non-root
                    user.
                    - In tmp-file-mgr-test : Even this failure looks same
                    as it also does chmod and the tries allocating space.
                    Since I'm running these tests using root user, I would
                    not get failure in accessing the dir/file even after
                    removing write permissions if the user is root.
                    - In llvm-codegen-test: I tried debugging this test and
                    found a crash in llvm::Type::getVoidTy(...).
        I'll keep on investigating other crashes and segmentation faults.
        But in case, you find any of the failures familiar or existing on
        x86 platforms, please let us know.

        Another news is we have got approval to share our patches. Soon,
        I'll be uploading a patch with LLVM up-gradation work.

        Thanks,
        Nishidha


        Thanks,
        Nishidha
        Tim Armstrong ---03/21/2016 09:10:46 PM---We generally run the full
        test suite on machines with at least 32GB of memory: it's pretty
        memory hu

        From: Tim Armstrong <tarmstrong@cloudera.com>
        To: Jim Apple <jbapple@cloudera.com>
        Cc: dev@impala.incubator.apache.org, Sudarshan
        Jagadale/Austin/Contr/IBM@IBMUS, Manish
        Patil/Austin/Contr/IBM@IBMUS, David Clissold/Austin/IBM@IBMUS,
        Nishidha Panpaliya/Austin/Contr/IBM@IBMUS
        Date: 03/21/2016 09:10 PM
        Subject: Re: Fw: Debugging Impala code



        We generally run the full test suite on machines with at least 32GB
        of memory: it's pretty memory hungry because you have 3 Impalads
        running side-by-side. I believe we tend to run the full data load
        on machines with even more memory. You can start the test cluster
        with a single impalad before running tests
        (./bin/start-impala-cluster -s1 && ./tests/run-tests.py). Some
        tests will fail since they assume 3 Impalads but most should work
        ok.

        Starting with the backend tests sounds like a good idea - they do
        exercise some of the codegen and other architecture-dependent parts
        that will likely be tricky.

        - Tim

        On Mon, Mar 21, 2016 at 5:09 AM, Jim Apple <jbapple@cloudera.com>
        wrote:
                    I think you should be able to run the backend tests
                    without data loading:

                    ./bin/run-backend-tests.sh
                    # or
                    ctest

                    As in the frontend tests, you can specify which test
                    you want to run:

                    ctest --output-on-failure -R expr-test # also shows
                    what broke, if anything

                    To only build the backend test run:

                    make be-test

                    On Mon, Mar 21, 2016 at 4:12 AM, Nishidha Panpaliya <
                    nishidha@us.ibm.com> wrote:
                    Thanks Jim and Tim for your replies. Really appreciate
                    your co-operation and promptness.

                    I've a few more queries -

                    1. What is the memory requirement of Impala to run all
                    the tests? Currently, I see test data creation and
                    loading is consuming almost 7GB of RAM. And after this,
                    it gets stopped with bad_alloc exception. I've already
                    requested to increase RAM of my VM. But just wanted to
                    know if 16GB will suffice.

                    2. Can we skip load testing at this stage and simply
                    run basic unit tests at first? Or is there any setting
                    by means of which we can lower the volume of test data
                    being generated/loaded? Once basic tests are working,
                    we can focus on load testing.

                    Also, we wish to have a call with you to discuss all
                    this. We are located in India.

                    Thanks,
                    Nishidha


                    Sudarshan Jagadale---03/18/2016 11:04:45 AM---Thanks
                    and Regards Sudarshan Jagadale

                    From: Sudarshan Jagadale/Austin/Contr/IBM
                    To: Nishidha Panpaliya/Austin/Contr/IBM@IBMUS
                    Cc: vserrao@us.ibm.com, Manish
                    Patil/Austin/Contr/IBM@IBMUS
                    Date: 03/18/2016 11:04 AM
                    Subject: Fw: Debugging Impala code



                    Thanks and Regards
                    Sudarshan Jagadale
                    Power Open Source Solutions
                    ----- Forwarded by Sudarshan Jagadale/Austin/Contr/IBM
                    on 03/18/2016 11:04 AM -----

                    From: Tim Armstrong <tarmstrong@cloudera.com>
                    To: dev@impala.incubator.apache.org
                    Cc: Sudarshan Jagadale/Austin/Contr/IBM@IBMUS
                    Date: 03/17/2016 10:39 PM
                    Subject: Re: Debugging Impala code



                    Was it the impalad process that crashed? If so, there
                    are a few places you can check:
                                                                    Look
                                                                    in /tmp/impalad.ERROR,
/tmp/impalad_node1.ERROR
 and /tmp/impalad_node2.ERROR for error messages. If it hit an assertion,
                                                                    you
                                                                    will
                                                                    get the
                                                                    message
                                                                    in
                                                                    there.
                                                                    Look in
                                                                    the
                                                                    equivalent
 INFO logs for other error messages (for some crashes, there is info sent
                                                                    to INFO
                                                                    but not
                                                                    ERROR)
                                                                    Look
                                                                    for
                                                                    hs_err_pid*.log
 files in the directory you ran Impala from. These are crash reports from
                                                                    the
                                                                    embedded
 JVM in the impalad process
                                                                    Get
                                                                    impala
                                                                    to
                                                                    produce
                                                                    a core
                                                                    dump
                                                                    (make
                                                                    sure
                                                                    you
                                                                    have
                                                                    ulimit
                                                                    -c
                                                                    unlimited
 set when starting the cluster. I have it set in my .bashrc file) then
                                                                    debug
                                                                    with
                                                                    gdb.


                    On Thu, Mar 17, 2016 at 8:59 AM, Jim Apple <
                    jbapple@cloudera.com> wrote:
                                            I believe Hive is sometimes
                                            used for data loading, though
                                            I'm not sure.

                                            I haven't debugged impala
                                            during data loading, but when I
                                            do need to debug
                                            the backend, I often do

                                            sudo gdb -p $(ps -C impalad -o
                                            pid | tail -1 | awk '{print
                                            $1}')


                                            On Thu, Mar 17, 2016 at 8:50
                                            AM, Nishidha Panpaliya <
                                            nishidha@us.ibm.com>
                                            wrote:

                                            >
                                            > Hi All,
                                            >
                                            > I'm able to build Impala on
                                            Ubuntu ppc64le but getting
                                            crashes while
                                            > loading test data.
                                            >
                                            > I wanted to know how do you
                                            normally debug Impala code
                                            while loading test
                                            > data before running unit
                                            tests. Other than core dump,
                                            what are the other
                                            > ways to find out causes of
                                            crash in Impala?
                                            >
                                            >
                                            > Thanks,
                                            > Nishidha
                                            >











Mime
  • Unnamed multipart/related (inline, None, 0 bytes)
View raw message