Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 33C74200B4C for ; Fri, 22 Jul 2016 20:45:12 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 324CD160A6D; Fri, 22 Jul 2016 18:45:12 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id D8501160A5A for ; Fri, 22 Jul 2016 20:45:10 +0200 (CEST) Received: (qmail 46196 invoked by uid 500); 22 Jul 2016 18:45:10 -0000 Mailing-List: contact dev-help@impala.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@impala.incubator.apache.org Delivered-To: mailing list dev@impala.incubator.apache.org Received: (qmail 46183 invoked by uid 99); 22 Jul 2016 18:45:09 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Jul 2016 18:45:09 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 5D8FF187166 for ; Fri, 22 Jul 2016 18:45:06 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.499 X-Spam-Level: ** X-Spam-Status: No, score=2.499 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=2, KAM_LINEPADDING=1.2, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001, TVD_FW_GRAPHIC_NAME_MID=0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=cloudera-com.20150623.gappssmtp.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id nN0byx4SYLjd for ; Fri, 22 Jul 2016 18:45:03 +0000 (UTC) Received: from mail-vk0-f47.google.com (mail-vk0-f47.google.com [209.85.213.47]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 4B76560E71 for ; Fri, 22 Jul 2016 18:45:03 +0000 (UTC) Received: by mail-vk0-f47.google.com with SMTP id w127so169110208vkh.2 for ; Fri, 22 Jul 2016 11:45:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cloudera-com.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=Vkmt5kdiEoujtvCdcJD31skmgrMU7FUetHdEyXb3AZs=; b=QA+GgO9UqCRRLEBZy4/LwmbD4w+wEvdik1AJaiOLYaa2TnsVdNHwrNJ06QVggmVG9s qjf/vFxvIjc9YZh9V+gUtl4eqsne5oVEn7SOOHlY1C56Bhz6JhROigye9/zGKo+Hl0YX MLTyv0zFV+2W4ySdzNexbkiA/jN7UiftjoDgfHZzC9pfe68xECAVV8n/FbUoGCoPSV5U cKONknm/YWaXKPwmxPFzE0wLGt/LUvsnzl9kqK7VT4Lh09Yg+pTlEN1Qd/2QZbIq1oC1 55dpafO54LX8fhPg2e5Ar4mJdsEVhFdgKR14y3+unXUlKI18sOymbSUodpUPy2x9TXkj MfWw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=Vkmt5kdiEoujtvCdcJD31skmgrMU7FUetHdEyXb3AZs=; b=Syzz9vXZ5yYcxw92DHopub506AaRVIvMMk0sGp5RpAfqXSH9kIykMap5VkPxDHe24o 3TZxwi8SloTSHLcQBa+Et1YR2Ly2beeezhL/o80ujr6DFfV6WG5ku3FF5+CZTm8bk6O1 Sfqo99skgU0tbqJf79fQ3N9IcfjQpG7V9gv8QtHEk0sHG1w+pPrTdk6cDid+cDXCcbUR b0WKWGyeKKGESjgj+JeB5JR+q5D9W6DQT/EPRdv6Kc0WNidYtPU+a/bjXI8pUX7wfZP6 HYPYxHGaE8fWuF/CWbsyTrpSRTj1qnmJO4LUTC4tMqFxdU+WPkXFTEe9wJrJJZj4VBhF i84w== X-Gm-Message-State: AEkoousHRjwyMcSLmTibZDnFbxjrXv4q+nqWc9JhAaieYvCzhK8aMkjC0lWIgALsAkgFWhlT85+qEZKXyVAHsyOW X-Received: by 10.31.199.135 with SMTP id x129mr2219622vkf.27.1469213101978; Fri, 22 Jul 2016 11:45:01 -0700 (PDT) MIME-Version: 1.0 Received: by 10.176.2.16 with HTTP; Fri, 22 Jul 2016 11:44:32 -0700 (PDT) In-Reply-To: References: From: Tim Armstrong Date: Fri, 22 Jul 2016 11:44:32 -0700 Message-ID: Subject: Re: Issues with tests in Release-mode Impala build To: Valencia Serrao Cc: dev@impala.incubator.apache.org, Manish Patil , Nishidha Panpaliya , Sudarshan Jagadale Content-Type: multipart/related; boundary=001a114847ae0b246a05383dd53f archived-at: Fri, 22 Jul 2016 18:45:12 -0000 --001a114847ae0b246a05383dd53f Content-Type: multipart/alternative; boundary=001a114847ae0b246705383dd53e --001a114847ae0b246705383dd53e Content-Type: text/plain; charset=UTF-8 2a. Exhaustive is a superset of core. We run the core tests pre-commit on CentOS 6 + HDFS and the full exhaustive tests post-commit on a wider range of configurations. We don't release Impala unless all exhaustive tests passed on all configurations we test (if there's a valid reason why something doesn't work on a given platform we skip the test). 2b. Exhaustive is a superset of core, so if exhaustive passes then core should do. The exhaustive build takes much longer than core so it makes sense to run it less frequently (e.g. we run it nightly for some configurations and weekly for others). 2c. Confusingly, the core/exhaustive data load doesn't map to core/exhaustive tests. We actually use the same data load for all test configurations. See testdata/bin/create-load-data.sh for how the core/exhaustive data load is invoked. E.g. we load the functional data with exhaustive (i.e. all supported file formats) and the larger tpc-h/tpc-ds data sets for only a subset of file forms. On Wed, Jul 20, 2016 at 9:39 PM, Valencia Serrao wrote: > Hi Tim, > > Thank you for the insight on the issues. > > 1. *BE test -issue: benchmark-test hangs* > As you suggested, I increased the "batch_size" value to upto 125000000, > however, the sw.ElapsedTime() does not increase inside the while and again > gets caught up in an infinite loop. The optimization level seems to cause > this behavior. I am still working in this. > > 2. *Custom cluster tests:* skipping some tests in test_spilling > I found in the logs that the "test_spilling" test was skipped as the > exploration strategy was set to "core" on our Impala setup. > > Some question here, > a. From a Impala release perspective how significant are these strategies > (core, exhaustive, etc.) ? > b. Do we have to test with all combinations (core|release mode build and > exhaustive|release mode build). > c. Does the exploration strategy selection also affect the test data > loaded ? (data loaded is different in each exploration strategy ? ) > > Please let me know your comments. > > Regards, > Valencia > > [image: Inactive hide details for Tim Armstrong ---07/19/2016 09:11:48 > PM---With 2, it's a little strange that test_spilling is being s]Tim > Armstrong ---07/19/2016 09:11:48 PM---With 2, it's a little strange that > test_spilling is being skipped - I think that one should be run. > > From: Tim Armstrong > To: Valencia Serrao/Austin/Contr/IBM@IBMUS > Cc: dev@impala.incubator.apache.org, Manish Patil/Austin/Contr/IBM@IBMUS, > Nishidha Panpaliya/Austin/Contr/IBM@IBMUS, Sudarshan > Jagadale/Austin/Contr/IBM@IBMUS > Date: 07/19/2016 09:11 PM > > Subject: Re: Issues with tests in Release-mode Impala build > ------------------------------ > > > > With 2, it's a little strange that test_spilling is being skipped - I > think that one should be run. > > On Tue, Jul 19, 2016 at 8:39 AM, Tim Armstrong <*tarmstrong@cloudera.com* > > wrote: > > It looks like the benchmark-test issue is something to do with the > granularity of the clock. It can get stuck in an infinite loop if the > function call below always takes less than the smallest measurable unit of > time (i.e. Start() and Stop() are called in the same time quantum). > > while (sw.ElapsedTime() < target_cycles) { > sw.Start(); > function(batch_size, args); > sw.Stop(); > iters += batch_size; > } > > We use Intel's rdtsc instruction for a timer here, so I guess whatever > PPC alternative you used may work a little differently. This is probably > ok, but it's possible that it could affect timers elsewhere in Impala. > > One solution would be to increase the default batch size. > > On Tue, Jul 19, 2016 at 5:29 AM, Valencia Serrao <*vserrao@us.ibm.com* > > wrote: > Hi Tim, > > Following are some observations: > > 1. *BE test -issue: benchmark-test hangs* > Putting trace logs like below in benchmark.cc: > > > > > > > > * while (sw.ElapsedTime() < target_cycles) { LOG(INFO) <<" in > while(sw.ElapsedTime() < target_cycles)"; sw.Start(); function(batch_size, > args); sw.Stop(); iters += batch_size; LOG(INFO) <<" In while:::::::: > sw.ElapsedTime() "<< sw.ElapsedTime(); LOG(INFO) <<" In while:::::::: iters > = " << iters ;* > > In Release mode, I observed that the *sw.ElapsedTime()* remains > constant and does not increase, therefore, it is caught up in an infinite > loop and the benchmark-test hangs. In Debug mode, *sw.ElapsedTime()* > keeps on increasing and therefore is able to come out of the while loop and > benchmark-test doesn't hang in Debug mode. > I'm working on this issue, however, if you could give any pointers > about it, that would be really great. > > 2. *Custom cluster tests: *I have included the code changes in my > branch and many of the earlier 36 skipped tests have now executed and they > pass, but with the following exception(when compared to the output in the > *https://issues.cloudera.org/browse/IMPALA-3614* > ): > custom_cluster/test_spilling.py sss. > > * Current CC test stats:* 34 passed, 7 skipped, 3 warnings. > > 3.* End-to-End tests:* I couldn't dive into the EE tests. I will > surely let you know more about them as soon as I'm done with them. > > Regards, > Valencia > > [image: Inactive hide details for Valencia Serrao---07/19/2016 > 10:26:31 AM---Hi Tim, Thank you for the information.]Valencia > Serrao---07/19/2016 10:26:31 AM---Hi Tim, Thank you for the information. > > From: Valencia Serrao/Austin/Contr/IBM > To: Tim Armstrong <*tarmstrong@cloudera.com* > > Cc: *dev@impala.incubator.apache.org* , > Manish Patil/Austin/Contr/IBM@IBMUS, Nishidha > Panpaliya/Austin/Contr/IBM@IBMUS, Sudarshan > Jagadale/Austin/Contr/IBM@IBMUS > Date: 07/19/2016 10:26 AM > Subject: Re: Issues with tests in Release-mode Impala build > ------------------------------ > > > Hi Tim, > > Thank you for the information. > > I am working on the pointers you have given and also on the fix for > Custom cluster (skipped) tests. I will inform you on the findings. > > Regards, > Valencia > > > > [image: Inactive hide details for Tim Armstrong ---07/18/2016 09:19:52 > PM---Hi Valencia, 1. We run tests in release mode nightly and it]Tim > Armstrong ---07/18/2016 09:19:52 PM---Hi Valencia, 1. We run tests in > release mode nightly and it doesn't look like we've seen > > From: Tim Armstrong <*tarmstrong@cloudera.com* > > > To: *dev@impala.incubator.apache.org* > Cc: Valencia Serrao/Austin/Contr/IBM@IBMUS, Nishidha > Panpaliya/Austin/Contr/IBM@IBMUS, Sudarshan > Jagadale/Austin/Contr/IBM@IBMUS, Manish Patil/Austin/Contr/IBM@IBMUS > Date: 07/18/2016 09:19 PM > Subject: Re: Issues with tests in Release-mode Impala build > ------------------------------ > > > > Hi Valencia, > > 1. We run tests in release mode nightly and it doesn't look like we've > seen this hang. I'd suggest you attach a debugger to the benchmark-test > process and see what it's doing. It could either be an actual hang, or an > infinite/very long loop. That test is only testing our benchmarking > utilities, not Impala itself, but IMO it's always good to understand why > something like that is happening in case there's a more general problem. > 2. Sounds like *https://issues.cloudera.org/browse/IMPALA-3614* > . Have you got the > fix for that in your branch? > 3. Look forward to hearing more. > > Cheers, > Tim > > On Mon, Jul 18, 2016 at 2:49 AM, Valencia Serrao <*vserrao@us.ibm.com* > > wrote: > > Hi All, > > I have built Impala in Release mode. I executed the tests, > following are > some observations: > > 1. BE test: The test execution hangs at the "benchmark-test". > There are no > errors shown and it hangs at this test. Earlier, running the BE > tests in > debug mode this issue did not occur. > 2. Custom Cluster test: 5 tests passed and 36 tests skipped. All > of the > skipped cases give the message: "INSERT not implemented for S3" > 3. EE tests: I've also seen some failures here (yet to check the > details) > > As for FE and JDBC tests, everything works fine, release mode > test output > is same as that of debug mode test output. > > Is the "benchmark-test" test known to fail in Release mode or > am I missing > out on any configuration. Also, I want to understand the > significance of > this test, if in case we could ignore it and move ahead. > > > > Regards, > Valencia > > > > > > > > --001a114847ae0b246705383dd53e Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
2a.
Exhaustive is a superset of cor= e. We run the core tests pre-commit on CentOS 6 + HDFS and the full exhaust= ive tests post-commit on a wider range of configurations. We don't rele= ase Impala unless all exhaustive tests passed on all configurations we test= (if there's a valid reason why something doesn't work on a given p= latform we skip the test).

2b.
Exhaustive is a = superset of core, so if exhaustive passes then core should do. The exhausti= ve build takes much longer than core so it makes sense to run it less frequ= ently (e.g. we run it nightly for some configurations and weekly for others= ).

2c.
Confusingly, the core/exhaustive dat= a load doesn't map to core/exhaustive tests. We actually use the same d= ata load for all test configurations. See testdata/bin/create-load-data.sh = for how the core/exhaustive data load is invoked. E.g. we load the function= al data with exhaustive (i.e. all supported file formats) and the larger tp= c-h/tpc-ds data sets for only a subset of file forms.


On Wed, Jul 20, 201= 6 at 9:39 PM, Valencia Serrao <vserrao@us.ibm.com> wrote:

Hi Tim,

Thank you for the i= nsight on the issues.

1. BE tes= t -issue: benchmark-test hangs
 As you suggested, I= increased the "batch_size" value to upto 125000000, however, the= sw.ElapsedTime() does not increase inside the while and again gets caught = up in an infinite loop. The optimization level seems to cause this behavior= . I am still working in this.

2. Custom cluster= tests: skipping some tests in test_spilling
I found in t= he logs that the "test_spilling" test was skipped as the explorat= ion strategy was set to "core" on our Impala setup.

= Some question here,
a. From a Impala release perspective how s= ignificant are these strategies (core, exhaustive, etc.) ?
b. Do we= have to test with all combinations (core|release mode build and exhaustive= |release mode build).
c. Does the exploration strategy selection a= lso affect the test data loaded ? (data loaded is different in each explora= tion strategy ? )

Please let me know your comments.

Regards,<= br>Valencia

3D"InactiveTim Armstrong= ---07/19/2016 09:11:48 PM---With 2, it's a little strange that test_sp= illing is being skipped - I think that one should be run.

From: Tim Ar= mstrong <ta= rmstrong@cloudera.com>
= To: Valencia Serrao/Austin/Contr/IBM@IBMUS
Cc: dev@impala.incubator.apache.org, Manish Patil/Austin/Con= tr/IBM@IBMUS, Nishidha Panpaliya/Austin/Contr/IBM@IBMUS, Sudarshan Jagadale= /Austin/Contr/IBM@IBMUS
Date: 07/19/2016 09:11 PM


Subject: Re: Issues with tests in Release-mode Impala build



With 2, it's a little strange th= at test_spilling is being skipped - I think that one should be run.<= br>
On Tue, Jul 19, 2016 at 8:39 AM, Tim Armstrong <= tarmstrong@cloudera.com> wrote:


--001a114847ae0b246705383dd53e-- --001a114847ae0b246a05383dd53f--