Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 15D1F200D27 for ; Wed, 11 Oct 2017 06:28:12 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 14513160BE1; Wed, 11 Oct 2017 04:28:12 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 30BB3160BE0 for ; Wed, 11 Oct 2017 06:28:11 +0200 (CEST) Received: (qmail 62130 invoked by uid 500); 11 Oct 2017 04:28:10 -0000 Mailing-List: contact issues-help@drill.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@drill.apache.org Delivered-To: mailing list issues@drill.apache.org Received: (qmail 62121 invoked by uid 99); 11 Oct 2017 04:28:10 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Oct 2017 04:28:10 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 6823919527F for ; Wed, 11 Oct 2017 04:28:09 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -98.952 X-Spam-Level: X-Spam-Status: No, score=-98.952 tagged_above=-999 required=6.31 tests=[KAM_ASCII_DIVIDERS=0.8, KAM_LOTSOFHASH=0.25, RP_MATCHES_RCVD=-0.001, SPF_PASS=-0.001, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id b8vuzgrnVpF8 for ; Wed, 11 Oct 2017 04:28:07 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTP id 407F65FD42 for ; Wed, 11 Oct 2017 04:28:06 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 12DC9E0F2F for ; Wed, 11 Oct 2017 04:28:05 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id DBD682538E for ; Wed, 11 Oct 2017 04:28:01 +0000 (UTC) Date: Wed, 11 Oct 2017 04:28:01 +0000 (UTC) From: "ASF GitHub Bot (JIRA)" To: issues@drill.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (DRILL-5783) Make code generation in the TopN operator more modular and test it MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 archived-at: Wed, 11 Oct 2017 04:28:12 -0000 [ https://issues.apache.org/jira/browse/DRILL-5783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16199790#comment-16199790 ] ASF GitHub Bot commented on DRILL-5783: --------------------------------------- GitHub user ilooner opened a pull request: https://github.com/apache/drill/pull/984 DRILL-5783 Made a unit test for generated Priority Queue. DRILL-5841 Fix tmp folder errors. ## DRILL-5783 - A unit test is created for the priority queue in the TopN operator - The code generation classes passed around a completely unused function registry reference in some places so I removed it. - The priority queue had unused parameters for some of its methods so I removed them. - I added an add method to value vectors so that value vectors can easily be generated in unit tests. ## DRILL-5841 - There were many many ways in which temporary folders were created in unit tests. I have unified the way these folders are created with the DirTestWatcher, SubDirTestWatcher, and BaseDirTestWatcher. All the unit tests have been updated to use these. The test watchers create temp directories in ./target//. So all the files generated and used in the context of a test can easily be found in the same consistent location. - This change should fix the sporadic hashagg test failures, as well as failures caused by stray files in /tmp Misc - Added a TableBuilder so we don't have to becom human json parsers to generate test data. - A Pstore test could fail sporadically - General code cleanup. - There are many places where String.format is used unnecessarily. The test builder methods already use String.format for you when you pass them args. I cleaned some of these up. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ilooner/drill DRILL-5783 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/drill/pull/984.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #984 ---- commit c9c554340a4a2ca8bab257a478d8201cd5f4c92a Author: Timothy Farkas Date: 2017-09-18T23:21:44Z - Added property for codegen dump. - Removed unnecessary FunctionRegistry argument to the code generator. commit 68f77a2e9a904bcd5a808fefd824f2f108134aec Author: Timothy Farkas Date: 2017-09-18T23:31:38Z - Removed unused imports commit eff908b4b18fe5ccb67480f6e0c36a61e27d12e2 Author: Timothy Farkas Date: 2017-09-18T23:34:33Z - Removed unnecessary modifiers from the priority queue interface. commit 319fae58faaf78f807d8d6f9c0928f801ec7e352 Author: Timothy Farkas Date: 2017-09-19T00:28:58Z - Made creating a priority queue more modular by refactoring the createNewPriorityQueue method commit e82e316a628e3c2f3a1f9ef7005299065eb4379d Author: Timothy Farkas Date: 2017-09-19T21:08:19Z - Have some codegen dump test code done commit e2057d6c9abc301889ab46a12848cee7506f45d5 Author: Timothy Farkas Date: 2017-09-21T22:07:52Z - Made fix commit 6289cff376f37c8d4e2984b309d7f6eb13703f2f Author: Timothy Farkas Date: 2017-09-28T23:24:38Z - Removed unused imports - Removed commented out code - Formatting fixes commit cb80f885c86a00cda82e114b3aec212890ce6d90 Author: Timothy Farkas Date: 2017-09-29T23:03:49Z - Added the option to the DirTestWatcher to not delete the tmp dir at the end of the test - Added a TableFileBuilder to be used in testing - Changed the External sort test to use the TableFileBuilder commit bb58b21ad16d6adbeb11c66d5a38d11c20a0b05d Author: Timothy Farkas Date: 2017-09-29T23:06:15Z - Delete the directory at the beginning of a test in DirTestWatcher commit f0d89abf2dc3422081bcd49183b8bf0ae3742996 Author: Timothy Farkas Date: 2017-09-30T01:51:13Z - Used existing test framework for the TopNBatchTest commit 7f8d978c12d0e2b2efd3b3c64f5450ca76f0ef0e Author: Timothy Farkas Date: 2017-10-05T06:27:55Z - Removed passing around contexts unnecessarily in the TopNBatch operator. - Fixed some javadocs commit c539ebd20c421d8672440ad3bb863c446f39e775 Author: Timothy Farkas Date: 2017-10-06T03:22:04Z - Added a record batch builder, and added an add method to value vector mutators. commit 89a5f8576bd2f50ce540b633a28ace55b31f08e6 Author: Timothy Farkas Date: 2017-10-06T05:46:48Z - Added Javadoc to the PriorityQueue. - Removed an unnecessary method - Got the unit test to work commit 540715d250601590367d7bc71614ab52e891c6ec Author: Timothy Farkas Date: 2017-10-06T10:27:46Z - Added utilities for comparing vector batch results - Removed redundant unit tests commit 9b0c557ec0c6030969dae620c5954e5dfb2119e9 Author: Timothy Farkas Date: 2017-10-06T10:30:40Z - Reduced the amount of data generated for the topN unit test. commit 7ed3dcf7bf0ce807b81e72a17d91d909ac788415 Author: Timothy Farkas Date: 2017-10-06T16:14:21Z - Moved comparator function in DrillTestWrapper into a comparator class - Made the TopN unit test work commit 3e9d8905699a56c36cbdbf253daf374ae0983e65 Author: Timothy Farkas Date: 2017-10-06T16:20:03Z - Fixed missing license commit 55e35ab8a84b9eb32fdf1b674a23a3f45e1e9e55 Author: Timothy Farkas Date: 2017-10-06T16:31:07Z - Fixed checkstyle errors commit 0c310225d47a78a35c9cafd70869b7520edfa17c Author: Timothy Farkas Date: 2017-10-07T04:00:00Z - Fixed javadoc errors in FixtureBuilder commit 133a44b88355de031695cf8d9c3503618219e4fe Author: Timothy Farkas Date: 2017-10-07T04:10:32Z - Cleaning up unused variables and methods in FixtureBuilder commit a685b5718f2c25d1797d378bdf35be9a88559048 Author: Timothy Farkas Date: 2017-10-07T04:34:33Z - Fixed errors after rebase commit b920eb3ed4e29cf72a74ceaf43bc4c418f4dcefa Author: Timothy Farkas Date: 2017-10-07T09:55:10Z - Used DirTestWatcher to create temporary directories in the ClusterFixture - Fixed broken unit tests commit 96defcfcee6b602df0051e7ec8516ef627a44388 Author: Timothy Farkas Date: 2017-10-08T00:57:32Z - Made the RecordBatchBuilder and BatchUtils use the same table format as the DrillTestWrapper. Also added a more detailed comparison function for tables. commit 8812043bfed74c6f0f6db1a23c0d029b4fd29222 Author: Timothy Farkas Date: 2017-10-08T00:59:40Z - Moved files in testutils package to test package commit 1b1dada9c712c9283b4765cb7142af967ba5dd24 Author: Timothy Farkas Date: 2017-10-08T01:02:10Z - Moved more testing utilities into the org.apache.drill.test package commit 5878073cf0ebb4d28a6da6fb4594ea52576d5b1d Author: Timothy Farkas Date: 2017-10-08T01:05:35Z - Fixed broken javadoc links commit d9ac7d485b6e580c7becf6caa8dc96ebe20465ed Author: Timothy Farkas Date: 2017-10-08T01:08:27Z - Moved unit tests out of org.apache.drill.test since that package should only contain testing utilities commit fe33936818d1b170651e6ea36b560d06af0033e7 Author: Timothy Farkas Date: 2017-10-10T09:19:24Z - Made all the tests use a uniform tmp directory system - Fixed sporadic failures due to stale files being present on build machines - Use an isolated workspace for each test not global /tmp folder - Fixed test failures due to stray files in /tmp commit dd8a2c230008739f20432389cb61515c6459eb05 Author: Timothy Farkas Date: 2017-10-10T11:07:41Z - Fixed broken tests commit 99115492fd621403d0862e2ff8b650479a5935e5 Author: Timothy Farkas Date: 2017-10-10T13:07:03Z - Fixed more testing bugs ---- > Make code generation in the TopN operator more modular and test it > ------------------------------------------------------------------ > > Key: DRILL-5783 > URL: https://issues.apache.org/jira/browse/DRILL-5783 > Project: Apache Drill > Issue Type: Improvement > Reporter: Timothy Farkas > Assignee: Timothy Farkas > -- This message was sent by Atlassian JIRA (v6.4.14#64029)