From issues-return-35422-archive-asf-public=cust-asf.ponee.io@carbondata.apache.org Tue Feb 13 03:13:46 2018 Return-Path: X-Original-To: archive-asf-public@eu.ponee.io Delivered-To: archive-asf-public@eu.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by mx-eu-01.ponee.io (Postfix) with ESMTP id EE8AC180652 for ; Tue, 13 Feb 2018 03:13:46 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id DE57A160C61; Tue, 13 Feb 2018 02:13:46 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 0CAE6160C3F for ; Tue, 13 Feb 2018 03:13:45 +0100 (CET) Received: (qmail 39987 invoked by uid 500); 13 Feb 2018 02:13:45 -0000 Mailing-List: contact issues-help@carbondata.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@carbondata.apache.org Delivered-To: mailing list issues@carbondata.apache.org Received: (qmail 39974 invoked by uid 99); 13 Feb 2018 02:13:45 -0000 Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Feb 2018 02:13:45 +0000 Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33) id 2857CDFF5A; Tue, 13 Feb 2018 02:13:45 +0000 (UTC) From: jackylk To: issues@carbondata.apache.org Reply-To: issues@carbondata.apache.org Message-ID: Subject: [GitHub] carbondata pull request #1976: Support generating assembling JAR for store-s... Content-Type: text/plain Date: Tue, 13 Feb 2018 02:13:45 +0000 (UTC) GitHub user jackylk opened a pull request: https://github.com/apache/carbondata/pull/1976 Support generating assembling JAR for store-sdk module Support generating assembling JAR for store-sdk module and remove junit dependency - [X] Any interfaces changed? No - [X] Any backward compatibility impacted? No - [X] Document update required? No - [X] Testing done No - [X] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. NA You can merge this pull request into a Git repository by running: $ git pull https://github.com/jackylk/incubator-carbondata junit Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/1976.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1976 ---- commit 952665a8c1c52f28951463fef989333ae0e6d83e Author: Jacky Li Date: 2018-01-06T12:28:44Z [CARBONDATA-1992] Remove partitionId in CarbonTablePath In CarbonTablePath, there is a deprecated partition id which is always 0, it should be removed to avoid confusion. This closes #1765 commit 111c3821557820241d1114d87eae2f7cd017e610 Author: Jacky Li Date: 2018-01-02T15:46:14Z [CARBONDATA-1968] Add external table support This PR adds support for creating external table with existing carbondata files, using Hive syntax. CREATE EXTERNAL TABLE tableName STORED BY 'carbondata' LOCATION 'path' This closes #1749 commit 80b42ac662ebd2bc243ca91c86b035717223daf4 Author: SangeetaGulia Date: 2017-09-21T09:26:26Z [CARBONDATA-1827] S3 Carbon Implementation 1.Provide support for s3 in carbondata. 2.Added S3Example to create carbon table on s3. 3.Added S3CSVExample to load carbon table using csv from s3. This closes #1805 commit 71c2d8ca4a3212cff1eedbe78ee03e521f57fbbc Author: Jacky Li Date: 2018-01-31T16:25:31Z [REBASE] Solve conflict after rebasing master commit 15b4e192ee904a2e7c845ac67e0fcf1ba151a683 Author: Jacky Li Date: 2018-01-30T13:24:04Z [CARBONDATA-2099] Refactor query scan process to improve readability Unified concepts in scan process flow: 1.QueryModel contains all parameter for scan, it is created by API in CarbonTable. (In future, CarbonTable will be the entry point for various table operations) 2.Use term ColumnChunk to represent one column in one blocklet, and use ChunkIndex in reader to read specified column chunk 3.Use term ColumnPage to represent one page in one ColumnChunk 4.QueryColumn => ProjectionColumn, indicating it is for projection This closes #1874 commit c3e99681bcd397ed33bc90e8d73b1fd33e0e60f7 Author: Jacky Li Date: 2018-01-31T08:14:27Z [CARBONDATA-2025] Unify all path construction through CarbonTablePath static method Refactory CarbonTablePath: 1.Remove CarbonStorePath and use CarbonTablePath only. 2.Make CarbonTablePath an utility without object creation, it can avoid creating object before using it, thus code is cleaner and GC is less. This closes #1768 commit e502c59a2d0b95d80db3aff04c749654254eadbe Author: Jatin Date: 2018-01-25T11:23:00Z [CARBONDATA-2080] [S3-Implementation] Propagated hadoopConf from driver to executor for s3 implementation in cluster mode. Problem : hadoopconf was not getting propagated from driver to the executor that's why load was failing to the distributed environment. Solution: Setting the Hadoop conf in base class CarbonRDD How to verify this PR : Execute the load in the cluster mode It should be a success using location s3. This closes #1860 commit cae74a8cecea74e8899a87dcb7d12e0dec1b8069 Author: sounakr Date: 2017-09-28T10:51:05Z [CARBONDATA-1480]Min Max Index Example for DataMap Datamap Example. Implementation of Min Max Index through Datamap. And Using the Index while prunning. This closes #1359 commit e972fd3d5cc8f392d47ca111b2d8f262edb29ac6 Author: ravipesala Date: 2017-11-15T14:18:40Z [CARBONDATA-1544][Datamap] Datamap FineGrain implementation Implemented interfaces for FG datamap and integrated to filterscanner to use the pruned bitset from FG datamap. FG Query flow as follows. 1.The user can add FG datamap to any table and implement there interfaces. 2. Any filter query which hits the table with datamap will call prune method of FGdatamap. 3. The prune method of FGDatamap return list FineGrainBlocklet , these blocklets contain the information of block, blocklet, page and rowids information as well. 4. The pruned blocklets are internally wriitten to file and returns only the block , blocklet and filepath information as part of Splits. 5. Based on the splits scanrdd schedule the tasks. 6. In filterscanner we check the datamapwriterpath from split and reNoteads the bitset if exists. And pass this bitset as input to it. This closes #1471 commit cd7eed66bdd7b0044953cb5bf037f6cce38c9e12 Author: xuchuanyin Date: 2018-02-08T07:39:45Z [HotFix][CheckStyle] Fix import related checkstyle This closes #1952 commit de92ea9a123b17d903f2d1d4662299315c792954 Author: xuchuanyin Date: 2018-02-08T06:35:14Z [CARBONDATA-2018][DataLoad] Optimization in reading/writing for sort temp row Pick up the no-sort fields in the row and pack them as bytes array and skip parsing them during merge sort to reduce CPU consumption This closes #1792 commit 6dd8b038fc898dbf48ad30adfc870c19eb38e3d0 Author: xuchuanyin Date: 2018-02-08T06:42:39Z [CARBONDATA-2023][DataLoad] Add size base block allocation in data loading Carbondata assign blocks to nodes at the beginning of data loading. Previous block allocation strategy is block number based and it will suffer skewed data problem if the size of input files differs a lot. We introduced a size based block allocation strategy to optimize data loading performance in skewed data scenario. This closes #1808 commit e5c32ac96f4cf85ef7a42f2a14c31c19418a789b Author: Jacky Li Date: 2018-02-10T02:34:59Z Revert "[CARBONDATA-2023][DataLoad] Add size base block allocation in data loading" This reverts commit 6dd8b038fc898dbf48ad30adfc870c19eb38e3d0. commit e1c6448cdbfa8d5eab1a861485f953eea3984f1f Author: Jacky Li Date: 2018-02-10T12:11:25Z Revert "[CARBONDATA-2018][DataLoad] Optimization in reading/writing for sort temp row" This reverts commit de92ea9a123b17d903f2d1d4662299315c792954. commit 7f5751a78c28c8a428fa62a5f82858ac65415c86 Author: Jacky Li Date: 2018-02-11T02:12:10Z [CARBONDATA-2156] Add interface annotation InterfaceAudience and InterfaceStability annotation should be added for user and developer 1.InetfaceAudience can be User and Developer 2.InterfaceStability can be Stable, Evolving, Unstable This closes #1968 commit a848ccff8aaf3c10970c61b1f85bce56478ca0ac Author: Jacky Li Date: 2018-02-10T11:44:23Z [CARBONDATA-1997] Add CarbonWriter SDK API Added a new module called store-sdk, and added a CarbonWriter API, it can be used to write Carbondata files to a specified folder, without Spark and Hadoop dependency. User can use this API in any environment. This closes #1967 commit 0d50f65461ae3855db66f44fa06e01174de50ccd Author: Jacky Li Date: 2018-02-11T13:37:04Z [CARBONDATA-2159] Remove carbon-spark dependency in store-sdk module To make assembling JAR of store-sdk module, it should not depend on carbon-spark module This closes #1970 commit 937bdb867aeac5159f51de3582c4556c949bfb5c Author: xuchuanyin Date: 2018-02-08T06:35:14Z [CARBONDATA-2018][DataLoad] Optimization in reading/writing for sort temp row Pick up the no-sort fields in the row and pack them as bytes array and skip parsing them during merge sort to reduce CPU consumption This closes #1792 commit fd450b151cb5858504116f560da5cd7a357894e5 Author: xuchuanyin Date: 2018-02-08T06:42:39Z [CARBONDATA-2023][DataLoad] Add size base block allocation in data loading Carbondata assign blocks to nodes at the beginning of data loading. Previous block allocation strategy is block number based and it will suffer skewed data problem if the size of input files differs a lot. We introduced a size based block allocation strategy to optimize data loading performance in skewed data scenario. This closes #1808 commit b1921806a125eff83dcc1882ad6b702512c387f8 Author: Jacky Li Date: 2018-02-13T01:12:09Z wip commit 4bc1bf34fc7cb508a9e45b95740028fa82daf1b3 Author: Jacky Li Date: 2018-02-13T01:36:56Z add test scope for junit and jmockit ---- ---