Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id CE69D200BD7 for ; Thu, 3 Nov 2016 05:16:00 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id CD03E160B0A; Thu, 3 Nov 2016 04:16:00 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id C7096160AFB for ; Thu, 3 Nov 2016 05:15:59 +0100 (CET) Received: (qmail 58522 invoked by uid 500); 3 Nov 2016 04:15:59 -0000 Mailing-List: contact issues-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hive.apache.org Delivered-To: mailing list issues@hive.apache.org Received: (qmail 58510 invoked by uid 99); 3 Nov 2016 04:15:59 -0000 Received: from arcas.apache.org (HELO arcas) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Nov 2016 04:15:58 +0000 Received: from arcas.apache.org (localhost [127.0.0.1]) by arcas (Postfix) with ESMTP id D19E22C1F5A for ; Thu, 3 Nov 2016 04:15:58 +0000 (UTC) Date: Thu, 3 Nov 2016 04:15:58 +0000 (UTC) From: "Hive QA (JIRA)" To: issues@hive.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (HIVE-14815) Implement Parquet vectorization reader for Primitive types MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 archived-at: Thu, 03 Nov 2016 04:16:01 -0000 [ https://issues.apache.org/jira/browse/HIVE-14815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15631499#comment-15631499 ] Hive QA commented on HIVE-14815: -------------------------------- Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12836718/HIVE-14815.4.patch {color:red}ERROR:{color} -1 due to build exiting with an error Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1935/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1935/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-1935/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Tests exited with: NonZeroExitCodeException Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ date '+%Y-%m-%d %T.%3N' 2016-11-03 04:14:52.111 + [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]] + export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + export PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m ' + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m ' + export 'MAVEN_OPTS=-Xmx1g -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128' + MAVEN_OPTS='-Xmx1g -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128' + cd /data/hiveptest/working/ + tee /data/hiveptest/logs/PreCommit-HIVE-Build-1935/source-prep.txt + [[ false == \t\r\u\e ]] + mkdir -p maven ivy + [[ git = \s\v\n ]] + [[ git = \g\i\t ]] + [[ -z master ]] + [[ -d apache-github-source-source ]] + [[ ! -d apache-github-source-source/.git ]] + [[ ! -d apache-github-source-source ]] + date '+%Y-%m-%d %T.%3N' 2016-11-03 04:14:52.114 + cd apache-github-source-source + git fetch origin + git reset --hard HEAD HEAD is now at 345353c HIVE-15039: A better job monitor console output for HoS (Rui reviewed by Xuefu and Ferdinand) + git clean -f -d Removing common/src/java/org/apache/hadoop/hive/common/ValidWriteIds.java Removing common/src/java/org/apache/hadoop/hive/conf/HiveConf.java.orig Removing common/src/test/org/apache/hive/common/util/MockFileSystem.java Removing metastore/scripts/upgrade/derby/037-HIVE-14637.derby.sql Removing metastore/scripts/upgrade/mssql/022-HIVE-14637.mssql.sql Removing metastore/scripts/upgrade/mysql/037-HIVE-14637.mysql.sql Removing metastore/scripts/upgrade/oracle/037-HIVE-14637.oracle.sql Removing metastore/scripts/upgrade/postgres/036-HIVE-14637.postgres.sql Removing metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/FinalizeWriteIdRequest.java Removing metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/FinalizeWriteIdResult.java Removing metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetNextWriteIdRequest.java Removing metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetNextWriteIdResult.java Removing metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetValidWriteIdsRequest.java Removing metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetValidWriteIdsResult.java Removing metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/HeartbeatWriteIdRequest.java Removing metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/HeartbeatWriteIdResult.java Removing metastore/src/java/org/apache/hadoop/hive/metastore/MmCleanerThread.java Removing metastore/src/model/org/apache/hadoop/hive/metastore/model/MTableWrite.java Removing ql/src/java/org/apache/hadoop/hive/ql/exec/ImportCommitTask.java Removing ql/src/java/org/apache/hadoop/hive/ql/exec/ImportCommitWork.java Removing ql/src/test/queries/clientnegative/mm_concatenate.q Removing ql/src/test/queries/clientnegative/mm_truncate_cols.q Removing ql/src/test/queries/clientpositive/mm_all.q Removing ql/src/test/queries/clientpositive/mm_all2.q Removing ql/src/test/queries/clientpositive/mm_conversions.q Removing ql/src/test/queries/clientpositive/mm_current.q Removing ql/src/test/queries/clientpositive/mm_insertonly_acid.q Removing ql/src/test/results/clientnegative/mm_concatenate.q.out Removing ql/src/test/results/clientnegative/mm_truncate_cols.q.out Removing ql/src/test/results/clientpositive/llap/mm_all.q.out Removing ql/src/test/results/clientpositive/llap/mm_all2.q.out Removing ql/src/test/results/clientpositive/llap/mm_conversions.q.out Removing ql/src/test/results/clientpositive/llap/mm_current.q.out Removing ql/src/test/results/clientpositive/mm_insertonly_acid.q.out + git checkout master Already on 'master' Your branch is up-to-date with 'origin/master'. + git reset --hard origin/master HEAD is now at 345353c HIVE-15039: A better job monitor console output for HoS (Rui reviewed by Xuefu and Ferdinand) + git merge --ff-only origin/master Already up-to-date. + date '+%Y-%m-%d %T.%3N' 2016-11-03 04:14:53.221 + patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh + patchFilePath=/data/hiveptest/working/scratch/build.patch + [[ -f /data/hiveptest/working/scratch/build.patch ]] + chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh + /data/hiveptest/working/scratch/smart-apply-patch.sh /data/hiveptest/working/scratch/build.patch Going to apply patch with: patch -p0 patching file ql/src/java/org/apache/hadoop/hive/ql/io/parquet/MapredParquetInputFormat.java patching file ql/src/java/org/apache/hadoop/hive/ql/io/parquet/ParquetRecordReaderBase.java patching file ql/src/java/org/apache/hadoop/hive/ql/io/parquet/VectorizedParquetInputFormat.java patching file ql/src/java/org/apache/hadoop/hive/ql/io/parquet/read/DataWritableReadSupport.java patching file ql/src/java/org/apache/hadoop/hive/ql/io/parquet/read/ParquetRecordReaderWrapper.java patching file ql/src/java/org/apache/hadoop/hive/ql/io/parquet/timestamp/NanoTimeUtils.java patching file ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/VectorizedColumnReader.java patching file ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/VectorizedParquetRecordReader.java patching file ql/src/test/org/apache/hadoop/hive/ql/io/parquet/TestVectorizedColumnReader.java patching file ql/src/test/queries/clientpositive/parquet_types_vectorization.q patching file ql/src/test/results/clientpositive/llap/vectorized_parquet.q.out patching file ql/src/test/results/clientpositive/llap/vectorized_parquet_types.q.out patching file ql/src/test/results/clientpositive/parquet_types_vectorization.q.out patching file ql/src/test/results/clientpositive/vectorized_parquet_types.q.out + [[ maven == \m\a\v\e\n ]] + rm -rf /data/hiveptest/working/maven/org/apache/hive + mvn -B clean install -DskipTests -T 4 -q -Dmaven.repo.local=/data/hiveptest/working/maven ANTLR Parser Generator Version 3.4 org/apache/hadoop/hive/metastore/parser/Filter.g DataNucleus Enhancer (version 4.1.6) for API "JDO" DataNucleus Enhancer : Classpath >> /usr/share/maven/boot/plexus-classworlds-2.x.jar ENHANCED (Persistable) : org.apache.hadoop.hive.metastore.model.MDatabase ENHANCED (Persistable) : org.apache.hadoop.hive.metastore.model.MFieldSchema ENHANCED (Persistable) : org.apache.hadoop.hive.metastore.model.MType ENHANCED (Persistable) : org.apache.hadoop.hive.metastore.model.MTable ENHANCED (Persistable) : org.apache.hadoop.hive.metastore.model.MConstraint ENHANCED (Persistable) : org.apache.hadoop.hive.metastore.model.MSerDeInfo ENHANCED (Persistable) : org.apache.hadoop.hive.metastore.model.MOrder ENHANCED (Persistable) : org.apache.hadoop.hive.metastore.model.MColumnDescriptor ENHANCED (Persistable) : org.apache.hadoop.hive.metastore.model.MStringList ENHANCED (Persistable) : org.apache.hadoop.hive.metastore.model.MStorageDescriptor ENHANCED (Persistable) : org.apache.hadoop.hive.metastore.model.MPartition ENHANCED (Persistable) : org.apache.hadoop.hive.metastore.model.MIndex ENHANCED (Persistable) : org.apache.hadoop.hive.metastore.model.MRole ENHANCED (Persistable) : org.apache.hadoop.hive.metastore.model.MRoleMap ENHANCED (Persistable) : org.apache.hadoop.hive.metastore.model.MGlobalPrivilege ENHANCED (Persistable) : org.apache.hadoop.hive.metastore.model.MDBPrivilege ENHANCED (Persistable) : org.apache.hadoop.hive.metastore.model.MTablePrivilege ENHANCED (Persistable) : org.apache.hadoop.hive.metastore.model.MPartitionPrivilege ENHANCED (Persistable) : org.apache.hadoop.hive.metastore.model.MTableColumnPrivilege ENHANCED (Persistable) : org.apache.hadoop.hive.metastore.model.MPartitionColumnPrivilege ENHANCED (Persistable) : org.apache.hadoop.hive.metastore.model.MPartitionEvent ENHANCED (Persistable) : org.apache.hadoop.hive.metastore.model.MMasterKey ENHANCED (Persistable) : org.apache.hadoop.hive.metastore.model.MDelegationToken ENHANCED (Persistable) : org.apache.hadoop.hive.metastore.model.MTableColumnStatistics ENHANCED (Persistable) : org.apache.hadoop.hive.metastore.model.MPartitionColumnStatistics ENHANCED (Persistable) : org.apache.hadoop.hive.metastore.model.MVersionTable ENHANCED (Persistable) : org.apache.hadoop.hive.metastore.model.MResourceUri ENHANCED (Persistable) : org.apache.hadoop.hive.metastore.model.MFunction ENHANCED (Persistable) : org.apache.hadoop.hive.metastore.model.MNotificationLog ENHANCED (Persistable) : org.apache.hadoop.hive.metastore.model.MNotificationNextId DataNucleus Enhancer completed with success for 30 classes. Timings : input=175 ms, enhance=218 ms, total=393 ms. Consult the log for full details Generating vector expression code Generating vector expression test code ANTLR Parser Generator Version 3.4 org/apache/hadoop/hive/ql/parse/HiveLexer.g org/apache/hadoop/hive/ql/parse/HiveParser.g [ERROR] Failed to execute goal org.apache.maven.plugins:maven-remote-resources-plugin:1.5:process (default) on project hive-exec: Error resolving project artifact: Could not transfer artifact org.pentaho:pentaho-aggdesigner-algorithm:pom:5.1.5-jhyde from/to datanucleus (http://www.datanucleus.org/downloads/maven2): Connect to localhost:3128 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused for project org.pentaho:pentaho-aggdesigner-algorithm:jar:5.1.5-jhyde -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn -rf :hive-exec + exit 1 ' {noformat} This message is automatically generated. ATTACHMENT ID: 12836718 - PreCommit-HIVE-Build > Implement Parquet vectorization reader for Primitive types > ----------------------------------------------------------- > > Key: HIVE-14815 > URL: https://issues.apache.org/jira/browse/HIVE-14815 > Project: Hive > Issue Type: Sub-task > Reporter: Ferdinand Xu > Assignee: Ferdinand Xu > Attachments: HIVE-14815.1.patch, HIVE-14815.2.patch, HIVE-14815.3.patch, HIVE-14815.4.patch, HIVE-14815.patch > > > Parquet doesn't provide a vectorized reader which can be used by Hive directly. Also for Decimal Column batch, it consists of a batch of HiveDecimal which is a Hive type which is unknown for Parquet. To support Hive vectorization execution engine in Hive, we have to implement the vectorized Parquet reader in Hive side. To limit the performance impacts, we need to implement a page level vectorized reader. -- This message was sent by Atlassian JIRA (v6.3.4#6332)