Return-Path: X-Original-To: apmail-hive-dev-archive@www.apache.org Delivered-To: apmail-hive-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id C9B5C11F5D for ; Fri, 15 Aug 2014 22:09:19 +0000 (UTC) Received: (qmail 49508 invoked by uid 500); 15 Aug 2014 22:09:19 -0000 Delivered-To: apmail-hive-dev-archive@hive.apache.org Received: (qmail 49436 invoked by uid 500); 15 Aug 2014 22:09:19 -0000 Mailing-List: contact dev-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hive.apache.org Delivered-To: mailing list dev@hive.apache.org Received: (qmail 49419 invoked by uid 500); 15 Aug 2014 22:09:19 -0000 Delivered-To: apmail-hadoop-hive-dev@hadoop.apache.org Received: (qmail 49416 invoked by uid 99); 15 Aug 2014 22:09:19 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 15 Aug 2014 22:09:19 +0000 Date: Fri, 15 Aug 2014 22:09:19 +0000 (UTC) From: "Hive QA (JIRA)" To: hive-dev@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (HIVE-7617) optimize bytes mapjoin hash table read path wrt serialization, at least for common cases MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HIVE-7617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14099248#comment-14099248 ] Hive QA commented on HIVE-7617: ------------------------------- {color:red}Overall{color}: -1 no tests executed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12662140/HIVE-7617.04.patch Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/345/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/345/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-345/ Messages: {noformat} **** This message was trimmed, see log for full details **** warning(200): IdentifiersParser.g:179:5: Decision can match input such as "KW_STRUCT" using multiple alternatives: 4, 6 As a result, alternative(s) 6 were disabled for that input warning(200): IdentifiersParser.g:179:5: Decision can match input such as "KW_ARRAY" using multiple alternatives: 2, 6 As a result, alternative(s) 6 were disabled for that input warning(200): IdentifiersParser.g:179:5: Decision can match input such as "KW_UNIONTYPE" using multiple alternatives: 5, 6 As a result, alternative(s) 6 were disabled for that input warning(200): IdentifiersParser.g:261:5: Decision can match input such as "KW_NULL" using multiple alternatives: 1, 8 As a result, alternative(s) 8 were disabled for that input warning(200): IdentifiersParser.g:261:5: Decision can match input such as "KW_TRUE" using multiple alternatives: 3, 8 As a result, alternative(s) 8 were disabled for that input warning(200): IdentifiersParser.g:261:5: Decision can match input such as "KW_DATE StringLiteral" using multiple alternatives: 2, 3 As a result, alternative(s) 3 were disabled for that input warning(200): IdentifiersParser.g:261:5: Decision can match input such as "KW_FALSE" using multiple alternatives: 3, 8 As a result, alternative(s) 8 were disabled for that input warning(200): IdentifiersParser.g:393:5: Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_CLUSTER KW_BY" using multiple alternatives: 2, 9 As a result, alternative(s) 9 were disabled for that input warning(200): IdentifiersParser.g:393:5: Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_MAP LPAREN" using multiple alternatives: 2, 9 As a result, alternative(s) 9 were disabled for that input warning(200): IdentifiersParser.g:393:5: Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_INSERT KW_OVERWRITE" using multiple alternatives: 2, 9 As a result, alternative(s) 9 were disabled for that input warning(200): IdentifiersParser.g:393:5: Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_GROUP KW_BY" using multiple alternatives: 2, 9 As a result, alternative(s) 9 were disabled for that input warning(200): IdentifiersParser.g:393:5: Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_INSERT KW_INTO" using multiple alternatives: 2, 9 As a result, alternative(s) 9 were disabled for that input warning(200): IdentifiersParser.g:393:5: Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_LATERAL KW_VIEW" using multiple alternatives: 2, 9 As a result, alternative(s) 9 were disabled for that input warning(200): IdentifiersParser.g:393:5: Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_SORT KW_BY" using multiple alternatives: 2, 9 As a result, alternative(s) 9 were disabled for that input warning(200): IdentifiersParser.g:393:5: Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_UNION KW_ALL" using multiple alternatives: 2, 9 As a result, alternative(s) 9 were disabled for that input warning(200): IdentifiersParser.g:393:5: Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_ORDER KW_BY" using multiple alternatives: 2, 9 As a result, alternative(s) 9 were disabled for that input warning(200): IdentifiersParser.g:393:5: Decision can match input such as "KW_BETWEEN KW_MAP LPAREN" using multiple alternatives: 8, 9 As a result, alternative(s) 9 were disabled for that input warning(200): IdentifiersParser.g:393:5: Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_DISTRIBUTE KW_BY" using multiple alternatives: 2, 9 As a result, alternative(s) 9 were disabled for that input warning(200): IdentifiersParser.g:518:5: Decision can match input such as "{AMPERSAND..BITWISEXOR, DIV..DIVIDE, EQUAL..EQUAL_NS, GREATERTHAN..GREATERTHANOREQUALTO, KW_AND, KW_ARRAY, KW_BETWEEN..KW_BOOLEAN, KW_CASE, KW_DOUBLE, KW_FLOAT, KW_IF, KW_IN, KW_INT, KW_LIKE, KW_MAP, KW_NOT, KW_OR, KW_REGEXP, KW_RLIKE, KW_SMALLINT, KW_STRING..KW_STRUCT, KW_TINYINT, KW_UNIONTYPE, KW_WHEN, LESSTHAN..LESSTHANOREQUALTO, MINUS..NOTEQUAL, PLUS, STAR, TILDE}" using multiple alternatives: 1, 3 As a result, alternative(s) 3 were disabled for that input [INFO] [INFO] --- maven-remote-resources-plugin:1.5:process (default) @ hive-exec --- [INFO] [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ hive-exec --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] Copying 2 resources [INFO] Copying 3 resources [INFO] [INFO] --- maven-antrun-plugin:1.7:run (define-classpath) @ hive-exec --- [INFO] Executing tasks main: [INFO] Executed tasks [INFO] [INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ hive-exec --- [INFO] Compiling 1736 source files to /data/hive-ptest/working/apache-svn-trunk-source/ql/target/classes [WARNING] /data/hive-ptest/working/apache-svn-trunk-source/ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinBytesTableContainer.java: Some input files use or override a deprecated API. [WARNING] /data/hive-ptest/working/apache-svn-trunk-source/ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinBytesTableContainer.java: Recompile with -Xlint:deprecation for details. [WARNING] /data/hive-ptest/working/apache-svn-trunk-source/ql/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/ql/plan/api/Query.java: Some input files use unchecked or unsafe operations. [WARNING] /data/hive-ptest/working/apache-svn-trunk-source/ql/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/ql/plan/api/Query.java: Recompile with -Xlint:unchecked for details. [INFO] [INFO] --- build-helper-maven-plugin:1.8:add-test-source (add-test-sources) @ hive-exec --- [INFO] Test Source directory: /data/hive-ptest/working/apache-svn-trunk-source/ql/target/generated-test-sources/java added. [INFO] [INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ hive-exec --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] Copying 4 resources [INFO] Copying 3 resources [INFO] [INFO] --- maven-antrun-plugin:1.7:run (setup-test-dirs) @ hive-exec --- [INFO] Executing tasks main: [mkdir] Created dir: /data/hive-ptest/working/apache-svn-trunk-source/ql/target/tmp [mkdir] Created dir: /data/hive-ptest/working/apache-svn-trunk-source/ql/target/warehouse [mkdir] Created dir: /data/hive-ptest/working/apache-svn-trunk-source/ql/target/tmp/conf [copy] Copying 7 files to /data/hive-ptest/working/apache-svn-trunk-source/ql/target/tmp/conf [INFO] Executed tasks [INFO] [INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ hive-exec --- [INFO] Compiling 209 source files to /data/hive-ptest/working/apache-svn-trunk-source/ql/target/test-classes [INFO] ------------------------------------------------------------- [WARNING] COMPILATION WARNING : [INFO] ------------------------------------------------------------- [WARNING] /data/hive-ptest/working/apache-svn-trunk-source/ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestInputOutputFormat.java: Some input files use or override a deprecated API. [WARNING] /data/hive-ptest/working/apache-svn-trunk-source/ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestInputOutputFormat.java: Recompile with -Xlint:deprecation for details. [WARNING] /data/hive-ptest/working/apache-svn-trunk-source/ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezTask.java: Some input files use unchecked or unsafe operations. [WARNING] /data/hive-ptest/working/apache-svn-trunk-source/ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezTask.java: Recompile with -Xlint:unchecked for details. [INFO] 4 warnings [INFO] ------------------------------------------------------------- [INFO] ------------------------------------------------------------- [ERROR] COMPILATION ERROR : [INFO] ------------------------------------------------------------- [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/ql/src/test/org/apache/hadoop/hive/ql/security/authorization/plugin/sqlstd/TestSQLStdHiveAccessController.java:[47,51] constructor SQLStdHiveAccessController in class org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAccessController cannot be applied to given types; required: org.apache.hadoop.hive.ql.security.authorization.plugin.HiveMetastoreClientFactory,org.apache.hadoop.hive.conf.HiveConf,org.apache.hadoop.hive.ql.security.HiveAuthenticationProvider,org.apache.hadoop.hive.ql.security.authorization.plugin.HiveAuthzSessionContext found: ,org.apache.hadoop.hive.conf.HiveConf,org.apache.hadoop.hive.ql.security.HadoopDefaultAuthenticator reason: actual and formal argument lists differ in length [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/ql/src/test/org/apache/hadoop/hive/ql/security/authorization/plugin/sqlstd/TestSQLStdHiveAccessController.java:[97,51] constructor SQLStdHiveAccessController in class org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAccessController cannot be applied to given types; required: org.apache.hadoop.hive.ql.security.authorization.plugin.HiveMetastoreClientFactory,org.apache.hadoop.hive.conf.HiveConf,org.apache.hadoop.hive.ql.security.HiveAuthenticationProvider,org.apache.hadoop.hive.ql.security.authorization.plugin.HiveAuthzSessionContext found: ,org.apache.hadoop.hive.conf.HiveConf,org.apache.hadoop.hive.ql.security.HadoopDefaultAuthenticator reason: actual and formal argument lists differ in length [INFO] 2 errors [INFO] ------------------------------------------------------------- [INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary: [INFO] [INFO] Hive .............................................. SUCCESS [10.707s] [INFO] Hive Ant Utilities ................................ SUCCESS [6.070s] [INFO] Hive Shims Common ................................. SUCCESS [4.349s] [INFO] Hive Shims 0.20 ................................... SUCCESS [2.548s] [INFO] Hive Shims Secure Common .......................... SUCCESS [3.817s] [INFO] Hive Shims 0.20S .................................. SUCCESS [2.844s] [INFO] Hive Shims 0.23 ................................... SUCCESS [7.176s] [INFO] Hive Shims ........................................ SUCCESS [0.786s] [INFO] Hive Common ....................................... SUCCESS [9.015s] [INFO] Hive Serde ........................................ SUCCESS [11.483s] [INFO] Hive Metastore .................................... SUCCESS [32.625s] [INFO] Hive Query Language ............................... FAILURE [1:08.002s] [INFO] Hive Service ...................................... SKIPPED [INFO] Hive JDBC ......................................... SKIPPED [INFO] Hive Beeline ...................................... SKIPPED [INFO] Hive CLI .......................................... SKIPPED [INFO] Hive Contrib ...................................... SKIPPED [INFO] Hive HBase Handler ................................ SKIPPED [INFO] Hive HCatalog ..................................... SKIPPED [INFO] Hive HCatalog Core ................................ SKIPPED [INFO] Hive HCatalog Pig Adapter ......................... SKIPPED [INFO] Hive HCatalog Server Extensions ................... SKIPPED [INFO] Hive HCatalog Webhcat Java Client ................. SKIPPED [INFO] Hive HCatalog Webhcat ............................. SKIPPED [INFO] Hive HCatalog Streaming ........................... SKIPPED [INFO] Hive HWI .......................................... SKIPPED [INFO] Hive ODBC ......................................... SKIPPED [INFO] Hive Shims Aggregator ............................. SKIPPED [INFO] Hive TestUtils .................................... SKIPPED [INFO] Hive Packaging .................................... SKIPPED [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 2:42.088s [INFO] Finished at: Fri Aug 15 18:08:45 EDT 2014 [INFO] Final Memory: 74M/530M [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:testCompile (default-testCompile) on project hive-exec: Compilation failure: Compilation failure: [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/ql/src/test/org/apache/hadoop/hive/ql/security/authorization/plugin/sqlstd/TestSQLStdHiveAccessController.java:[47,51] constructor SQLStdHiveAccessController in class org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAccessController cannot be applied to given types; [ERROR] required: org.apache.hadoop.hive.ql.security.authorization.plugin.HiveMetastoreClientFactory,org.apache.hadoop.hive.conf.HiveConf,org.apache.hadoop.hive.ql.security.HiveAuthenticationProvider,org.apache.hadoop.hive.ql.security.authorization.plugin.HiveAuthzSessionContext [ERROR] found: ,org.apache.hadoop.hive.conf.HiveConf,org.apache.hadoop.hive.ql.security.HadoopDefaultAuthenticator [ERROR] reason: actual and formal argument lists differ in length [ERROR] /data/hive-ptest/working/apache-svn-trunk-source/ql/src/test/org/apache/hadoop/hive/ql/security/authorization/plugin/sqlstd/TestSQLStdHiveAccessController.java:[97,51] constructor SQLStdHiveAccessController in class org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAccessController cannot be applied to given types; [ERROR] required: org.apache.hadoop.hive.ql.security.authorization.plugin.HiveMetastoreClientFactory,org.apache.hadoop.hive.conf.HiveConf,org.apache.hadoop.hive.ql.security.HiveAuthenticationProvider,org.apache.hadoop.hive.ql.security.authorization.plugin.HiveAuthzSessionContext [ERROR] found: ,org.apache.hadoop.hive.conf.HiveConf,org.apache.hadoop.hive.ql.security.HadoopDefaultAuthenticator [ERROR] reason: actual and formal argument lists differ in length [ERROR] -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn -rf :hive-exec + exit 1 ' {noformat} This message is automatically generated. ATTACHMENT ID: 12662140 > optimize bytes mapjoin hash table read path wrt serialization, at least for common cases > ---------------------------------------------------------------------------------------- > > Key: HIVE-7617 > URL: https://issues.apache.org/jira/browse/HIVE-7617 > Project: Hive > Issue Type: Improvement > Reporter: Sergey Shelukhin > Assignee: Sergey Shelukhin > Attachments: HIVE-7617.01.patch, HIVE-7617.02.patch, HIVE-7617.03.patch, HIVE-7617.04.patch, HIVE-7617.patch, HIVE-7617.prelim.patch, hashmap-wb-fixes.png > > > BytesBytes has table stores keys in the byte array for compact representation, however that means that the straightforward implementation of lookups serializes lookup keys to byte arrays, which is relatively expensive. > We can either shortcut hashcode and compare for common types on read path (integral types which would cover most of the real-world keys), or specialize hashtable and from BytesBytes... create LongBytes, StringBytes, or whatever. First one seems simpler now. -- This message was sent by Atlassian JIRA (v6.2#6252)