hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hive QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-13884) Disallow queries fetching more than a configured number of partitions in PartitionPruner
Date Mon, 13 Jun 2016 19:23:30 GMT

    [ https://issues.apache.org/jira/browse/HIVE-13884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15327997#comment-15327997
] 

Hive QA commented on HIVE-13884:
--------------------------------



Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12809966/HIVE-13884.2.patch

{color:red}ERROR:{color} -1 due to build exiting with an error

Test results: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/115/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/115/console
Test logs: http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-115/

Messages:
{noformat}
**** This message was trimmed, see log for full details ****
[INFO] 
[INFO] --- maven-antrun-plugin:1.7:run (setup-test-dirs) @ hive-service-rpc ---
[INFO] Executing tasks

main:
    [mkdir] Created dir: /data/hive-ptest/working/apache-github-source-source/service-rpc/target/tmp
    [mkdir] Created dir: /data/hive-ptest/working/apache-github-source-source/service-rpc/target/warehouse
    [mkdir] Created dir: /data/hive-ptest/working/apache-github-source-source/service-rpc/target/tmp/conf
     [copy] Copying 15 files to /data/hive-ptest/working/apache-github-source-source/service-rpc/target/tmp/conf
[INFO] Executed tasks
[INFO] 
[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ hive-service-rpc
---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-surefire-plugin:2.16:test (default-test) @ hive-service-rpc ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:2.2:jar (default-jar) @ hive-service-rpc ---
[INFO] Building jar: /data/hive-ptest/working/apache-github-source-source/service-rpc/target/hive-service-rpc-2.2.0-SNAPSHOT.jar
[INFO] 
[INFO] --- maven-site-plugin:3.3:attach-descriptor (attach-descriptor) @ hive-service-rpc
---
[INFO] 
[INFO] --- maven-jar-plugin:2.2:test-jar (default) @ hive-service-rpc ---
[INFO] Building jar: /data/hive-ptest/working/apache-github-source-source/service-rpc/target/hive-service-rpc-2.2.0-SNAPSHOT-tests.jar
[INFO] 
[INFO] --- maven-install-plugin:2.4:install (default-install) @ hive-service-rpc ---
[INFO] Installing /data/hive-ptest/working/apache-github-source-source/service-rpc/target/hive-service-rpc-2.2.0-SNAPSHOT.jar
to /data/hive-ptest/working/maven/org/apache/hive/hive-service-rpc/2.2.0-SNAPSHOT/hive-service-rpc-2.2.0-SNAPSHOT.jar
[INFO] Installing /data/hive-ptest/working/apache-github-source-source/service-rpc/pom.xml
to /data/hive-ptest/working/maven/org/apache/hive/hive-service-rpc/2.2.0-SNAPSHOT/hive-service-rpc-2.2.0-SNAPSHOT.pom
[INFO] Installing /data/hive-ptest/working/apache-github-source-source/service-rpc/target/hive-service-rpc-2.2.0-SNAPSHOT-tests.jar
to /data/hive-ptest/working/maven/org/apache/hive/hive-service-rpc/2.2.0-SNAPSHOT/hive-service-rpc-2.2.0-SNAPSHOT-tests.jar
[INFO]                                                                         
[INFO] ------------------------------------------------------------------------
[INFO] Building Hive Serde 2.2.0-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO] 
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ hive-serde ---
[INFO] Deleting /data/hive-ptest/working/apache-github-source-source/serde/target
[INFO] Deleting /data/hive-ptest/working/apache-github-source-source/serde (includes = [datanucleus.log,
derby.log], excludes = [])
[INFO] 
[INFO] --- maven-enforcer-plugin:1.3.1:enforce (enforce-no-snapshots) @ hive-serde ---
[INFO] 
[INFO] --- build-helper-maven-plugin:1.8:add-source (add-source) @ hive-serde ---
[INFO] Source directory: /data/hive-ptest/working/apache-github-source-source/serde/src/gen/protobuf/gen-java
added.
[INFO] Source directory: /data/hive-ptest/working/apache-github-source-source/serde/src/gen/thrift/gen-javabean
added.
[INFO] 
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @ hive-serde ---
[INFO] 
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ hive-serde ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /data/hive-ptest/working/apache-github-source-source/serde/src/main/resources
[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-antrun-plugin:1.7:run (define-classpath) @ hive-serde ---
[INFO] Executing tasks

main:
[INFO] Executed tasks
[INFO] 
[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ hive-serde ---
[INFO] Compiling 414 source files to /data/hive-ptest/working/apache-github-source-source/serde/target/classes
[WARNING] /data/hive-ptest/working/apache-github-source-source/serde/src/java/org/apache/hadoop/hive/serde2/AbstractSerDe.java:
Some input files use or override a deprecated API.
[WARNING] /data/hive-ptest/working/apache-github-source-source/serde/src/java/org/apache/hadoop/hive/serde2/AbstractSerDe.java:
Recompile with -Xlint:deprecation for details.
[WARNING] /data/hive-ptest/working/apache-github-source-source/serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/primitive/AbstractPrimitiveLazyObjectInspector.java:
Some input files use unchecked or unsafe operations.
[WARNING] /data/hive-ptest/working/apache-github-source-source/serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/primitive/AbstractPrimitiveLazyObjectInspector.java:
Recompile with -Xlint:unchecked for details.
[INFO] 
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ hive-serde ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 2 resources
[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-antrun-plugin:1.7:run (setup-test-dirs) @ hive-serde ---
[INFO] Executing tasks

main:
    [mkdir] Created dir: /data/hive-ptest/working/apache-github-source-source/serde/target/tmp
    [mkdir] Created dir: /data/hive-ptest/working/apache-github-source-source/serde/target/warehouse
    [mkdir] Created dir: /data/hive-ptest/working/apache-github-source-source/serde/target/tmp/conf
     [copy] Copying 15 files to /data/hive-ptest/working/apache-github-source-source/serde/target/tmp/conf
[INFO] Executed tasks
[INFO] 
[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ hive-serde ---
[INFO] Compiling 55 source files to /data/hive-ptest/working/apache-github-source-source/serde/target/test-classes
[WARNING] /data/hive-ptest/working/apache-github-source-source/serde/src/test/org/apache/hadoop/hive/serde2/lazy/TestLazySimpleSerDe.java:
Some input files use or override a deprecated API.
[WARNING] /data/hive-ptest/working/apache-github-source-source/serde/src/test/org/apache/hadoop/hive/serde2/lazy/TestLazySimpleSerDe.java:
Recompile with -Xlint:deprecation for details.
[WARNING] /data/hive-ptest/working/apache-github-source-source/serde/src/test/org/apache/hadoop/hive/serde2/lazy/TestLazyArrayMapStruct.java:
Some input files use unchecked or unsafe operations.
[WARNING] /data/hive-ptest/working/apache-github-source-source/serde/src/test/org/apache/hadoop/hive/serde2/lazy/TestLazyArrayMapStruct.java:
Recompile with -Xlint:unchecked for details.
[INFO] 
[INFO] --- maven-surefire-plugin:2.16:test (default-test) @ hive-serde ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:2.2:jar (default-jar) @ hive-serde ---
[INFO] Building jar: /data/hive-ptest/working/apache-github-source-source/serde/target/hive-serde-2.2.0-SNAPSHOT.jar
[INFO] 
[INFO] --- maven-site-plugin:3.3:attach-descriptor (attach-descriptor) @ hive-serde ---
[INFO] 
[INFO] --- maven-install-plugin:2.4:install (default-install) @ hive-serde ---
[INFO] Installing /data/hive-ptest/working/apache-github-source-source/serde/target/hive-serde-2.2.0-SNAPSHOT.jar
to /data/hive-ptest/working/maven/org/apache/hive/hive-serde/2.2.0-SNAPSHOT/hive-serde-2.2.0-SNAPSHOT.jar
[INFO] Installing /data/hive-ptest/working/apache-github-source-source/serde/pom.xml to /data/hive-ptest/working/maven/org/apache/hive/hive-serde/2.2.0-SNAPSHOT/hive-serde-2.2.0-SNAPSHOT.pom
[INFO]                                                                         
[INFO] ------------------------------------------------------------------------
[INFO] Building Hive Metastore 2.2.0-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO] 
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ hive-metastore ---
[INFO] Deleting /data/hive-ptest/working/apache-github-source-source/metastore/target
[INFO] Deleting /data/hive-ptest/working/apache-github-source-source/metastore (includes =
[datanucleus.log, derby.log], excludes = [])
[INFO] 
[INFO] --- maven-enforcer-plugin:1.3.1:enforce (enforce-no-snapshots) @ hive-metastore ---
[INFO] 
[INFO] --- build-helper-maven-plugin:1.8:add-source (add-source) @ hive-metastore ---
[INFO] Source directory: /data/hive-ptest/working/apache-github-source-source/metastore/src/model
added.
[INFO] Source directory: /data/hive-ptest/working/apache-github-source-source/metastore/src/gen/thrift/gen-javabean
added.
[INFO] Source directory: /data/hive-ptest/working/apache-github-source-source/metastore/src/gen/protobuf/gen-java
added.
[INFO] 
[INFO] --- antlr3-maven-plugin:3.4:antlr (default) @ hive-metastore ---
[INFO] ANTLR: Processing source directory /data/hive-ptest/working/apache-github-source-source/metastore/src/java
ANTLR Parser Generator  Version 3.4
org/apache/hadoop/hive/metastore/parser/Filter.g
[INFO] 
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @ hive-metastore ---
[INFO] 
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ hive-metastore ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 1 resource
[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-antrun-plugin:1.7:run (define-classpath) @ hive-metastore ---
[INFO] Executing tasks

main:
[INFO] Executed tasks
[INFO] 
[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ hive-metastore ---
[INFO] Compiling 317 source files to /data/hive-ptest/working/apache-github-source-source/metastore/target/classes
[INFO] -------------------------------------------------------------
[ERROR] COMPILATION ERROR : 
[INFO] -------------------------------------------------------------
[ERROR] /data/hive-ptest/working/apache-github-source-source/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java:[3134,107]
';' expected
[INFO] 1 error
[INFO] -------------------------------------------------------------
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] Hive .............................................. SUCCESS [2.855s]
[INFO] Hive Shims Common ................................. SUCCESS [4.533s]
[INFO] Hive Shims 0.23 ................................... SUCCESS [2.459s]
[INFO] Hive Shims Scheduler .............................. SUCCESS [0.699s]
[INFO] Hive Shims ........................................ SUCCESS [0.666s]
[INFO] Hive Storage API .................................. SUCCESS [1.399s]
[INFO] Hive ORC .......................................... SUCCESS [4.653s]
[INFO] Hive Common ....................................... SUCCESS [6.177s]
[INFO] Hive Service RPC .................................. SUCCESS [3.386s]
[INFO] Hive Serde ........................................ SUCCESS [4.753s]
[INFO] Hive Metastore .................................... FAILURE [4.618s]
[INFO] Hive Ant Utilities ................................ SKIPPED
[INFO] Hive Llap Common .................................. SKIPPED
[INFO] Hive Llap Client .................................. SKIPPED
[INFO] Hive Llap Tez ..................................... SKIPPED
[INFO] Spark Remote Client ............................... SKIPPED
[INFO] Hive Query Language ............................... SKIPPED
[INFO] Hive Llap Server .................................. SKIPPED
[INFO] Hive Service ...................................... SKIPPED
[INFO] Hive Accumulo Handler ............................. SKIPPED
[INFO] Hive JDBC ......................................... SKIPPED
[INFO] Hive Beeline ...................................... SKIPPED
[INFO] Hive CLI .......................................... SKIPPED
[INFO] Hive Contrib ...................................... SKIPPED
[INFO] Hive HBase Handler ................................ SKIPPED
[INFO] Hive HCatalog ..................................... SKIPPED
[INFO] Hive HCatalog Core ................................ SKIPPED
[INFO] Hive HCatalog Pig Adapter ......................... SKIPPED
[INFO] Hive HCatalog Server Extensions ................... SKIPPED
[INFO] Hive HCatalog Webhcat Java Client ................. SKIPPED
[INFO] Hive HCatalog Webhcat ............................. SKIPPED
[INFO] Hive HCatalog Streaming ........................... SKIPPED
[INFO] Hive HPL/SQL ...................................... SKIPPED
[INFO] Hive HWI .......................................... SKIPPED
[INFO] Hive Llap External Client ......................... SKIPPED
[INFO] Hive Shims Aggregator ............................. SKIPPED
[INFO] Hive TestUtils .................................... SKIPPED
[INFO] Hive Packaging .................................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 37.086s
[INFO] Finished at: Mon Jun 13 19:25:17 GMT 2016
[INFO] Final Memory: 81M/780M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:compile
(default-compile) on project hive-metastore: Compilation failure
[ERROR] /data/hive-ptest/working/apache-github-source-source/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java:[3134,107]
';' expected
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following
articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <goals> -rf :hive-metastore
'
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12809966 - PreCommit-HIVE-MASTER-Build

> Disallow queries fetching more than a configured number of partitions in PartitionPruner
> ----------------------------------------------------------------------------------------
>
>                 Key: HIVE-13884
>                 URL: https://issues.apache.org/jira/browse/HIVE-13884
>             Project: Hive
>          Issue Type: Improvement
>            Reporter: Mohit Sabharwal
>            Assignee: Sergio Peña
>         Attachments: HIVE-13884.1.patch, HIVE-13884.2.patch
>
>
> Currently the PartitionPruner requests either all partitions or partitions based on filter
expression. In either scenarios, if the number of partitions accessed is large there can be
significant memory pressure at the HMS server end.
> We already have a config {{hive.limit.query.max.table.partition}} that enforces limits
on number of partitions that may be scanned per operator. But this check happens after the
PartitionPruner has already fetched all partitions.
> We should add an option at PartitionPruner level to disallow queries that attempt to
access number of partitions beyond a configurable limit.
> Note that {{hive.mapred.mode=strict}} disallow queries without a partition filter in
PartitionPruner, but this check accepts any query with a pruning condition, even if partitions
fetched are large. In multi-tenant environments, admins could use more control w.r.t. number
of partitions allowed based on HMS memory capacity.
> One option is to have PartitionPruner first fetch the partition names (instead of partition
specs) and throw an exception if number of partitions exceeds the configured value. Otherwise,
fetch the partition specs.
> Looks like the existing {{listPartitionNames}} call could be used if extended to take
partition filter expressions like {{getPartitionsByExpr}} call does.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message