hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hive QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-13884) Disallow queries fetching more than a configured number of partitions in PartitionPruner
Date Mon, 06 Jun 2016 04:02:59 GMT

    [ https://issues.apache.org/jira/browse/HIVE-13884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15316165#comment-15316165
] 

Hive QA commented on HIVE-13884:
--------------------------------



Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12808078/HIVE-13884.1.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 13 failed/errored test(s), 10220 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_table_stats
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_insert_values_orig_table_use_metadata
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_limit_partition_metadataonly
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_constprog_partitioner
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_limit_partition
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_limit_partition_metadataonly
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/13/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/13/console
Test logs: http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-13/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 13 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12808078 - PreCommit-HIVE-MASTER-Build

> Disallow queries fetching more than a configured number of partitions in PartitionPruner
> ----------------------------------------------------------------------------------------
>
>                 Key: HIVE-13884
>                 URL: https://issues.apache.org/jira/browse/HIVE-13884
>             Project: Hive
>          Issue Type: Improvement
>            Reporter: Mohit Sabharwal
>            Assignee: Sergio Peña
>         Attachments: HIVE-13884.1.patch
>
>
> Currently the PartitionPruner requests either all partitions or partitions based on filter
expression. In either scenarios, if the number of partitions accessed is large there can be
significant memory pressure at the HMS server end.
> We already have a config {{hive.limit.query.max.table.partition}} that enforces limits
on number of partitions that may be scanned per operator. But this check happens after the
PartitionPruner has already fetched all partitions.
> We should add an option at PartitionPruner level to disallow queries that attempt to
access number of partitions beyond a configurable limit.
> Note that {{hive.mapred.mode=strict}} disallow queries without a partition filter in
PartitionPruner, but this check accepts any query with a pruning condition, even if partitions
fetched are large. In multi-tenant environments, admins could use more control w.r.t. number
of partitions allowed based on HMS memory capacity.
> One option is to have PartitionPruner first fetch the partition names (instead of partition
specs) and throw an exception if number of partitions exceeds the configured value. Otherwise,
fetch the partition specs.
> Looks like the existing {{listPartitionNames}} call could be used if extended to take
partition filter expressions like {{getPartitionsByExpr}} call does.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message