hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gunther Hagleitner (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-6492) limit partition number involved in a table scan
Date Fri, 07 Mar 2014 07:40:42 GMT

    [ https://issues.apache.org/jira/browse/HIVE-6492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13923657#comment-13923657
] 

Gunther Hagleitner commented on HIVE-6492:
------------------------------------------

Looked at it some more. Finally get what you were saying about metadata only. I think we can
go two ways:

a) use patch as is. since metadata only still launches a job with potentially a lot of tasks
(one split per file it seems).
b) fix it like you were, but change the variable to count files not partitions (you don't
have access to partitions anymore in the lower layers.) and move the code to driver so it
works for both mr and tez.

[~selinazh] - what works better for you? since i sent you on this wild goose chase, i can
take another shot at updating it...

> limit partition number involved in a table scan
> -----------------------------------------------
>
>                 Key: HIVE-6492
>                 URL: https://issues.apache.org/jira/browse/HIVE-6492
>             Project: Hive
>          Issue Type: New Feature
>          Components: Query Processor
>    Affects Versions: 0.12.0
>            Reporter: Selina Zhang
>             Fix For: 0.13.0
>
>         Attachments: HIVE-6492.1.patch.txt, HIVE-6492.2.patch.txt, HIVE-6492.3.patch.txt,
HIVE-6492.4.patch.txt, HIVE-6492.4.patch_suggestion, HIVE-6492.5.patch.txt
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> To protect the cluster, a new configure variable "hive.limit.query.max.table.partition"
is added to hive configuration to
> limit the table partitions involved in a table scan. 
> The default value will be set to -1 which means there is no limit by default. 
> This variable will not affect "metadata only" query.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message