hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rajesh Balamohan (Jira)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-24368) Optimise AcidUtils::getAcidFilesForStats for ACID tables
Date Thu, 12 Nov 2020 07:35:00 GMT

    [ https://issues.apache.org/jira/browse/HIVE-24368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17230399#comment-17230399
] 

Rajesh Balamohan commented on HIVE-24368:
-----------------------------------------

Appears that this is handled in master codebase via https://issues.apache.org/jira/browse/HIVE-23791

h1.  

> Optimise AcidUtils::getAcidFilesForStats for ACID tables
> --------------------------------------------------------
>
>                 Key: HIVE-24368
>                 URL: https://issues.apache.org/jira/browse/HIVE-24368
>             Project: Hive
>          Issue Type: Improvement
>          Components: HiveServer2
>            Reporter: Rajesh Balamohan
>            Priority: Major
>
> After insert, hive gathers statistics for ACID table and that becomes expensive over
time, due to number of delta folders and scanning .
> [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java#L2648]
>  
> {noformat}
> public static List<FileStatus> getAcidFilesForStats(
>       	Table table, Path dir, Configuration jc, FileSystem fs) throws IOException {
>       ...
>       Directory acidInfo = AcidUtils.getAcidState(fs, dir, jc, idList, null, false, hdfsDirSnapshots);
>       ...
>       ..+ other calls
>       ...
>       }
>  {noformat}
>  
> Runtime keeps increasing as more deltas are generated. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message