hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sergio Peña (JIRA) <j...@apache.org>
Subject [jira] [Commented] (HIVE-16014) HiveMetastoreChecker should use hive.metastore.fshandler.threads instead of hive.mv.files.thread for pool size
Date Mon, 27 Feb 2017 19:18:45 GMT

    [ https://issues.apache.org/jira/browse/HIVE-16014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15886350#comment-15886350
] 

Sergio Peña commented on HIVE-16014:
------------------------------------

The change is good [~vihangk1]
+1

> HiveMetastoreChecker should use hive.metastore.fshandler.threads instead of hive.mv.files.thread
for pool size
> --------------------------------------------------------------------------------------------------------------
>
>                 Key: HIVE-16014
>                 URL: https://issues.apache.org/jira/browse/HIVE-16014
>             Project: Hive
>          Issue Type: Improvement
>            Reporter: Vihang Karajgaonkar
>            Assignee: Vihang Karajgaonkar
>         Attachments: HIVE-16014.01.patch, HIVE-16014.02.patch
>
>
> HiveMetastoreChecker uses hive.mv.files.thread configuration value for determining the
pool size as below :
> {noformat}
> private void checkPartitionDirs(Path basePath, Set<Path> allDirs, int maxDepth)
throws IOException, HiveException {
>     ConcurrentLinkedQueue<Path> basePaths = new ConcurrentLinkedQueue<>();
>     basePaths.add(basePath);
>     Set<Path> dirSet = Collections.newSetFromMap(new ConcurrentHashMap<Path,
Boolean>());
>     // Here we just reuse the THREAD_COUNT configuration for
>     // HIVE_MOVE_FILES_THREAD_COUNT
>     int poolSize = conf.getInt(ConfVars.HIVE_MOVE_FILES_THREAD_COUNT.varname, 15);
>     // Check if too low config is provided for move files. 2x CPU is reasonable max count.
>     poolSize = poolSize == 0 ? poolSize : Math.max(poolSize,
>         Runtime.getRuntime().availableProcessors() * 2);
> {noformat}
> msck is commonly used to add the missing partitions for the table from the Filesystem.
In such a case different pool sizes for HMSHandler and HiveMetastoreChecker can affect the
performance. Eg. If {{hive.metastore.fshandler.threads}} is set to a lower value like 15 and
{{hive.mv.files.thread}} is much higher like 100 or vice versa the smaller pool will become
the bottleneck. If would be good to use {{hive.metastore.fshandler.threads}} to size the pool
for HiveMetastoreChecker since the number missing partitions and number of partitions to be
added will most likely be the same. In such a case the performance of the query will be optimum
when both the pool sizes are same.
> Since it is possible to tune both the configs individually it will be very likely that
they may be different. But since there is a strong co-relation between amount of work done
by HiveMetastoreChecker and HiveMetastore.add_partitions call it might be a good idea to use
{{hive.metastore.fshandler.threads}} for pool size instead of {{hive.mv.files.thread}}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message