hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
Subject [jira] [Updated] (HIVE-21210) CombineHiveInputFormat Thread Pool Sizing
Date Tue, 05 Feb 2019 02:45:00 GMT


BELUGA BEHR updated HIVE-21210:
    Status: Patch Available  (was: Open)

> CombineHiveInputFormat Thread Pool Sizing
> -----------------------------------------
>                 Key: HIVE-21210
>                 URL:
>             Project: Hive
>          Issue Type: Improvement
>    Affects Versions: 4.0.0, 3.2.0
>            Reporter: BELUGA BEHR
>            Assignee: BELUGA BEHR
>            Priority: Major
>         Attachments: HIVE-21210.1.patch, HIVE-21210.2.patch
> Threadpools.
> Hive uses threadpools in several different places and each implementation is a little
different and requires different configurations. I think that Hive needs to reign in and standardize
the way that threadpools are used and threadpools should scale automatically without manual
configuration. At any given time, there are many hundreds of threads running in the HS2 as
the number of simultaneous connections increases and they surely cause contention with one-another.
> Here is an example:
> {code:java|}
>   // max number of threads we can use to check non-combinable paths
>   private static final int MAX_CHECK_NONCOMBINABLE_THREAD_NUM = 50;
>   private static final int DEFAULT_NUM_PATH_PER_THREAD = 100;
> {code}
> When building the splits for a MR job, there are up to 50 threads running per query and
there is not much scaling here, it's simply 1 thread : 100 files ratio.  This implies that
to process 5000 files, there are 50 threads, after that, 50 threads are still used. Many Hive
jobs these days involve more than 5000 files so it's not scaling well on bigger sizes.
> This is not configurable (even manually), it doesn't change when the hardware specs increase,
and 50 threads seems like a lot when a service must support up to 80 connections:
> []
> Not to mention, I have never seen a scenario where HS2 is running on a host all by itself
and has the entire system dedicated to it. Therefore it should be more friendly and spin up
fewer threads.
> I am attaching a patch here that provides a few features:
>  * Common module that produces {{ExecutorService}} which caps the number of threads it
spins up at the number of processors a host has. Keep in mind that a class may submit as much
work units ({{Callables}} as they would like, but the number of threads in the pool is capped.
>  * Common module for partitioning work. That is, allow for a generic framework for dividing
work into partitions (i.e. batches)
>  * Modify {{CombineHiveInputFormat}} to take advantage of both modules, performing its
same duties in a more Java OO way that is currently implemented
>  * Add a partitioning (batching) implementation that enforces partitioning of a {{Collection}}
based on the natural log of the {{Collection}} size so that it scales more slowly than a simple
1:100 ratio.
>  * Simplify unit test code for {{CombineHiveInputFormat}}
> My hope is to introduce these tools to {{CombineHiveInputFormat}} and then to drop it
into other places.  One of the things I will introduce here is a "direct thread" {{ExecutorService}}
so that even if there is a configuration for a thread pool to be disabled, it will still use
an {{ExecutorService}} so that the project can avoid logic like "if this function is services
by a thread pool, use a {{ExecutorService}} (and remember to close it later!) otherwise, create
a single thread" so that things like [HIVE-16949] can be avoided in the future.  Everything
will just use an {{ExecutorService}}.

This message was sent by Atlassian JIRA

View raw message