kudu-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Adar Lieber-Dembo <a...@cloudera.com>
Subject Re: clarification on Partitioning Guidelines and CPU cores
Date Thu, 18 Oct 2018 01:05:34 GMT
The 60 tablets per table per node limit is just at table creation time. You
can create a table that maxes out the number of tablets, then add more
range partitions afterwards.

On Wed, Oct 17, 2018 at 6:00 PM Boris Tyukin <boris@boristyukin.com> wrote:

> thanks for replying, Adar. Did some math and in our case we are hitting
> another Kudu limit - 60 tablets per node. We use high density nodes with 2
> 24-core CPUs so we have 88 hyperthreaded cores total per node or 88*24=2112
> cores total. But I cannot create more than 60*24=1440 tablets per table.
> Looks like my tablets for the largest table will be around 8-10Gb in size.
> Should I be worried since recommendation is to keep tablets about 1Gb in
> size?
> On Wed, Oct 17, 2018 at 8:06 PM Adar Lieber-Dembo <adar@cloudera.com>
> wrote:
>> Hi Boris,
>> > Also, when they say tablets - I assume this is before replication? so
>> in reality, it is number of nodes x cpu cores / replication factor? If this
>> is the case, it is not looking good...
>> No, I think this is post-replication. The underlying assumption is
>> that you want to maximize parallelism for large tables, and since
>> Impala only uses one read thread per tablet, that means ensuring the
>> number of tablets is close or equal to the overall number of cores.
>> However, during a scan Impala will choose one of the tablet's replicas
>> to read from, so you don't need to "reserve" a core for the other
>> replicas.
>> >> can someone clarify if this recommendation below - does it mean
>> physical or hyper-threaded CPU cores? quite a big difference...
>> I think this refers to hyper-threaded CPU cores (i.e. a CPU unit
>> capable of executing an OS thread). But I'd be curious to hear if your
>> workload is substantially more or less performant either way.

View raw message