kudu-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Todd Lipcon <t...@cloudera.com>
Subject Re: Imroving the insert peformance with INSERT INTO SELECT - gFlagfile
Date Mon, 30 May 2016 18:14:47 GMT
Hi Amit

Answers below

On Sun, May 29, 2016 at 11:37 AM, Amit Adhau <amit.adhau@globant.com> wrote:

> Hi,
>
> What is the significance of using below gflags, which can help in imroving
> the insert peformance with INSERT INTO SELECT clause.
>
> --num_tablets_to_open_simultaneously=8
>

This only affects the startup time of a tablet server, and should not
affect the insert performance at all.


> --scanner_batch_size_rows=1000
>

This only affects the read performance. I've seen it have a noticeable
affect at times, but can also cause some memory management issues with
wider tables - that's why the default is 100. It won't affect write
performance at all.


>
> and maintenance_manager_num_threads(Kudu Tablet Server Maintenance
> Threads) in cloudera manager.
>
>
This could improve write performance, since it increases the number of
threads available to perform compaction and flushes. Assuming your hardware
looks like typical Hadoop nodes (eg 10-12 disks, 8-16 cores), I would try
setting it to 4 as a starting point.


> As playing with these configs, gives most of the time errors like "Timed
> out: Failed to write batch of ops to tablet " OR "Illegal state: Tablet not
> RUNNING: NOT_STARTED"
>
>
Those errors seem to indicate you are probably trying to perform
reads/writes while the servers are still in the process of starting up.
Maybe you are not giving the cluster enough time to fully restart before
you are restarting the workload after changing the tuning?

-Todd

Mime
View raw message