kudu-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Todd Lipcon <t...@cloudera.com>
Subject Re: abnormal high disk I/O rate when upsert into kudu table?
Date Tue, 16 Aug 2016 17:58:55 GMT
Hi Jacky,

Answers inline below

On Tue, Aug 16, 2016 at 8:13 AM, jacky.he@gmail.com <jacky.he@gmail.com>

> Dear Kudu Developers,
> I am a new tester for kudu, our kudu cluster has 3+12 nodes, 3 seperated
> master node and 12 tablet node,
> each node has 128GB memory, and 1 SSD for WAL, 6 1TB SAS for data
> we are using CDH 5.7.0 with impala-kudu 2.7.0 and kudu 0.9.1 parcels,
> we set 16GB memory hard limit for each tablet node.
> Sounds like a good cluster setup. Thanks for providing the details.

> one of our test table is about 80-100 columns and 1 key column, with java
> client, we can insert/upsert into the kudu table about 100,000/s
> the kudu table has 300m rows, and about 300,000 rows update per day, we
> also use java client upsert API to update the rows
> we found the kudu cluster maybe encounter abnormal high disk I/O rate,
> about 1.5-2.0Gb/s, even we just update 1,000~10,000 rows/s
> i would like to know, with our row update frequency, is the cluster high
> disk rate normal or not?

Are you upserts randomly spread across the range of rows in the table? If
so, then when the updates flush, they'll trigger compactions of the updates
and inserted rows into the existing data. This will cause, over time, a
rewrite of the whole table, in order to incorporate the updates.

This background I/O is run by the "maintenance manager". You can visit
http://tablet-server:8050/maintenance-manager to see a dashboard of
currently running maintenance operations such as compactions.

The maintenance manager runs a preset number of threads, so the amount of
background I/O you're experiencing won't increase if you increase the
number of upserts.

I'm curious, is the background I/O causing an issue, or just unexpected?

Todd Lipcon
Software Engineer, Cloudera

View raw message