kudu-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jason Heo <jason.heo....@gmail.com>
Subject Re: Load is high on the Kudu dedicated node.
Date Wed, 15 Mar 2017 01:01:12 GMT
Hi Todd.

Thank you for your kind reply.

I'll try your recommendation and look forward for next releases.

Thanks.

2017-03-15 0:26 GMT+09:00 Todd Lipcon <todd@cloudera.com>:

> Hi Jason,
>
> By "bulk indexing only" you mean you are loading data with a high rate of
> inserts?
>
> It seems that there is a lot of contention on the memory trackers.
> https://issues.apache.org/jira/browse/KUDU-1502 is one JIRA where I noted
> this was the case. If that's the culprit, I would look into the following:
>
> - try to change your insert pattern so that it is more sequential in
> nature (random inserts will cause a lot of block cache lookups to check for
> duplicate keys)
> - if you have RAM available, increase both the block cache capacity and
> the server's memory limit accordingly, so that the bloom lookups will hit
> Kudu's cache instead of having to go to the operating system cache.
>
> Aside from that, we'll be spending some time on improving performance of
> write-heavy workloads in upcoming releases, and I think fixing this
> MemTracker contention will be one of the issues tackled.
>
> In case the above isn't the issue, do you think you could use 'perf record
> -g -a' and generate a flame graph? http://www.
> brendangregg.com/FlameGraphs/cpuflamegraphs.html
>
> -Todd
>
> On Tue, Mar 14, 2017 at 6:14 AM, Jason Heo <jason.heo.sde@gmail.com>
> wrote:
>
>> Hi. I'm experiencing high load and high cpu usage. Kudu is running on 5
>> kudu dedicated nodes. 2 nodes' load is 40, while 3 nodes' load is 15.
>>
>> Here is the output of `perf record -a & perf report` during bulk indexing
>> only operation.
>>
>> http://imgur.com/8lz1CRk
>>
>> I'm wondering this is a reasonable situation.
>>
>> I'm using Kudu on CDH 5.10
>>
>> Thanks.
>>
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>

Mime
View raw message