accumulo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Russ Weeks <rwe...@newbrightidea.com>
Subject Re: Question about configuring the linux niceness of tablet servers?
Date Tue, 18 Aug 2015 21:40:36 GMT
Hi, Jeff,

Yes, we have had mapreduce jobs "lock out" the tserver so long that the
> master removes them from its pool for not responding. But also when doing a
> query from another service while the mapreduce jobs are maxed out I want to
> make sure the tserver takes priority.
>

I've observed this too! Zookeeper connection loss on the tablet server,
right? The problem seems to correlate with times of high load avg (avg > #
cores) on the tablet server. Our fix was to reduce the CPU and RAM
available to YARN. Fewer active containers leads to less CPU contention.
Less CPU contention can also improve throughput on the MR job, so if you
hit the sweet spot it might even improve performance. Definitely interested
to hear alternative solutions though.

Regards,
-Russ


>
>> > However, if you do this, I think you actually want HDFS to -2, Accumulo
>> to -1,
>> > and Mapreduce tasks 0 (you'll have to edit the system security limits
>> to allow
>> > niceness below 0 for non-root users).
>> > Otherwise, you're lowering the priority, not increasing it (and
>> lowering it
>> > below other, normal user processes, since the default niceness is 0).
>>
>
> Agreed. That is what I wanted to do originally but it requires much more
> reconfiguration. Ideally all OS root processes currently at 0 get reniced
> to -2, HDFS to -2, Accumulo to -2, and mapreduce to -1 and all other user
> processes stay at 0. Given the amount of threading a datanode does by
> default it could lockout a tserver if its priority was lower, so I think
> the HDFS and Accumulo should have equal priority.
>
>

Mime
View raw message