hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Colin Patrick McCabe (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HADOOP-12975) Add jitter to CachingGetSpaceUsed's thread
Date Thu, 14 Apr 2016 20:08:25 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-12975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15241805#comment-15241805
] 

Colin Patrick McCabe edited comment on HADOOP-12975 at 4/14/16 8:08 PM:
------------------------------------------------------------------------

bq. But a percentage is chosen as it makes the jitter scale with anyone who changes du periods.
If it's a set number then someone with a refresh period of days won't get any benefit from
the jitter.

Hmm.  It seems like a fixed amount of jitter still provides a benefit, even to someone with
a longer refresh interval.  Let's say my refresh period is 7 days.  At the end of that, I
would still appreciate having my DU processes launch at slightly different times on the 7th
day, rather than all launching at once.

My concern with varying based on a percentage is that there will be enormous variations in
how long different volumes go between DU operations, when longer refresh intervals are in
use.  Like if I have a 7 day period and one volume refreshes after 3.5 days, and the other
waits for the full 7 days, that's quite a variation.  Similarly, if our period is short --
like 1 hour-- having some datanodes refresh after only 30 minutes seems unwelcome.  That's
why I suggested a fixed jitter amount, to be configured by the sysadmin.

I don't feel very strongly about this, though, so if you want to make it percentage-based,
that's fine too.  As long as it's configurable and the defaults are reasonable.  I definitely
think that a maximum jitter percentage of 0.15 or 0.20 seems more reasonable than 0.5.


was (Author: cmccabe):
bq. But a percentage is chosen as it makes the jitter scale with anyone who changes du periods.
If it's a set number then someone with a refresh period of days won't get any benefit from
the jitter.

Hmm.  It seems like a fixed amount of jitter still provides a benefit, even to someone with
a longer refresh interval.  Let's say my refresh period is 7 days.  At the end of that, I
would still appreciate having my DU processes launch at slightly different times on the 7th
day, rather than all launching at once.

My concern with varying based on a percentage is that there will be enormous variations in
how long different volumes go between DU operations, when longer refresh intervals are in
use.  Like if I have a 7 day period and one volume refreshes after 3.5 days, and the other
waits for the full 7 days, that's quite a variation.  Similarly, if our period is short --
like 1 hour-- having some datanodes refresh after only 30 minutes seems unwelcome.  That's
why I suggested a fixed jitter amount, to be configured by the sysadmin.

I don't feel very strongly about this, though, so if you want to make it percentage-based,
that's fine too.  As long as it's configurable and the defaults are reasonable.

> Add jitter to CachingGetSpaceUsed's thread
> ------------------------------------------
>
>                 Key: HADOOP-12975
>                 URL: https://issues.apache.org/jira/browse/HADOOP-12975
>             Project: Hadoop Common
>          Issue Type: Sub-task
>    Affects Versions: 2.9.0
>            Reporter: Elliott Clark
>            Assignee: Elliott Clark
>         Attachments: HADOOP-12975v0.patch, HADOOP-12975v1.patch, HADOOP-12975v2.patch
>
>
> Running DU across lots of disks is very expensive and running all of the processes at
the same time creates a noticeable IO spike. We should add some jitter.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message