hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Xianyin Xin (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-4177) yarn.util.Clock should not be used to time a duration or time interval
Date Mon, 21 Sep 2015 09:46:05 GMT

    [ https://issues.apache.org/jira/browse/YARN-4177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14900450#comment-14900450

Xianyin Xin commented on YARN-4177:

Hi [~stevel@apache.org], thanks for your comment. I've read your post and did some investgations
on this.
1.Inconsistent across cores, hence non-monotonic on reads, especially reads likely to trigger
thread suspend/resume (anything with sleep(), wait(), IO, accessing synchronized data under
This was once a bug on some old OSs, but it seems not a problem on Linux newer than 2.6 or
windows newer than XP SP2, if i understand your comment correctly. See http://stackoverflow.com/questions/510462/is-system-nanotime-completely-useless,
and the refered https://blogs.oracle.com/dholmes/entry/inside_the_hotspot_vm_clocks.
2.Not actually monotonic.
Can you explain in detail? As a reference, there're some discussion on clock_gettime which
nanoTime depends in http://stackoverflow.com/questions/4943733/is-clock-monotonic-process-or-thread-specific?rq=1,
especially in the second post that has 4 supports.
3.Achieving a consistency by querying heavyweight counters with possible longer function execution
time and lower granularity than the wall clock.
That is: modern NUMA, multi-socket servers are essentially multiple computers wired together,
and we have a term for that: distributed system
You mean achieving a consistent time across nodes in a cluster? I think the monotonic time
we plan to offer should be limited to node-local. It's hard to make it cluster wide. 
I've known for a long time that CPU frequency could change its rate
I remembered Linux higher than 2.6.18 takes some measures to overcome this problem. http://stackoverflow.com/questions/510462/is-system-nanotime-completely-useless#comment40382219_510940
has little discussion.

> yarn.util.Clock should not be used to time a duration or time interval
> ----------------------------------------------------------------------
>                 Key: YARN-4177
>                 URL: https://issues.apache.org/jira/browse/YARN-4177
>             Project: Hadoop YARN
>          Issue Type: Bug
>            Reporter: Xianyin Xin
>            Assignee: Xianyin Xin
>         Attachments: YARN-4177.001.patch, YARN-4177.002.patch
> There're many places uses Clock to time intervals, which is dangerous as commented by
[~stevel@apache.org] in HADOOP-12409. Instead, we should use hadoop.util.Timer#monotonicNow()
to get monotonic time. Or we could provide a MonotonicClock in yarn.util considering the consistency
of code.

This message was sent by Atlassian JIRA

View raw message