mesos-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Benjamin Mahler (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MESOS-920) Set GLOG_drop_log_memory=false in environment prior to logging initialization.
Date Tue, 08 Aug 2017 02:48:00 GMT

    [ https://issues.apache.org/jira/browse/MESOS-920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16117765#comment-16117765
] 

Benjamin Mahler commented on MESOS-920:
---------------------------------------

Looks like a fix was upstreamed, possibly making it safe to use use the drop_log_memory flag:
https://github.com/google/glog/pull/145

> Set GLOG_drop_log_memory=false in environment prior to logging initialization.
> ------------------------------------------------------------------------------
>
>                 Key: MESOS-920
>                 URL: https://issues.apache.org/jira/browse/MESOS-920
>             Project: Mesos
>          Issue Type: Improvement
>    Affects Versions: 0.15.0, 0.16.0
>            Reporter: Benjamin Mahler
>            Assignee: Kapil Arya
>            Priority: Blocker
>              Labels: tech-debt
>             Fix For: 0.24.2, 0.25.1, 0.26.1, 0.27.0
>
>
> We've observed issues where the masters are slow to respond. Two perf traces collected
while the masters were slow to respond:
> {noformat}
>  25.84%  [kernel]                [k] default_send_IPI_mask_sequence_phys
>  20.44%  [kernel]                [k] native_write_msr_safe
>   4.54%  [kernel]                [k] _raw_spin_lock
>   2.95%  libc-2.5.so             [.] _int_malloc
>   1.82%  libc-2.5.so             [.] malloc
>   1.55%  [kernel]                [k] apic_timer_interrupt
>   1.36%  libc-2.5.so             [.] _int_free
> {noformat}
> {noformat}
>  29.03%  [kernel]                [k] default_send_IPI_mask_sequence_phys
>   9.64%  [kernel]                [k] _raw_spin_lock
>   7.38%  [kernel]                [k] native_write_msr_safe
>   2.43%  libc-2.5.so             [.] _int_malloc
>   2.05%  libc-2.5.so             [.] _int_free
>   1.67%  [kernel]                [k] apic_timer_interrupt
>   1.58%  libc-2.5.so             [.] malloc
> {noformat}
> These have been found to be attributed to the posix_fadvise calls made by glog. We can
disable these via the environment:
> {noformat}
> GLOG_DEFINE_bool(drop_log_memory, true, "Drop in-memory buffers of log contents. "
>                  "Logs can grow very quickly and they are rarely read before they "
>                  "need to be evicted from memory. Instead, drop them from memory "
>                  "as soon as they are flushed to disk.");
> {noformat}
> {code}
>     if (FLAGS_drop_log_memory) {
>       if (file_length_ >= logging::kPageSize) {
>         // don't evict the most recent page
>         uint32 len = file_length_ & ~(logging::kPageSize - 1);
>         posix_fadvise(fileno(file_), 0, len, POSIX_FADV_DONTNEED);
>       }
>     }
> {code}
> We should set GLOG_drop_log_memory=false prior to making our call to google::InitGoogleLogging,
to avoid others running into this issue.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message