subversion-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Charlie Smurthwaite <char...@atechmedia.com>
Subject Re: Kernel CPU load on Apache/2.4.10 with SVN 1.8.10 (r1615264)
Date Wed, 17 Dec 2014 14:29:04 GMT

On 17/12/14 09:25, Charlie Smurthwaite wrote:
> Hi,
>
> I am running an SVN HTTP server using Apache/2.4.10 with SVN 1.8.10 
> (r1615264) and I am often seeing very high kernel CPU load.
>
> The CPU time seems to be consumed in the kernel by "_raw_spin_lock", 
> with the httpd processes spending much of their time waiting on calls 
> to "futex".
>
> Here's a syscall analysis from svn: 
> http://paste.codebasehq.com/pastes/7qzt68lx2eghz2gjns
> Here's the kernel CPU time analysis: http://i.imgur.com/37Ryt5V.png
> Here's a snapshot from htop showing why this is a problem: 
> http://i.imgur.com/I3mDDbi.png
>
> I'd appreciate if anyone could tell me whether I have likely 
> configured something incorrectly, whether there is an obvious 
> workaround, or whether this needs to be escalated as a bug, and if so, 
> to whom?
>
> Thank you!
>
> Charlie
>
>
> My httpd is configured as follows:
>
> root@storage02:~# /opt/subversion-server/bin/httpd -V
> Server version: Apache/2.4.10 (Unix)
> Server built:   Nov  7 2014 15:16:58
> Server's Module Magic Number: 20120211:36
> Server loaded:  APR 1.5.1, APR-UTIL 1.5.4
> Compiled using: APR 1.5.1, APR-UTIL 1.5.4
> Architecture:   64-bit
> Server MPM:     worker
>   threaded:     yes (fixed thread count)
>     forked:     yes (variable process count)
> Server compiled with....
>  -D APR_HAS_SENDFILE
>  -D APR_HAS_MMAP
>  -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
>  -D APR_USE_SYSVSEM_SERIALIZE
>  -D APR_USE_PTHREAD_SERIALIZE
>  -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
>  -D APR_HAS_OTHER_CHILD
>  -D AP_HAVE_RELIABLE_PIPED_LOGS
>  -D DYNAMIC_MODULE_LIMIT=256
>  -D HTTPD_ROOT="/opt/subversion-server"
>  -D SUEXEC_BIN="/opt/subversion-server/bin/suexec"
>  -D DEFAULT_PIDLOG="logs/httpd.pid"
>  -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
>  -D DEFAULT_ERRORLOG="logs/error_log"
>  -D AP_TYPES_CONFIG_FILE="conf/mime.types"
>  -D SERVER_CONFIG_FILE="conf/httpd.conf"
>

It was suggested that this might be related to memory management, so I 
have looked into a couple of things:

1) I disabled swap on the server in question. The host had 25GB of RAM 
free and should not have been swapping active memory, however I believe 
disabling swap has solved the problem.
2) The server is running NUMA, rather than SMP memory configuration. I 
suspect that this is the reason for the problem (though I have no 
evidence) and that it is not specific to SVN/Apache.

Thanks to Bert on Freenode for his assistance in finding this 
workaround. I suspect that changing NUMA config, or changing Apache's 
threading model would also prevent the problem.

Charlie



Mime
View raw message