cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Timothy Ehlers <ehle...@gmail.com>
Subject Major stability problems lately
Date Thu, 14 Nov 2013 02:34:24 GMT
We are experiencing massive instability and cannot determine  whats causing
this.

Every so often jvsvc triggers the following in our system logs:

Nov 13 18:59:31 cpegh0009 kernel: \[15188599.258955\] BUG: soft lockup -
CPU#24 stuck for 22s\! \[jsvc:60385\]
Nov 13 18:59:31 cpegh0009 kernel: \[15188599.266229\] Modules linked in:
mptctl mptbase vhost_net macvtap macvlan 8021q garp ip6table_filter
ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat
nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJECT
xt_CHECKSUM iptable_mangle xt_tcpudp iptable_filter ip_tables x_tables nfsd
kvm_amd kvm ghash_clmulni_intel aesni_intel cryptd aes_x86_64 nfs microcode
psmouse radeon serio_raw ttm drm_kms_helper amd64_edac_mod joydev drm
edac_core fam15h_power k10temp edac_mce_amd i2c_algo_bit sp5100_tco
i2c_piix4 hpilo hpwdt lockd bridge stp mac_hid llc fscache auth_rpcgss
acpi_power_meter nfs_acl bonding sunrpc lp parport hid_generic usbhid hid
pata_atiixp ixgbe dca hpsa mdio
Nov 13 18:59:31 cpegh0009 kernel: \[15188599.266322\] CPU 24
Nov 13 18:59:31 cpegh0009 kernel: \[15188599.266323\] Modules linked in:
mptctl mptbase vhost_net macvtap macvlan 8021q garp ip6table_filter
ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat
nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJECT
xt_CHECKSUM iptable_mangle xt_tcpudp iptable_filter ip_tables x_tables nfsd
kvm_amd kvm ghash_clmulni_intel aesni_intel cryptd aes_x86_64 nfs microcode
psmouse radeon serio_raw ttm drm_kms_helper amd64_edac_mod joydev drm
edac_core fam15h_power k10temp edac_mce_amd i2c_algo_bit sp5100_tco
i2c_piix4 hpilo hpwdt lockd bridge stp mac_hid llc fscache auth_rpcgss
acpi_power_meter nfs_acl bonding sunrpc lp parport hid_generic usbhid hid
pata_atiixp ixgbe dca hpsa mdio
Nov 13 18:59:31 cpegh0009 kernel: \[15188599.266378\]
Nov 13 18:59:31 cpegh0009 kernel: \[15188599.266382\] Pid: 60385, comm:
jsvc Not tainted 3.5.0-23-generic #35~precise1-Ubuntu HP ProLiant DL585

I am not sure if this is the cause of the high load or an after effect..

03:25:01 PM runq-sz plist-sz ldavg-1 ldavg-5 ldavg-15 blocked 06:45:01 PM
31 982 36.95 39.33 41.50 0 06:55:01 PM 17 1000 28.53 37.28 40.06 0 07:05:01
PM 60 954 114.52 91.36 63.66 0 07:15:01 PM 48 961 29.55 53.94 60.76 0
07:25:01 PM 12 895 13.23 24.64 42.47 0 07:35:01 PM 5 772 8.02 13.32 28.31 0


We run ubuntu 12.04.3 LTS on HP DL585s with 64 AMD cores and .5 TB of ram.
This will host approx 40~50 vms (centos 5 guest).

Agent version is:
Version: 1:4.0.2

Any ideas?

Perhaps gathering cpu usage data on the jsvc pid ?

-- 
Tim Ehlers

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message