cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrija Panic <andrija.pa...@gmail.com>
Subject Re: Agent dies every night/morning.... memory violation
Date Sat, 21 Feb 2015 20:18:51 GMT
HI Simon,

selinux is disabled, I have just double checked.

BTW, this is what I can see in the cloudstack-agent.err log - seems like
some CEPH related issues, but not sure why would agent die...
If I recall correclty, this might be happening since the CEPH update from
0.80.3? to 0.87 - and this seesm like some crash in librados....


libust[1907/2046]: Warning: HOME environment variable not set. Disabling
LTTng-UST per-user tracing. (in setup_local_apps() at lttng-ust-comm.c:305)
libvirt:  error : name in virDomainLookupByName must not be NULL
libvirt:  error : name in virDomainLookupByName must not be NULL
libvirt:  error : name in virDomainLookupByName must not be NULL
libvirt:  error : name in virDomainLookupByName must not be NULL
libvirt: Storage Driver error : failed to remove volume
'cloudstack/bd751250-de35-4d2e-a4e3-3ee4b636c2a7': Device or resource busy
./log/SubsystemMap.h: In function 'bool
ceph::log::SubsystemMap::should_gather(unsigned int, int)' thread
7f04427fc700 time 2015-02-21 06:39:38.839210
./log/SubsystemMap.h: 62: FAILED assert(sub < m_subsys.size())
 ceph version 0.87 (c51c8f9d80fa4e0168aa52685b8de40e42758578)
 1: (()+0x1fe223) [0x7f060c932223]
 2: (ObjectCacher::flusher_entry()+0x155) [0x7f060c9866e5]
 3: (ObjectCacher::FlusherThread::entry()+0xd) [0x7f060c9976cd]
 4: (()+0x79d1) [0x7f06605ee9d1]
 5: (clone()+0x6d) [0x7f066033bb5d]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed
to interpret this.
terminate called after throwing an instance of 'ceph::FailedAssertion'
21/02/2015 06:39:38 1905 jsvc.exec error: Service did not exit cleanly

On 20 February 2015 at 21:56, Simon Weller <sweller@ena.com> wrote:

> Andrija,
>
> What is SELinux set to on this host?
>
>
> - SI
>
>
> ________________________________________
> From: Andrija Panic <andrija.panic@gmail.com>
> Sent: Friday, February 20, 2015 6:06 AM
> To: dev@cloudstack.apache.org; users@cloudstack.apache.org
> Subject: Agent dies every night/morning.... memory violation
>
> Hi,
>
> I have crazy agent on one of the hosts, that is being killed each morning
> and I found this in /var/log/audit.log:
>
> type=ANOM_ABEND msg=audit(1424321463.930:430678): auid=0 uid=0 gid=0
> ses=68891 pid=10831 comm="jsvc" reason="memory violation" sig=6
>
> I dont remember changing anything on the system, but this keeps happening
> each morning arrond same time 5.20am-5.40am.
>
> I'm wondering what the hack is happening, any suggestions where to
> troubleshoot ?
> Will check logs in details anyway...
>
> --
>
> Andrija Panić
>



-- 

Andrija Panić

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message