Mailing-List: contact users-help@cloudstack.apache.org; run by ezmlm
Precedence: bulk
Reply-To: users@cloudstack.apache.org
Received-SPF: pass (athena.apache.org: domain of shadowsor@gmail.com
 designates 74.125.82.174 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAMvtBPPEZeQcMW6rZPSKtzL3pxEH4PzDtUMtiy-SrVa+g0-gSw@mail.gmail.com>
References: 
 <CAMvtBPOmQMQGy3n_R5uPOCGtmM2pj3d_5wAi-EQ_n=_pVKkHKA@mail.gmail.com>
	<1424465766441.19696@ena.com>
	<CAMvtBPNWDEtYd9RAsqB4Cy7W_ePJOADou7UKy-RPXPqQ4tUY-Q@mail.gmail.com>
	<CAMvtBPPEZeQcMW6rZPSKtzL3pxEH4PzDtUMtiy-SrVa+g0-gSw@mail.gmail.com>
Date: Mon, 23 Feb 2015 08:21:58 -0800
Message-ID: 
 <CALFpzo5cUE=OU4+pM7bLEO+GvJ1D4Uvj4Lr3GojuWXPWpsS=jw@mail.gmail.com>
Subject: Re: Agent dies every night/morning.... memory violation
From: Marcus <shadowsor@gmail.com>
To: "dev@cloudstack.apache.org" <dev@cloudstack.apache.org>
Cc: "users@cloudstack.apache.org" <users@cloudstack.apache.org>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

It doesn't really sound like an agent problem, but some other root
problem that is causing issues for the agent. Perhaps it is specific
to the host simply because there is a particular VM that always runs
on that host and the VM itself is triggering the issue. Perhaps a
heavy logrotate or cron job on the vm causes issues for librados. Just
grasping at straws here. From the output provided it does seem that
the libvirt bindings that include ceph code are terminating the agent
execution.  My guess is that if you focus on "why this host" as
opposed to "what's going on", you'll find the answer to both. Sorry, I
know that's not much help.

On Mon, Feb 23, 2015 at 7:29 AM, Andrija Panic <andrija.panic@gmail.com> wr=
ote:
> Anybody?, before I start to cry :(
>
> On 21 February 2015 at 21:18, Andrija Panic <andrija.panic@gmail.com> wro=
te:
>
>> HI Simon,
>>
>> selinux is disabled, I have just double checked.
>>
>> BTW, this is what I can see in the cloudstack-agent.err log - seems like
>> some CEPH related issues, but not sure why would agent die...
>> If I recall correclty, this might be happening since the CEPH update fro=
m
>> 0.80.3? to 0.87 - and this seesm like some crash in librados....
>>
>>
>> libust[1907/2046]: Warning: HOME environment variable not set. Disabling
>> LTTng-UST per-user tracing. (in setup_local_apps() at lttng-ust-comm.c:3=
05)
>> libvirt:  error : name in virDomainLookupByName must not be NULL
>> libvirt:  error : name in virDomainLookupByName must not be NULL
>> libvirt:  error : name in virDomainLookupByName must not be NULL
>> libvirt:  error : name in virDomainLookupByName must not be NULL
>> libvirt: Storage Driver error : failed to remove volume
>> 'cloudstack/bd751250-de35-4d2e-a4e3-3ee4b636c2a7': Device or resource bu=
sy
>> ./log/SubsystemMap.h: In function 'bool
>> ceph::log::SubsystemMap::should_gather(unsigned int, int)' thread
>> 7f04427fc700 time 2015-02-21 06:39:38.839210
>> ./log/SubsystemMap.h: 62: FAILED assert(sub < m_subsys.size())
>>  ceph version 0.87 (c51c8f9d80fa4e0168aa52685b8de40e42758578)
>>  1: (()+0x1fe223) [0x7f060c932223]
>>  2: (ObjectCacher::flusher_entry()+0x155) [0x7f060c9866e5]
>>  3: (ObjectCacher::FlusherThread::entry()+0xd) [0x7f060c9976cd]
>>  4: (()+0x79d1) [0x7f06605ee9d1]
>>  5: (clone()+0x6d) [0x7f066033bb5d]
>>  NOTE: a copy of the executable, or `objdump -rdS <executable>` is neede=
d
>> to interpret this.
>> terminate called after throwing an instance of 'ceph::FailedAssertion'
>> 21/02/2015 06:39:38 1905 jsvc.exec error: Service did not exit cleanly
>>
>> On 20 February 2015 at 21:56, Simon Weller <sweller@ena.com> wrote:
>>
>>> Andrija,
>>>
>>> What is SELinux set to on this host?
>>>
>>>
>>> - SI
>>>
>>>
>>> ________________________________________
>>> From: Andrija Panic <andrija.panic@gmail.com>
>>> Sent: Friday, February 20, 2015 6:06 AM
>>> To: dev@cloudstack.apache.org; users@cloudstack.apache.org
>>> Subject: Agent dies every night/morning.... memory violation
>>>
>>> Hi,
>>>
>>> I have crazy agent on one of the hosts, that is being killed each morni=
ng
>>> and I found this in /var/log/audit.log:
>>>
>>> type=3DANOM_ABEND msg=3Daudit(1424321463.930:430678): auid=3D0 uid=3D0 =
gid=3D0
>>> ses=3D68891 pid=3D10831 comm=3D"jsvc" reason=3D"memory violation" sig=
=3D6
>>>
>>> I dont remember changing anything on the system, but this keeps happeni=
ng
>>> each morning arrond same time 5.20am-5.40am.
>>>
>>> I'm wondering what the hack is happening, any suggestions where to
>>> troubleshoot ?
>>> Will check logs in details anyway...
>>>
>>> --
>>>
>>> Andrija Pani=C4=87
>>>
>>
>>
>>
>> --
>>
>> Andrija Pani=C4=87
>>
>
>
>
> --
>
> Andrija Pani=C4=87