Mailing-List: contact cloudstack-dev-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: cloudstack-dev@incubator.apache.org
Received-SPF: pass (nike.apache.org: domain of Prasanna.Santhanam@citrix.com
 designates 203.166.19.134 as permitted sender)
From: Prasanna Santhanam <Prasanna.Santhanam@citrix.com>
To: "'cloudstack-dev@incubator.apache.org'"
	<cloudstack-dev@incubator.apache.org>,
	"'cloudstack-users@incubator.apache.org'"
	<cloudstack-users@incubator.apache.org>
Date: Fri, 14 Sep 2012 22:48:46 +0530
Subject: Re: Xenserver 6.0.2/Cloudstack 3.0.2 stale socket files
Thread-Topic: Xenserver 6.0.2/Cloudstack 3.0.2 stale socket files
Thread-Index: Ac2Sm7byaSkBW4RnRKK41QjL0uNb4QAAUib9
Message-ID: 
 <02C38648D4635F4EB02DE11EE81CF1EE0120AB512AA1@BANPMAILBOX01.citrite.net>
In-Reply-To: <CC78B2BE.10D13%Ahmad.Emneina@citrix.com>
Accept-Language: en-US
Content-Language: en-US
acceptlanguage: en-US
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

Caleb - what kind of storage are you using? XenServer local store or NFS sh=
ared store. We faced this with local store but only worked around the issue=
.


----- Original Message -----
From: Ahmad Emneina [mailto:Ahmad.Emneina@citrix.com]
Sent: Friday, September 14, 2012 10:39 PM=0A=
To: cloudstack-users@incubator.apache.org <cloudstack-users@incubator.apach=
e.org>; cloudstack-dev@incubator.apache.org <cloudstack-dev@incubator.apach=
e.org>
Subject: Re: Xenserver 6.0.2/Cloudstack 3.0.2 stale socket files

This looks like a prime candidate for a bug. There might be time to get it
in before 4.0 goes out!

On 9/14/12 9:54 AM, "Caleb Call" <calebcall@me.com> wrote:

>We came across an interesting issue yesterday in one of our clusters.  We
>ran out of inodes on all of our cluster members (since when does this
>happen in 2012?).  When this happened, it in turn made the / filesystem a
>read-only filesystem which in turn made all the hosts go in to emergency
>maintenance mode and as a result get marked down by Cloudstack.  We found
>that it was caused by hundreds of thousands of stale socket files in /tmp
>named "stream-unix.####.######".  To resolve the issue, we had to delete
>those stale socket files (find /tmp -name "*stream*" -mtime +7 -exec rm
>-v {} \;), then kill and restart xapi, then correct the emergency
>maintenance mode.  These hosts had only been up for 45 days before this
>issue occurred. =20
>
>In our scouring of the interwebs, the only other instance we've been able
>to find of this (or similar) happening is in the same setup we are
>currently running.  Xenserver 6.0.2 with CS 3.0.2.  Do these stream-unix
>sockets have anything to do with Cloudstack?  I would think if this was a
>Xenserver issue (bug), there would be a lot more on the internet about
>this happening.  For a temporary workaround, we've added a cronjob to
>cleanup these files but we'd really like to address the actual issue
>that's causing these sockets to become stale and not get cleaned-up.
>
>Thoughts?
>
>Thanks,
>Caleb
>


--=20
=C6