incubator-cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ahmad Emneina <Ahmad.Emne...@citrix.com>
Subject Re: Xenserver 6.0.2/Cloudstack 3.0.2 stale socket files
Date Fri, 14 Sep 2012 17:09:37 GMT
This looks like a prime candidate for a bug. There might be time to get it
in before 4.0 goes out!

On 9/14/12 9:54 AM, "Caleb Call" <calebcall@me.com> wrote:

>We came across an interesting issue yesterday in one of our clusters.  We
>ran out of inodes on all of our cluster members (since when does this
>happen in 2012?).  When this happened, it in turn made the / filesystem a
>read-only filesystem which in turn made all the hosts go in to emergency
>maintenance mode and as a result get marked down by Cloudstack.  We found
>that it was caused by hundreds of thousands of stale socket files in /tmp
>named "stream-unix.####.######".  To resolve the issue, we had to delete
>those stale socket files (find /tmp -name "*stream*" -mtime +7 -exec rm
>-v {} \;), then kill and restart xapi, then correct the emergency
>maintenance mode.  These hosts had only been up for 45 days before this
>issue occurred.  
>
>In our scouring of the interwebs, the only other instance we've been able
>to find of this (or similar) happening is in the same setup we are
>currently running.  Xenserver 6.0.2 with CS 3.0.2.  Do these stream-unix
>sockets have anything to do with Cloudstack?  I would think if this was a
>Xenserver issue (bug), there would be a lot more on the internet about
>this happening.  For a temporary workaround, we've added a cronjob to
>cleanup these files but we'd really like to address the actual issue
>that's causing these sockets to become stale and not get cleaned-up.
>
>Thoughts?
>
>Thanks,
>Caleb
>


-- 
Æ




Mime
View raw message