incubator-cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Caleb Call <calebc...@me.com>
Subject Re: Xenserver 6.0.2/Cloudstack 3.0.2 stale socket files
Date Fri, 14 Sep 2012 17:23:16 GMT
Primary storage is on SAN (fiber attached), secondary is on NFS.  

I've submitted CLOUDSTACK-105.

On Sep 14, 2012, at 11:18 AM, Prasanna Santhanam <Prasanna.Santhanam@citrix.com> wrote:

> Caleb - what kind of storage are you using? XenServer local store or NFS shared store.
We faced this with local store but only worked around the issue.
> 
> 
> 
> ----- Original Message -----
> From: Ahmad Emneina [mailto:Ahmad.Emneina@citrix.com]
> Sent: Friday, September 14, 2012 10:39 PM
> To: cloudstack-users@incubator.apache.org <cloudstack-users@incubator.apache.org>;
cloudstack-dev@incubator.apache.org <cloudstack-dev@incubator.apache.org>
> Subject: Re: Xenserver 6.0.2/Cloudstack 3.0.2 stale socket files
> 
> This looks like a prime candidate for a bug. There might be time to get it
> in before 4.0 goes out!
> 
> On 9/14/12 9:54 AM, "Caleb Call" <calebcall@me.com> wrote:
> 
>> We came across an interesting issue yesterday in one of our clusters.  We
>> ran out of inodes on all of our cluster members (since when does this
>> happen in 2012?).  When this happened, it in turn made the / filesystem a
>> read-only filesystem which in turn made all the hosts go in to emergency
>> maintenance mode and as a result get marked down by Cloudstack.  We found
>> that it was caused by hundreds of thousands of stale socket files in /tmp
>> named "stream-unix.####.######".  To resolve the issue, we had to delete
>> those stale socket files (find /tmp -name "*stream*" -mtime +7 -exec rm
>> -v {} \;), then kill and restart xapi, then correct the emergency
>> maintenance mode.  These hosts had only been up for 45 days before this
>> issue occurred.  
>> 
>> In our scouring of the interwebs, the only other instance we've been able
>> to find of this (or similar) happening is in the same setup we are
>> currently running.  Xenserver 6.0.2 with CS 3.0.2.  Do these stream-unix
>> sockets have anything to do with Cloudstack?  I would think if this was a
>> Xenserver issue (bug), there would be a lot more on the internet about
>> this happening.  For a temporary workaround, we've added a cronjob to
>> cleanup these files but we'd really like to address the actual issue
>> that's causing these sockets to become stale and not get cleaned-up.
>> 
>> Thoughts?
>> 
>> Thanks,
>> Caleb
>> 
> 
> 
> -- 
> Æ
> 
> 
> 


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message