Return-Path: X-Original-To: apmail-incubator-cloudstack-users-archive@minotaur.apache.org Delivered-To: apmail-incubator-cloudstack-users-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 71777D0E5 for ; Fri, 14 Sep 2012 16:55:20 +0000 (UTC) Received: (qmail 7378 invoked by uid 500); 14 Sep 2012 16:55:20 -0000 Delivered-To: apmail-incubator-cloudstack-users-archive@incubator.apache.org Received: (qmail 7336 invoked by uid 500); 14 Sep 2012 16:55:20 -0000 Mailing-List: contact cloudstack-users-help@incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: cloudstack-users@incubator.apache.org Delivered-To: mailing list cloudstack-users@incubator.apache.org Received: (qmail 7322 invoked by uid 99); 14 Sep 2012 16:55:20 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Sep 2012 16:55:20 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of calebcall@me.com designates 17.158.236.239 as permitted sender) Received: from [17.158.236.239] (HELO nk11p04mm-asmtp004.mac.com) (17.158.236.239) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Sep 2012 16:55:10 +0000 MIME-version: 1.0 Content-type: multipart/alternative; boundary="Boundary_(ID_1U5chBxio6GWwrXiG3CLug)" Received: from acedia.overstock.com (unknown [65.116.116.6]) by nk11p04mm-asmtp004.mac.com (Oracle Communications Messaging Server 7u4-24.01(7.0.4.24.0) 64bit (built Jan 3 2012)) with ESMTPSA id <0MAC00CNSMYJ03B0@nk11p04mm-asmtp004.mac.com>; Fri, 14 Sep 2012 16:54:49 +0000 (GMT) X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:5.7.7855,1.0.431,0.0.0000 definitions=2012-09-14_07:2012-09-14,2012-09-14,1970-01-01 signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 ipscore=0 suspectscore=0 phishscore=0 bulkscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=6.0.2-1203120001 definitions=main-1209140187 From: Caleb Call Subject: Xenserver 6.0.2/Cloudstack 3.0.2 stale socket files Message-id: Date: Fri, 14 Sep 2012 10:54:09 -0600 To: "cloudstack-users@incubator.apache.org" , "cloudstack-dev@incubator.apache.org" X-Mailer: Apple Mail (2.1486) --Boundary_(ID_1U5chBxio6GWwrXiG3CLug) Content-type: text/plain; CHARSET=US-ASCII Content-transfer-encoding: 7BIT We came across an interesting issue yesterday in one of our clusters. We ran out of inodes on all of our cluster members (since when does this happen in 2012?). When this happened, it in turn made the / filesystem a read-only filesystem which in turn made all the hosts go in to emergency maintenance mode and as a result get marked down by Cloudstack. We found that it was caused by hundreds of thousands of stale socket files in /tmp named "stream-unix.####.######". To resolve the issue, we had to delete those stale socket files (find /tmp -name "*stream*" -mtime +7 -exec rm -v {} \;), then kill and restart xapi, then correct the emergency maintenance mode. These hosts had only been up for 45 days before this issue occurred. In our scouring of the interwebs, the only other instance we've been able to find of this (or similar) happening is in the same setup we are currently running. Xenserver 6.0.2 with CS 3.0.2. Do these stream-unix sockets have anything to do with Cloudstack? I would think if this was a Xenserver issue (bug), there would be a lot more on the internet about this happening. For a temporary workaround, we've added a cronjob to cleanup these files but we'd really like to address the actual issue that's causing these sockets to become stale and not get cleaned-up. Thoughts? Thanks, Caleb --Boundary_(ID_1U5chBxio6GWwrXiG3CLug)--