Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (athena.apache.org: domain of john.pyeatt@singlewire.com
 designates 74.125.149.151 as permitted sender)
MIME-Version: 1.0
Date: Mon, 2 Dec 2013 16:59:38 -0600
Message-ID: 
 <CAEisTLnEvp31YAJnRwHKx28y7WLNyyYnfbx-1brBqcLzDMhu2g@mail.gmail.com>
Subject: Stack trace from a node during a repair
From: John Pyeatt <john.pyeatt@singlewire.com>
To: user@cassandra.apache.org
Content-Type: multipart/alternative; boundary=bcaec520f2e36c752f04ec952273

--bcaec520f2e36c752f04ec952273
Content-Type: text/plain; charset=ISO-8859-1

We are running a 6-node AWS EC2 (m1.large) cluster of cassandra 1.2.9
across three availability zones with Ec2Snitch and NetworkTopologyStrategy.

One of our nodes was apparently sharing a physical box with another
customer who was really hogging the IO. So we needed to bring the node up
on a new ec2 instance.

We decommissioned the offending node, killed the instance and brought a new
instance into the cluster. Everything went fine so far.

After it came up I ran a nodetool repair -pr on each of the nodes in the
cluster. I ran these sequentially. When it got to doing the repair on the
new node three times the gossip service shut down. At the bottom of this
email is a copy of the stack trace we received.

It says it couldn't create a backups directory. I have no idea why this
would be the /data-1 partition is 400Gb in size and currently 1% utilized.
Does anyone have any idea what could be causing this?

my /etc/security/limits.conf file currently has
# resource settings added based on
# http://www.datastax.com/docs/1.2/install/recommended_settings
* soft nofile 65536
* hard nofile 65536
root soft nofile 65536
root hard nofile 65536
* soft memlock unlimited
* hard memlock unlimited
root soft memlock unlimited
root hard memlock unlimited
* soft as unlimited
* hard as unlimited
root soft as unlimited
root hard as unlimited


ERROR 2013-12-02 21:02:25,711 [Thread-3050] CassandraDaemon Exception in
thread Thread[Thread-3050,5,main]
FSWriteError in /data-1/cassandra/data/SinglewireSupport/Binaries/backups
        at
org.apache.cassandra.db.Directories.getOrCreate(Directories.java:483)
        at
org.apache.cassandra.db.Directories.getBackupsDirectory(Directories.java:242)
        at
org.apache.cassandra.db.DataTracker.maybeIncrementallyBackup(DataTracker.java:165)
        at
org.apache.cassandra.db.DataTracker.addSSTables(DataTracker.java:237)
        at
org.apache.cassandra.db.ColumnFamilyStore.addSSTables(ColumnFamilyStore.java:911)
        at
org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:186)
        at
org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:138)
        at
org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:238)
        at
org.apache.cassandra.net.IncomingTcpConnection.handleStream(IncomingTcpConnection.java:178)
        at
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:78)
Caused by: java.io.IOException: Unable to create directory
/data-1/cassandra/data/SinglewireSupport/Binaries/backups


-- 
John Pyeatt
Singlewire Software, LLC
www.singlewire.com
------------------
608.661.1184
john.pyeatt@singlewire.com

--bcaec520f2e36c752f04ec952273
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div><div><div><div><div>We are running a 6-node AWS EC2 (=
m1.large) cluster of cassandra 1.2.9 across three availability zones with E=
c2Snitch and NetworkTopologyStrategy.<br><br></div>One of our nodes was app=
arently sharing a physical box with another customer who was really hogging=
 the IO. So we needed to bring the node up on a new ec2 instance. <br>
<br></div>We decommissioned the offending node, killed the instance and bro=
ught a new instance into the cluster. Everything went fine so far.<br><br>A=
fter it came up I ran a nodetool repair -pr on each of the nodes in the clu=
ster. I ran these sequentially. When it got to doing the repair on the new =
node three times the gossip service shut down. At the bottom of this email =
is a copy of the stack trace we received.<br>
<br></div>It says it couldn&#39;t create a backups directory. I have no ide=
a why this would be the /data-1 partition is 400Gb in size and currently 1%=
 utilized. Does anyone have any idea what could be causing this?<br><br>
</div>my /etc/security/limits.conf file currently has<br><span style=3D"fon=
t-family:courier new,monospace"><font size=3D"1"># resource settings added =
based on<br># <a href=3D"http://www.datastax.com/docs/1.2/install/recommend=
ed_settings">http://www.datastax.com/docs/1.2/install/recommended_settings<=
/a><br>
* soft nofile 65536<br>* hard nofile 65536<br>root soft nofile 65536<br>roo=
t hard nofile 65536<br>* soft memlock unlimited<br>* hard memlock unlimited=
<br>root soft memlock unlimited<br>root hard memlock unlimited<br>* soft as=
 unlimited<br>
* hard as unlimited<br>root soft as unlimited<br>root hard as unlimited</fo=
nt></span><br><br></div><div><div><div><br><br><span style=3D"font-family:c=
ourier new,monospace"><font size=3D"1"><span style=3D"color:rgb(255,0,0)">E=
RROR 2013-12-02 21:02:25,711 [Thread-3050] CassandraDaemon Exception in thr=
ead Thread[Thread-3050,5,main]<br>
FSWriteError in /data-1/cassandra/data/SinglewireSupport/Binaries/backups</=
span><br>=A0=A0=A0=A0=A0=A0=A0 at org.apache.cassandra.db.Directories.getOr=
Create(Directories.java:483)<br>=A0=A0=A0=A0=A0=A0=A0 at org.apache.cassand=
ra.db.Directories.getBackupsDirectory(Directories.java:242)<br>
=A0=A0=A0=A0=A0=A0=A0 at org.apache.cassandra.db.DataTracker.maybeIncrement=
allyBackup(DataTracker.java:165)<br>=A0=A0=A0=A0=A0=A0=A0 at org.apache.cas=
sandra.db.DataTracker.addSSTables(DataTracker.java:237)<br>=A0=A0=A0=A0=A0=
=A0=A0 at org.apache.cassandra.db.ColumnFamilyStore.addSSTables(ColumnFamil=
yStore.java:911)<br>
=A0=A0=A0=A0=A0=A0=A0 at org.apache.cassandra.streaming.StreamInSession.clo=
seIfFinished(StreamInSession.java:186)<br>=A0=A0=A0=A0=A0=A0=A0 at org.apac=
he.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:=
138)<br>=A0=A0=A0=A0=A0=A0=A0 at org.apache.cassandra.net.IncomingTcpConnec=
tion.stream(IncomingTcpConnection.java:238)<br>
=A0=A0=A0=A0=A0=A0=A0 at org.apache.cassandra.net.IncomingTcpConnection.han=
dleStream(IncomingTcpConnection.java:178)<br>=A0=A0=A0=A0=A0=A0=A0 at org.a=
pache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:78=
)<br><span style=3D"color:rgb(255,0,0)">Caused by: java.io.IOException: Una=
ble to create directory /data-1/cassandra/data/SinglewireSupport/Binaries/b=
ackups</span></font></span><br>
<br clear=3D"all"><div><div><div><div><br>-- <br>
John Pyeatt<br>

Singlewire Software, LLC<br>
<a href=3D"http://www.singlewire.com/" target=3D"_blank">www.singlewire.com=
</a><br>
------------------<br>
608.661.1184<br><font color=3D"#888888">
<a href=3D"mailto:john.pyeatt@singlewire.com" target=3D"_blank">john.pyeatt=
@singlewire.com</a></font>
</div></div></div></div></div></div></div></div>

--bcaec520f2e36c752f04ec952273--