Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (nike.apache.org: domain of genericelias@gmail.com
 designates 209.85.219.48 as permitted sender)
MIME-Version: 1.0
Sender: genericelias@gmail.com
In-Reply-To: 
 <CAHUe4wAeJJTwkQM1YnLVxyYkEKNCyh+x_DJHvPDaHndu4qEW0A@mail.gmail.com>
References: 
 <CAKsEmEOzSqDC-hiLQHmRn8DkkWHnQLfC_RyS2cEY1N17pyRqYQ@mail.gmail.com>
	<CAHUe4wAeJJTwkQM1YnLVxyYkEKNCyh+x_DJHvPDaHndu4qEW0A@mail.gmail.com>
Date: Fri, 8 Nov 2013 10:31:27 -0800
Message-ID: 
 <CAKsEmEMjBozmpXNDz6Q4ksjTxEStjyvN6vn0fYakuW1MFhwXoQ@mail.gmail.com>
Subject: Re: cleanup failure; FileNotFoundException deleting (wrong?) db file
From: Elias Ross <genman@noderunner.net>
To: user@cassandra.apache.org
Content-Type: multipart/alternative; boundary=001a11c30fac1eabb004eaae974d

--001a11c30fac1eabb004eaae974d
Content-Type: text/plain; charset=ISO-8859-1

On Thu, Nov 7, 2013 at 7:01 PM, Krishna Chaitanya <bnsk1990rulz@gmail.com>wrote:

> Check if its an issue with permissions or broken links..
>
>
I don't think permissions are an issue. You might be on to something
regarding the links.

I've been seeing this on 4 nodes, configured identically.

Here's what I think the problem may be: (or may be a combination of a few
problems)

1. I have symlinked the data directories. This confuses Cassandra in some
way, causing it to create multiple files. Does Cassandra care if the data
directory was symlinked from someplace? Would this cause an issue.

lrwxrwxrwx    1 root root     6 Oct 30 18:37 data01 -> /data1 # [1]

Evidence for:
a. Somehow it's creating duplicate hard links.
b. It is unlikely other Cassandra users would have setup their directories
like this and this seems like a serious bug.
c. Also, my other cluster is nearly identical (OS, JVM, 6 drives, same
Cassandra/RHQ, hardware similar) and not seeing the same issues, although
that is a two node cluster.

If I were to grep through, I guess I would see if there's a chance the path
that Java sees, maybe File.getAbsoluteFile() (which might resolve the link)
doesn't match the path of another file. In other words, it is a Cassandra
bug, based on some assumptions from the JVM


2. When I created the cluster, I had a single data directory for each node.
I then added 5 more. Somehow Cassandra mis-remembers where the data was
put, causing all sorts of issues. How does Cassandra decide where to put
its data and where to read it from? What happens when additional data
directories are added? There could be a bug in the code.

Evidence for:
a. Somehow it's looking for data in the wrong directory. It also seems
unlikely a user would create a cluster, then add 5 more drives.

# [1] The reason the links are setup is because the mount points didn't
match my Puppet setup, which sets up my directory permissions. So I added
the links to compensate.

--001a11c30fac1eabb004eaae974d
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">On Thu, Nov 7, 2013 at 7:01 PM, Krishna Chaitanya <span di=
r=3D"ltr">&lt;<a href=3D"mailto:bnsk1990rulz@gmail.com" target=3D"_blank">b=
nsk1990rulz@gmail.com</a>&gt;</span> wrote:<br><div class=3D"gmail_extra"><=
div class=3D"gmail_quote">
<blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-=
left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;p=
adding-left:1ex"><p>Check if its an issue with permissions or broken links.=
.</p>
<div class=3D""><div class=3D"h5">
<div class=3D"gmail_quote"><br></div></div></div></blockquote><div><br></di=
v><div>I don&#39;t think permissions are an issue. You might be on to somet=
hing regarding the links.</div><div><br></div><div>I&#39;ve been seeing thi=
s on 4 nodes, configured identically.</div>
<div><br></div><div>Here&#39;s what I think the problem may be: (or may be =
a combination of a few problems)</div><div><br></div><div>1. I have symlink=
ed the data directories. This confuses Cassandra in some way, causing it to=
 create multiple files. Does Cassandra care if the data directory was symli=
nked from someplace? Would this cause an issue.<br>
</div><div><br></div><div><div>lrwxrwxrwx =A0 =A01 root root =A0 =A0 6 Oct =
30 18:37 data01 -&gt; /data1 # [1]</div></div><div><br></div><div>Evidence =
for:</div><div>a. Somehow it&#39;s creating duplicate hard links.</div><div=
>b. It is unlikely other Cassandra users would have setup their directories=
 like this and this seems like a serious bug.</div>
<div>c. Also, my other cluster is nearly identical (OS, JVM, 6 drives, same=
 Cassandra/RHQ, hardware similar) and not seeing the same issues, although =
that is a two node cluster.<br></div><div><br></div><div><div>If I were to =
grep through, I guess I would see if there&#39;s a chance the path that Jav=
a sees, maybe File.getAbsoluteFile() (which might resolve the link) doesn&#=
39;t match the path of another file. In other words, it is a Cassandra bug,=
 based on some assumptions from the JVM</div>
</div><div><br></div><div><br></div><div>2. When I created the cluster, I h=
ad a single data directory for each node. I then added 5 more. Somehow Cass=
andra mis-remembers where the data was put, causing all sorts of issues. Ho=
w does Cassandra decide where to put its data and where to read it from? Wh=
at happens when additional data directories are added? There could be a bug=
 in the code.</div>
<div><br></div><div>Evidence for:</div><div>a. Somehow it&#39;s looking for=
 data in the wrong directory. It also seems unlikely a user would create a =
cluster, then add 5 more drives.<br></div><div><br></div><div># [1] The rea=
son the links are setup is because the mount points didn&#39;t match my Pup=
pet setup, which sets up my directory permissions. So I added the links to =
compensate.</div>
</div></div></div>

--001a11c30fac1eabb004eaae974d--