Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
MIME-Version: 1.0
In-Reply-To: <CAKz4c0TGROtqnquDZrb=QvVADFRYMCs_9tQ5NT4yVL7D67UK8w@mail.gmail.com>
References: <CAKz4c0TnL-CVSwK9G_JnhxPfgszYDb5fMk_c+3S9aq2CiUH91w@mail.gmail.com>
 <CAPbVhuMOyQnYKGSm_bb5qUevjU8j4CbO+ERD0wFZzun9OgAyQw@mail.gmail.com> <CAKz4c0TGROtqnquDZrb=QvVADFRYMCs_9tQ5NT4yVL7D67UK8w@mail.gmail.com>
From: Simone Franzini <captainfranz@gmail.com>
Date: Tue, 7 Feb 2017 11:16:47 -0600
Message-ID: <CAKz4c0SSzHVVkRv2ncfKo+34g=b9ynCebHXQfgiQHTWoUQzxJw@mail.gmail.com>
Subject: Re: UnknownColumnFamilyException after removing all Cassandra data
To: user@cassandra.apache.org
Content-Type: multipart/alternative; boundary=001a114b2f0af7e9730547f3eaa6
archived-at: Tue, 07 Feb 2017 17:17:16 -0000

--001a114b2f0af7e9730547f3eaa6
Content-Type: text/plain; charset=UTF-8

To further add to my previous answer, the node in question is a seed node,
so it did not bootstrap.
Should I remove it from the list of seed nodes and then try to restart it?

Simone Franzini, PhD

http://www.linkedin.com/in/simonefranzini

On Tue, Feb 7, 2017 at 9:43 AM, Simone Franzini <captainfranz@gmail.com>
wrote:

> This is exactly what I did on the second node. If this is not the correct
> / best procedure to adopt in these cases, please advise:
>
> 1. Removed all the data, including the system table (rm -rf data/
> commitlog/ saved_caches).
> 2. Configured the node to replace itself, by adding the following line to
> cassandra-env.sh: JVM_OPTS="$JVM_OPTS -Dcassandra.replace_address=<node
> own IP address>"
> 3. Start the node.
>
> Noticeably, I did not do nodetool decommission or removenode. Is that the
> recommended approach?
>
> Given what I did, I am mystified as to what the problem is. If I query the
> system.schema_columnfamilies on the affected node, all CF IDs are there.
> Same goes for the only other node that is currently up. Also, the other
> node that is currently up has data for all those CF IDs in the data folder.
>
>
> Simone Franzini, PhD
>
> http://www.linkedin.com/in/simonefranzini
>
> On Tue, Feb 7, 2017 at 5:39 AM, kurt greaves <kurt@instaclustr.com> wrote:
>
>> The node is trying to communicate with another node, potentially
>> streaming data, and is receiving files/data for an "unknown column family".
>> That is, it doesn't know about the CF with the id
>> e36415b6-95a7-368c-9ac0-ae0ac774863d.
>> If you deleted some columnfamilies but not all the system keyspace and
>> restarted the node I'd expect this error to occur. Or I suppose if you
>> didn't decommission the node properly before blowing the data away and
>> restarting.
>>
>> You'll have to give us more information on what your exact steps were on
>> this 2nd node:
>>
>> When you say deleted all Cassandra data, did this include the system
>> tables? Were your steps to delete all the data and then just restart the
>> node? Did you remove the node from the cluster prior to deleting the data
>> and restarting it (nodetool decommission/removenode? Did the node rejoin
>> the cluster or did it have to bootstrap?
>>
>>
>>
>

--001a114b2f0af7e9730547f3eaa6
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">To further add to my previous answer, the node in question=
 is a seed node, so it did not bootstrap.=C2=A0<div>Should I remove it from=
 the list of seed nodes and then try to restart it?</div></div><div class=
=3D"gmail_extra"><br clear=3D"all"><div><div class=3D"gmail_signature" data=
-smartmail=3D"gmail_signature"><div dir=3D"ltr">Simone Franzini, PhD<br><br=
><a href=3D"http://www.linkedin.com/in/simonefranzini" target=3D"_blank">ht=
tp://www.linkedin.com/in/simonefranzini</a></div></div></div>
<br><div class=3D"gmail_quote">On Tue, Feb 7, 2017 at 9:43 AM, Simone Franz=
ini <span dir=3D"ltr">&lt;<a href=3D"mailto:captainfranz@gmail.com" target=
=3D"_blank">captainfranz@gmail.com</a>&gt;</span> wrote:<br><blockquote cla=
ss=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;pa=
dding-left:1ex"><div dir=3D"ltr">This is exactly what I did on the second n=
ode. If this is not the correct / best procedure to adopt in these cases, p=
lease advise:<div><br></div><div><span style=3D"font-size:12.8px">1. Remove=
d all the data, including the system table (rm -rf data/ commitlog/ saved_c=
aches).</span></div><div><span style=3D"font-size:12.8px">2. Configured the=
 node to replace itself, by adding the following line to cassandra-env.sh: =
JVM_OPTS=3D&quot;$JVM_OPTS -Dcassandra.replace_address=3D&lt;<wbr>node own =
IP address&gt;&quot;</span></div><div><span style=3D"font-size:12.8px">3. S=
tart the node.</span></div><div><span style=3D"font-size:12.8px"><br></span=
></div><div><span style=3D"font-size:12.8px">Noticeably, I did not do nodet=
ool decommission or removenode. Is that the recommended approach?</span></d=
iv><div><span style=3D"font-size:12.8px"><br></span></div><div><span style=
=3D"font-size:12.8px">Given what I did, I am mystified as to what the probl=
em is. If I query the system.schema_columnfamilies on the affected node, al=
l CF IDs are there. Same goes for the only other node that is currently up.=
 Also, the other node that is currently up has data for all those CF IDs in=
 the data folder.</span></div><div><span style=3D"font-size:12.8px"><br></s=
pan></div></div><div class=3D"gmail_extra"><span class=3D""><br clear=3D"al=
l"><div><div class=3D"m_7801654092693973091gmail_signature" data-smartmail=
=3D"gmail_signature"><div dir=3D"ltr">Simone Franzini, PhD<br><br><a href=
=3D"http://www.linkedin.com/in/simonefranzini" target=3D"_blank">http://www=
.linkedin.com/in/<wbr>simonefranzini</a></div></div></div>
<br></span><div><div class=3D"h5"><div class=3D"gmail_quote">On Tue, Feb 7,=
 2017 at 5:39 AM, kurt greaves <span dir=3D"ltr">&lt;<a href=3D"mailto:kurt=
@instaclustr.com" target=3D"_blank">kurt@instaclustr.com</a>&gt;</span> wro=
te:<br><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-=
left:1px #ccc solid;padding-left:1ex"><div dir=3D"ltr"><div>The node is try=
ing to communicate with another node, potentially streaming data, and is re=
ceiving files/data for an &quot;unknown column family&quot;. That is, it do=
esn&#39;t know about the CF with the id e36415b6-95a7-368c-9ac0-ae0ac7<wbr>=
74863d.<br></div><div>If you deleted some columnfamilies but not all the sy=
stem keyspace and restarted the node I&#39;d expect this error to occur. Or=
 I suppose if you didn&#39;t decommission the node properly before blowing =
the data away and restarting. <br><br>You&#39;ll have to give us more infor=
mation on what your exact steps were on this 2nd node:<br><br></div>When yo=
u say deleted all Cassandra data, did this include the system tables? Were =
your steps to delete all the data and then just restart the node? Did you r=
emove the node from the cluster prior to deleting the data and restarting i=
t (nodetool decommission/removenode? Did the node rejoin the cluster or did=
 it have to bootstrap?<br><div><div><div><br><br></div></div></div></div>
</blockquote></div><br></div></div></div>
</blockquote></div><br></div>

--001a114b2f0af7e9730547f3eaa6--