Mailing-List: contact commits-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@cassandra.apache.org
Date: Fri, 17 Aug 2012 11:11:38 +1100 (NCT)
From: "Florent Clairambault (JIRA)" <jira@apache.org>
To: commits@cassandra.apache.org
Message-ID: <679947526.21958.1345162298330.JavaMail.jiratomcat@arcas>
In-Reply-To: <1624491622.4291.1343895182925.JavaMail.jiratomcat@issues-vm>
Subject: [jira] [Comment Edited] (CASSANDRA-4481) Commitlog not replayed
 after restart - data lost
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable


    [ https://issues.apache.org/jira/browse/CASSANDRA-4481?page=3Dcom.atlas=
sian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D=
13436450#comment-13436450 ]=20

Florent Clairambault edited comment on CASSANDRA-4481 at 8/17/12 11:11 AM:
---------------------------------------------------------------------------

This bug is marked as resolved, so we're just documenting something that ne=
ver happened. We're not scaring anyone here, we're making sure we have all =
the documentation to prove that I we were wrong.

So just to make things clear, I didn't make any kind of change or deletion =
on my keyspaces. The two keyspaces were created by code (one with pelops an=
d one with hector) once and never changed. I know I told I did it with cass=
andra-cli earlier but it turns out that it was entirely by code.

While doing some tests, I did delete the keyspaces and in that cases it giv=
es an error that looks like: "Commit logs for non-existing Column Family 10=
36 were ignored" (I can't find the exact error in my logs).=20

When I deleted the keyspace files, they were recreated by reading the commi=
t logs (this is step 4 in my previous report). So I think they were in acco=
rdance with the schema stored in cassandra.

---=20

I wanted to actually test it.

The only last versions I could find were 1.0.11, 1.1.2 and 1.1.3. I created=
 a small testscript and it definitely works with them. But it would be good=
 to test it with 1.1.1 (which I didn't find) also.

{code}
#!/bin/sh
apt-get remove --purge cassandra -y
rm -Rf /var/log/cassandra /var/lib/cassandra

if [ ! -f cassandra_1.0.11_all.deb ]; then
  wget http://apache.mirrors.multidist.eu/cassandra/debian/pool/main/c/cass=
andra/cassandra_1.0.11_all.deb
fi

if [ ! -f cassandra_1.1.2_all.deb ]; then
   wget http://apache.mirrors.multidist.eu/cassandra/debian/pool/main/c/cas=
sandra/cassandra_1.1.2_all.deb
fi

if [ ! -f cassandra_1.1.3_all.deb ]; then
  wget http://apache.mirrors.multidist.eu/cassandra/debian/pool/main/c/cass=
andra/cassandra_1.1.3_all.deb
fi

wait_for_server() {
  while ! echo exit | nc localhost 9160; do sleep 1; done
}

dpkg -i cassandra_1.0.11_all.deb
tail -f /var/log/cassandra/output.log &

wait_for_server;

cassandra-cli -h localhost <<EOF

create keyspace m2mp;
use m2mp;

create column family Registry
  with column_type =3D 'Standard'
  and comparator =3D 'AsciiType'
  and default_validation_class =3D 'AsciiType'
  and key_validation_class =3D 'AsciiType';

set Registry['/user/florent']['first']=3D'Florent';
set Registry['/user/florent']['country']=3D'France';
set Registry['/version']['1.0.11']=3D'done';
EOF

cassandra-cli -h localhost -k m2mp <<EOF
list Registry;
exit;
EOF

dpkg -i cassandra_1.1.2_all.deb

wait_for_server;

cassandra-cli -h localhost -k m2mp <<EOF
set Registry['/version']['1.1.2']=3D'done';
list Registry;
exit;
EOF

dpkg -i cassandra_1.1.3_all.deb

wait_for_server;

cassandra-cli -h localhost -k m2mp <<EOF
set Registry['/version']['1.1.3']=3D'done';
list Registry;
exit;
EOF

service cassandra restart

wait_for_server;

cassandra-cli -h localhost -k m2mp <<EOF
list Registry;
exit;
EOF
{code}

In the end I do have:
{quote}
-------------------
RowKey: /user/florent
=3D> (column=3Dcountry, value=3DFrance, timestamp=3D1345161343036000)
=3D> (column=3Dfirst, value=3DFlorent, timestamp=3D1345161342992000)
-------------------
RowKey: /version
=3D> (column=3D1.0.11, value=3Ddone, timestamp=3D1345161343039000)
=3D> (column=3D1.1.2, value=3Ddone, timestamp=3D1345161366935000)
=3D> (column=3D1.1.3, value=3Ddone, timestamp=3D1345161389760000)
{quote}

So it's ok. But I would be pretty interested to see if we get the same resu=
lt if we don't skip any version.
               =20
      was (Author: superfc):
    This bug is marked as resolved, so we're just documenting something tha=
t never happened. We're not scaring anyone here, we're making sure we have =
all the documentation to prove that I we were wrong.

So just to make things clear, I didn't make any kind of change or deletion =
on my keyspaces. The two keyspaces were created by code (one with pelops an=
d one with hector) once and never changed. I know I told I did it with cass=
andra-cli earlier but it turns out that it was entirely by code.

While doing some tests, I did delete the keyspaces and in that cases it giv=
es an error that looks like: "Commit logs for non-existing Column Family 10=
36 were ignored" (I can't find the exact error in my logs).=20

When I deleted the keyspace files, they were recreated by reading the commi=
t logs (this is step 4 in my previous report). So I think they were in acco=
rdance with the schema stored in cassandra.

---=20

I wanted to actually test it.

The only last versions I could find were 1.0.11, 1.1.2 and 1.1.3. I created=
 a small testscript and it definitely works with them. But it would be good=
 to test it with 1.1.1 (which I didn't have good access to) also.

{code}
#!/bin/sh
apt-get remove --purge cassandra -y
rm -Rf /var/log/cassandra /var/lib/cassandra

if [ ! -f cassandra_1.0.11_all.deb ]; then
  wget http://apache.mirrors.multidist.eu/cassandra/debian/pool/main/c/cass=
andra/cassandra_1.0.11_all.deb
fi

if [ ! -f cassandra_1.1.2_all.deb ]; then
   wget http://apache.mirrors.multidist.eu/cassandra/debian/pool/main/c/cas=
sandra/cassandra_1.1.2_all.deb
fi

if [ ! -f cassandra_1.1.3_all.deb ]; then
  wget http://apache.mirrors.multidist.eu/cassandra/debian/pool/main/c/cass=
andra/cassandra_1.1.3_all.deb
fi

wait_for_server() {
  while ! echo exit | nc localhost 9160; do sleep 1; done
}

dpkg -i cassandra_1.0.11_all.deb
tail -f /var/log/cassandra/output.log &

wait_for_server;

cassandra-cli -h localhost <<EOF

create keyspace m2mp;
use m2mp;

create column family Registry
  with column_type =3D 'Standard'
  and comparator =3D 'AsciiType'
  and default_validation_class =3D 'AsciiType'
  and key_validation_class =3D 'AsciiType';

set Registry['/user/florent']['first']=3D'Florent';
set Registry['/user/florent']['country']=3D'France';
set Registry['/version']['1.0.11']=3D'done';
EOF

cassandra-cli -h localhost -k m2mp <<EOF
list Registry;
exit;
EOF

dpkg -i cassandra_1.1.2_all.deb

wait_for_server;

cassandra-cli -h localhost -k m2mp <<EOF
set Registry['/version']['1.1.2']=3D'done';
list Registry;
exit;
EOF

dpkg -i cassandra_1.1.3_all.deb

wait_for_server;

cassandra-cli -h localhost -k m2mp <<EOF
set Registry['/version']['1.1.3']=3D'done';
list Registry;
exit;
EOF

service cassandra restart

wait_for_server;

cassandra-cli -h localhost -k m2mp <<EOF
list Registry;
exit;
EOF
{code}

In the end I do have:
{quote}
-------------------
RowKey: /user/florent
=3D> (column=3Dcountry, value=3DFrance, timestamp=3D1345161343036000)
=3D> (column=3Dfirst, value=3DFlorent, timestamp=3D1345161342992000)
-------------------
RowKey: /version
=3D> (column=3D1.0.11, value=3Ddone, timestamp=3D1345161343039000)
=3D> (column=3D1.1.2, value=3Ddone, timestamp=3D1345161366935000)
=3D> (column=3D1.1.3, value=3Ddone, timestamp=3D1345161389760000)
{quote}

So it's ok. But I would be pretty interested to see if we get the same resu=
lt if we don't skip any version.
                 =20
> Commitlog not replayed after restart - data lost
> ------------------------------------------------
>
>                 Key: CASSANDRA-4481
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-4481
>             Project: Cassandra
>          Issue Type: Bug
>    Affects Versions: 1.1.2
>         Environment: Single node cluster on 64Bit CentOS
>            Reporter: Ivo Mei=C3=9Fner
>            Priority: Critical
>
> When data is written to the commitlog and I restart the machine, all comm=
ited data is lost that has not been flushed to disk.=20
> In the startup logs it says that it replays the commitlog successfully, b=
ut the data is not available then.=20
> When I open the commitlog file in an editor I can see the added data, but=
 after the restart it cannot be fetched from cassandra.=20
> {code}
>  INFO 09:59:45,362 Replaying /var/myproject/cassandra/commitlog/CommitLog=
-83203377067.log
>  INFO 09:59:45,476 Finished reading /var/myproject/cassandra/commitlog/Co=
mmitLog-83203377067.log
>  INFO 09:59:45,476 Log replay complete, 0 replayed mutations
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrato=
rs: https://issues.apache.org/jira/secure/ContactAdministrators!default.jsp=
a
For more information on JIRA, see: http://www.atlassian.com/software/jira