Mailing-List: contact commits-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@cassandra.apache.org
Date: Fri, 12 Oct 2012 23:43:03 +0000 (UTC)
From: "Florent Clairambault (JIRA)" <jira@apache.org>
To: commits@cassandra.apache.org
Message-ID: <1675457678.40223.1350085383165.JavaMail.jiratomcat@arcas>
In-Reply-To: <1624491622.4291.1343895182925.JavaMail.jiratomcat@issues-vm>
Subject: [jira] [Updated] (CASSANDRA-4481) Commitlog not replayed after
 restart - data lost
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable


     [ https://issues.apache.org/jira/browse/CASSANDRA-4481?page=3Dcom.atla=
ssian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Florent Clairambault updated CASSANDRA-4481:
--------------------------------------------

    Comment: was deleted

(was: I doesn't work, it failed again a week ago on a 1.1.5 that was runnin=
g for a little bit.

First of all, it's a commitLog writing and/or reading issue, so if you flus=
h your data frequently (every hour and in the stop command of the rc.d's sc=
ript) you reduce your risk of big data losses. You can lose days of data if=
 you don't do that. Restarting cassandra and going 2 days in the past is a =
very unpleasant situation.

So here is the new process I applied to fix my data (which is in fact resta=
rting from scatch [except we keep the data]):
- Export the keyspace's schema
{code}
cassandra-cli -k ks >schema.txt <<EOF=20
show schema;
exit;
EOF
{code}
- Simplify the export (all CF with key_validation_class in AsciiType, defau=
lt_validation_class in UTF8Type for most CF except the one that contains bi=
nary data where I used BytesTypes).

I simplify an export like that:
{code}
create column family User
  with column_type =3D 'Standard'
  and comparator =3D 'AsciiType'
  and default_validation_class =3D 'UTF8Type'
  and key_validation_class =3D 'AsciiType'
  and read_repair_chance =3D 0.1
  and dclocal_read_repair_chance =3D 0.0
  and gc_grace =3D 864000
  and min_compaction_threshold =3D 4
  and max_compaction_threshold =3D 32
  and replicate_on_write =3D true
  and compaction_strategy =3D 'org.apache.cassandra.db.compaction.SizeTiere=
dCompactionStrategy'
  and caching =3D 'KEYS_ONLY'
  and column_metadata =3D [
    {column_name : 'domain',
    validation_class : UTF8Type,
    index_name : 'User_domain_idx',
    index_type : 0},
    {column_name : 'username',
    validation_class : UTF8Type,
    index_name : 'User_username',
    index_type : 0}]
  and compression_options =3D {'sstable_compression' : 'org.apache.cassandr=
a.io.compress.SnappyCompressor'};
{code}

To something like that:
{code}
create column family User
  with column_type =3D 'Standard'
  and key_validation_class =3D 'AsciiType'
  and comparator =3D 'AsciiType'
  and default_validation_class =3D 'UTF8Type'
  and column_metadata =3D [
    {column_name : 'domain', validation_class : UTF8Type, index_type : 0},
    {column_name : 'username', validation_class : UTF8Type, index_type : 0}=
];
{code}

During this simplification process, I discovered that some default_validati=
on_class had incorrect type, so maybe it comes from that. It seems strange =
that we could "confuse" cassandra this way, but this problem is indeed very=
 strange...

- Stop cassandra
- Move the keyspace folder to somewhere else (mkdir backup; mv <ks> backup)
- Start cassandra (Not having a keyspace folder is like not having any data=
, it's not a problem).
- Delete the keyspace (I know deletion creates snapshots and moving is unec=
essary but it's easier to use sstableloader that way)
- Recreate the keyspace with the schema exported and simplified
- Use sstableloader to import data:
{code}
cd backup; find <ks> -type d -exec sstableloader -d localhost {} \;
{code}

NOTE: Don't think about replaying your commitLogs with your new schema, the=
 column families won't have the same id.

Any empty cassandra instance startup does at least 1 mutation replay becaus=
e of the "system" keyspace. So I still think 0 replayed mutations should ne=
ver occur and if they do, we should have some warning with them. And if it'=
s indeed "a CF that doesn't fully exist", it should be reported at startup.

I hope we can find a way to reproduce it.)
   =20
> Commitlog not replayed after restart - data lost
> ------------------------------------------------
>
>                 Key: CASSANDRA-4481
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-4481
>             Project: Cassandra
>          Issue Type: Bug
>    Affects Versions: 1.1.2
>         Environment: Single node cluster on 64Bit CentOS
>            Reporter: Ivo Mei=C3=9Fner
>            Priority: Critical
>
> When data is written to the commitlog and I restart the machine, all comm=
ited data is lost that has not been flushed to disk.=20
> In the startup logs it says that it replays the commitlog successfully, b=
ut the data is not available then.=20
> When I open the commitlog file in an editor I can see the added data, but=
 after the restart it cannot be fetched from cassandra.=20
> {code}
>  INFO 09:59:45,362 Replaying /var/myproject/cassandra/commitlog/CommitLog=
-83203377067.log
>  INFO 09:59:45,476 Finished reading /var/myproject/cassandra/commitlog/Co=
mmitLog-83203377067.log
>  INFO 09:59:45,476 Log replay complete, 0 replayed mutations
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrato=
rs
For more information on JIRA, see: http://www.atlassian.com/software/jira