cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Florent Clairambault (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (CASSANDRA-4481) Commitlog not replayed after restart - data lost
Date Wed, 15 Aug 2012 23:10:38 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-4481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13435597#comment-13435597
] 

Florent Clairambault edited comment on CASSANDRA-4481 at 8/16/12 10:09 AM:
---------------------------------------------------------------------------

So, I didn't find (or tried to find) a solution to reproduce this bug. But I found an (incomplete)
fix. I'm on Debian/6.0.5, still with Cassandra/1.1.3:

For a keyspace named "dom2dom" (so that I don't have to replace any name).

# In my case I removed all the commitlog files that were created prior to 1.1.3

# 1. Flush cassandra
nodetool flush

# 2. Stop cassandra
service cassandra stop

# 3. Move the sstable files to an other directory
mkdir /var/lib/cassandra/toload
mv /var/lib/cassandra/data/dom2dom /var/lib/cassandra/toload/m2mp

# In my case, I had to create a 127.0.0.2 loopback interface 
# and update the cassandra.yaml file to change rpc_address and listen_address settings 
# to 127.0.0.2 so that sstableloader could work.

# 4. Start cassandra
service cassandra stop

# At that point the commitlogs should work again and you should have some new sstable created
du -sh /var/lib/cassandra/dom2dom
# Returns: 236K

# You now have the new data and not the old one, so you need to load the old data using sstableloader:
find /var/lib/cassandra/toload/dom2dom/ -type d -exec sstableloader -d 127.0.0.2 {} \;

# In my case, I had to put back localhost in the cassandra.yaml for the rpc_address and listen_address
settings

# You can delete the /var/lib/cassandra/toload folder

IMPORTANT NOTE:
I'm not sure that putting back the old (prior to 1.1.3) commitlog files will work. From what
I've quickly tested it doesn't.

Still, I think there's should be more information in the logs to get what is happening. It
seems very strange that the system could get stuck like this without any a single error message.


                
      was (Author: superfc):
    So, I didn't find (or tried to find) a solution to reproduce this bug. But I found a fix.
I'm on Debian/6.0.5, still with Cassandra/1.1.3:

For a keyspace named "dom2dom" (so that I don't have to replace any name).

# In my case I removed all the commitlog files that were created prior to 1.1.3

# 1. Flush cassandra
nodetool flush

# 2. Stop cassandra
service cassandra stop

# 3. Move the sstable files to an other directory
mkdir /var/lib/cassandra/toload
mv /var/lib/cassandra/data/dom2dom /var/lib/cassandra/toload/m2mp

# In my case, I had to create a 127.0.0.2 loopback interface 
# and update the cassandra.yaml file to change rpc_address and listen_address settings 
# to 127.0.0.2 so that sstableloader could work.

# 4. Start cassandra
service cassandra stop

# At that point the commitlogs should work again and you should have some new sstable created
du -sh /var/lib/cassandra/dom2dom
# Returns: 236K

# You now have the new data and not the old one, so you need to load the old data using sstableloader:
find /var/lib/cassandra/toload/dom2dom/ -type d -exec sstableloader -d 127.0.0.2 {} \;

# In my case, I had to put back localhost in the cassandra.yaml for the rpc_address and listen_address
settings

# You can delete the /var/lib/cassandra/toload folder

IMPORTANT NOTE:
I'm not sure that putting back the old (prior to 1.1.3) commitlog files will work. From what
I've quickly tested it doesn't.

Still, I think there's should be more information in the logs to get what is happening. It
seems very strange that the system could get stuck like this without any a single error message.


                  
> Commitlog not replayed after restart - data lost
> ------------------------------------------------
>
>                 Key: CASSANDRA-4481
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-4481
>             Project: Cassandra
>          Issue Type: Bug
>    Affects Versions: 1.1.2
>         Environment: Single node cluster on 64Bit CentOS
>            Reporter: Ivo Mei├čner
>            Priority: Critical
>
> When data is written to the commitlog and I restart the machine, all commited data is
lost that has not been flushed to disk. 
> In the startup logs it says that it replays the commitlog successfully, but the data
is not available then. 
> When I open the commitlog file in an editor I can see the added data, but after the restart
it cannot be fetched from cassandra. 
> {code}
>  INFO 09:59:45,362 Replaying /var/myproject/cassandra/commitlog/CommitLog-83203377067.log
>  INFO 09:59:45,476 Finished reading /var/myproject/cassandra/commitlog/CommitLog-83203377067.log
>  INFO 09:59:45,476 Log replay complete, 0 replayed mutations
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

       

Mime
View raw message