Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 38D76636F for ; Tue, 12 Jul 2011 03:29:34 +0000 (UTC) Received: (qmail 53268 invoked by uid 500); 12 Jul 2011 03:29:31 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 52713 invoked by uid 500); 12 Jul 2011 03:29:23 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 52278 invoked by uid 99); 12 Jul 2011 03:29:22 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 12 Jul 2011 03:29:22 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of jwang@palantir.com designates 206.188.26.34 as permitted sender) Received: from [206.188.26.34] (HELO mx2.palantir.com) (206.188.26.34) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 12 Jul 2011 03:29:16 +0000 Received: from EX01-WEST.YOJOE.local ([fe80::c920:deb2:ea80:872d]) by ex01-west.YOJOE.local ([fe80::c920:deb2:ea80:872d%23]) with mapi id 14.01.0289.008; Mon, 11 Jul 2011 20:28:55 -0700 From: Jeffrey Wang To: "user@cassandra.apache.org" Subject: commitlog replay missing data Thread-Topic: commitlog replay missing data Thread-Index: AcxAQ8GvXn5QjWUPQVGTQ45hsqiPTg== Date: Tue, 12 Jul 2011 03:28:54 +0000 Message-ID: <008BB2800AE49940817156C538647C930A0E1A57@ex01-west.YOJOE.local> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: yes X-MS-TNEF-Correlator: x-originating-ip: [10.100.23.216] Content-Type: multipart/signed; protocol="application/x-pkcs7-signature"; micalg=SHA1; boundary="----=_NextPart_000_09AA_01CC4009.156A67C0" MIME-Version: 1.0 ------=_NextPart_000_09AA_01CC4009.156A67C0 Content-Type: multipart/alternative; boundary="----=_NextPart_001_09AB_01CC4009.156A67C0" ------=_NextPart_001_09AB_01CC4009.156A67C0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Hey all, Recently upgraded to 0.8.1 and noticed what seems to be missing data after a commitlog replay on a single-node cluster. I start the node, insert a bunch of stuff (~600MB), stop it, and restart it. There are log messages pertaining to the commitlog replay and no errors, but some of the data is missing. If I flush before stopping the node, everything is fine, and running cfstats in the two cases shows different amounts of data in the SSTables. Moreover, the amount of data that is missing is nondeterministic. Has anyone run into this? Thanks. Here is the output of a side-by-side diff between cfstats outputs for a single CF before restarting (left) and after (right). Somehow a 37MB memtable became a 2.9MB SSTable (note the difference in write count as well)? Column Family: Blocks Column Family: Blocks SSTable count: 0 | SSTable count: 1 Space used (live): 0 | Space used (live): 2907637 Space used (total): 0 | Space used (total): 2907637 Memtable Columns Count: 8198 | Memtable Columns Count: 0 Memtable Data Size: 37550510 | Memtable Data Size: 0 Memtable Switch Count: 0 | Memtable Switch Count: 1 Read Count: 0 Read Count: 0 Read Latency: NaN ms. Read Latency: NaN ms. Write Count: 8198 | Write Count: 1526 Write Latency: 0.018 ms. | Write Latency: 0.011 ms. Pending Tasks: 0 Pending Tasks: 0 Key cache capacity: 200000 Key cache capacity: 200000 Key cache size: 0 Key cache size: 0 Key cache hit rate: NaN Key cache hit rate: NaN Row cache: disabled Row cache: disabled Compacted row minimum size: 0 | Compacted row minimum size: 1110 Compacted row maximum size: 0 | Compacted row maximum size: 2299 Compacted row mean size: 0 | Compacted row mean size: 1960 Note that I patched https://issues.apache.org/jira/browse/CASSANDRA-2317 in my version, but there are no deletions involved so I don't think it's relevant unless I messed something up while patching. -Jeffrey ------=_NextPart_001_09AB_01CC4009.156A67C0 Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

Hey = all,

 

Recently upgraded to 0.8.1 and noticed what seems to = be missing data after a commitlog replay on a single-node cluster. I = start the node, insert a bunch of stuff (~600MB), stop it, and restart = it. There are log messages pertaining to the commitlog replay and no = errors, but some of the data is missing. If I flush before stopping the = node, everything is fine, and running cfstats in the two cases shows = different amounts of data in the SSTables. Moreover, the amount of data = that is missing is nondeterministic. Has anyone run into this? = Thanks.

 

Here is the output of a side-by-side diff between = cfstats outputs for a single CF before restarting (left) and after = (right). Somehow a 37MB memtable became a 2.9MB SSTable (note the = difference in write count as well)?

 

Column Family: = Blocks           &= nbsp;  =             &= nbsp;           &n= bsp;    Column Family: = Blocks

SSTable count: = 0            =             &= nbsp;     = |            =      SSTable count: 1

Space used = (live): = 0            =             &= nbsp; = |            =      Space used (live): = 2907637

Space used (total): = 0            =    =           |  =             &= nbsp;  Space used (total): 2907637

Memtable Columns = Count: = 8198           &nb= sp;      = |            =      Memtable Columns Count: = 0

Memtable Data Size: = 37550510           = ;       = |            =      Memtable Data Size: 0

Memtable Switch = Count: = 0            =           = |            =      Memtable Switch Count: = 1

Read Count: = 0            =             &= nbsp;           &n= bsp;           &nb= sp;  Read Count: 0

Read Latency: NaN = ms.            =             &= nbsp;           &n= bsp;      Read Latency: NaN = ms.

Write Count: = 8198           &nb= sp;           &nbs= p;     = |            =      Write Count: 1526

Write Latency: = 0.018 = ms.           &nbs= p;          = |            =      Write Latency: 0.011 = ms.

Pending Tasks: = 0            =             &= nbsp;           &n= bsp;           Pending = Tasks: 0

Key cache capacity: = 200000           &= nbsp;           &n= bsp;           &nb= sp;  Key cache capacity: 200000

Key cache size: = 0            =        =             &= nbsp;           &n= bsp;   Key cache size: 0

Key cache hit = rate: = NaN           &nbs= p;            = ;            =      Key cache hit rate: NaN

Row cache: = disabled           = ;            =             &= nbsp;         Row cache: = disabled

Compacted row minimum size: = 0            =      = |            =      Compacted row minimum size: = 1110

Compacted row maximum size: = 0            =      = |            =      Compacted row maximum size: = 2299

Compacted row mean size: = 0            =         = |            =      Compacted row mean size: = 1960

 

Note that I patched https://iss= ues.apache.org/jira/browse/CASSANDRA-2317 in my version, but there = are no deletions involved so I don’t think it’s relevant = unless I messed something up while patching.

 

-Jeffrey

=

------=_NextPart_001_09AB_01CC4009.156A67C0-- ------=_NextPart_000_09AA_01CC4009.156A67C0 Content-Type: application/x-pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIPkTCCBDIw ggMaoAMCAQICAQEwDQYJKoZIhvcNAQEFBQAwezELMAkGA1UEBhMCR0IxGzAZBgNVBAgMEkdyZWF0 ZXIgTWFuY2hlc3RlcjEQMA4GA1UEBwwHU2FsZm9yZDEaMBgGA1UECgwRQ29tb2RvIENBIExpbWl0 ZWQxITAfBgNVBAMMGEFBQSBDZXJ0aWZpY2F0ZSBTZXJ2aWNlczAeFw0wNDAxMDEwMDAwMDBaFw0y ODEyMzEyMzU5NTlaMHsxCzAJBgNVBAYTAkdCMRswGQYDVQQIDBJHcmVhdGVyIE1hbmNoZXN0ZXIx EDAOBgNVBAcMB1NhbGZvcmQxGjAYBgNVBAoMEUNvbW9kbyBDQSBMaW1pdGVkMSEwHwYDVQQDDBhB QUEgQ2VydGlmaWNhdGUgU2VydmljZXMwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC+ QJ30buHqdoccTUVEjr5GyIMGncEq/hgfjuQC+vOrXVCKFjELmgbQxXAizUktVGPMtm5oRgtT6stM JMC8ck7q8RWu9FSaEgrDerIzYOLaiVXzIljz3tzP74OGooyUT59o8piQRoQnx3a/48w1LIteB2Rl gsBIsKiR+WGfdiBQqJHHZrXreGIDVvCKGhPqMaMeoJn9OPb2JzJYbwf1a7j7FCuvt6rM1mNfc4za BZmoOKjLF3g2UazpnvR4Oo3PD9lC4pgMqy+fDgHe75+ZSfEt36x0TRuYtUfF5SnR+ZAYx2KcvoPH Jns+iiXHwN2d5jVoECCdj9je0sOEnA1e6C/JAgMBAAGjgcAwgb0wHQYDVR0OBBYEFKARCiM+lvEH 7OKvKe+CpX/QMKS0MA4GA1UdDwEB/wQEAwIBBjAPBgNVHRMBAf8EBTADAQH/MHsGA1UdHwR0MHIw OKA2oDSGMmh0dHA6Ly9jcmwuY29tb2RvY2EuY29tL0FBQUNlcnRpZmljYXRlU2VydmljZXMuY3Js MDagNKAyhjBodHRwOi8vY3JsLmNvbW9kby5uZXQvQUFBQ2VydGlmaWNhdGVTZXJ2aWNlcy5jcmww DQYJKoZIhvcNAQEFBQADggEBAAhW/ALwm+j/pPrWe8ZEgM5PxMX2AFjMpra8FEloBHbo5u5d7AIP YNaNUBhPJk4B4+awpe6/vHRUQb/9/BK4x09a9IlgBX9gtwVK8/bxwr/EuXSGti19a8zS80bdL8bg asPDNAMsfZbdWsIOpwqZwQWLqwwv81w6z2w3VQmH3lNAbFjv/LarZW4E9hvcPOBaFcae2fFZSDAh ZQNs7Okhc+ybA6HgN62gFRiP+roCzqcsqRATLNTlCCarIpdg+JBedNSimlO98qlo4KJuwtdssaMP nr/raOdW8q7y4ys4OgmBtWuF174t7T8at7Jj4vViLILUagBBUPE5g5+V6TaWmG4wggTdMIIDxaAD AgECAhBxkvvmGV+sTRKFdHE0ohinMA0GCSqGSIb3DQEBBQUAMHsxCzAJBgNVBAYTAkdCMRswGQYD VQQIDBJHcmVhdGVyIE1hbmNoZXN0ZXIxEDAOBgNVBAcMB1NhbGZvcmQxGjAYBgNVBAoMEUNvbW9k byBDQSBMaW1pdGVkMSEwHwYDVQQDDBhBQUEgQ2VydGlmaWNhdGUgU2VydmljZXMwHhcNMDQwMTAx MDAwMDAwWhcNMjgxMjMxMjM1OTU5WjCBrjELMAkGA1UEBhMCVVMxCzAJBgNVBAgTAlVUMRcwFQYD VQQHEw5TYWx0IExha2UgQ2l0eTEeMBwGA1UEChMVVGhlIFVTRVJUUlVTVCBOZXR3b3JrMSEwHwYD VQQLExhodHRwOi8vd3d3LnVzZXJ0cnVzdC5jb20xNjA0BgNVBAMTLVVUTi1VU0VSRmlyc3QtQ2xp ZW50IEF1dGhlbnRpY2F0aW9uIGFuZCBFbWFpbDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoC ggEBALI5haTyfatBO2JGN67NwWB1vDll+UoaR6K5zEjMapjVTTUZuaRC5c5J4oovHnzSMQfHTrSD ZJ0uKdWiZMSFvYVRNXmkTmiQexx6pJKoF/KYFfKTzMmkMpW7DE8wvZigC4vlbhuiRvp4vKJvq1le pS/Pytptqi/rrKGzaqq3Lmc1i3nhHmmI4uZGzaCl6r4LznY6eg6b6vzaJ1s9cx8i5khhxkzzabGo Lhu21DEgLLyCio6kDqXXiUP8FlqvHXHXEVnauocNr/rz4cLwpMVnjNbWVDreCqS6A3ezZcj9HtN0 YqoYymiTHqGFfvVHZcv4TVcodNI0/zC27vZiMBSMLOsCAwEAAaOCAScwggEjMB8GA1UdIwQYMBaA FKARCiM+lvEH7OKvKe+CpX/QMKS0MB0GA1UdDgQWBBSJgmd9xJ0mcABLtFBIfN49rgRufTAOBgNV HQ8BAf8EBAMCAQYwDwYDVR0TAQH/BAUwAwEB/zAdBgNVHSUEFjAUBggrBgEFBQcDAgYIKwYBBQUH AwQwEQYDVR0gBAowCDAGBgRVHSAAMHsGA1UdHwR0MHIwOKA2oDSGMmh0dHA6Ly9jcmwuY29tb2Rv Y2EuY29tL0FBQUNlcnRpZmljYXRlU2VydmljZXMuY3JsMDagNKAyhjBodHRwOi8vY3JsLmNvbW9k by5uZXQvQUFBQ2VydGlmaWNhdGVTZXJ2aWNlcy5jcmwwEQYJYIZIAYb4QgEBBAQDAgEGMA0GCSqG SIb3DQEBBQUAA4IBAQCdlcs8uH6lCcQevwvCx3aOOTyUxhCqTwzJ4KuEXYlU4GU7820cfDcsJVRf liH8N4SRnRXcFE+Bz1Qda2xFYMct+ZdRTPlmyjyggoymyPDi6dRK+ew/VsnddozDggFPbADzHhph dARHA6nGQFeRvGUixSdnT1fbZFrZjR+6hi/0Bq6cae3p9M8pF9jgSp8aIC+XTFG7RgfEijdOIOMJ MWjHnsSLneh+EbwyaBCWEZhE2CpRYE2I63Q630MGMsg5Vow6EVLTQaRDA/Tt7zMn2zngFE4mydj1 OeKJuJNdtykmQeqzm66D/Hd1yujKtf7iZUpjPkTE0MNeh3OpmByvfxV/MIIGdjCCBV6gAwIBAgIR APam6MORK2txy64lXxarT88wDQYJKoZIhvcNAQEFBQAwga4xCzAJBgNVBAYTAlVTMQswCQYDVQQI EwJVVDEXMBUGA1UEBxMOU2FsdCBMYWtlIENpdHkxHjAcBgNVBAoTFVRoZSBVU0VSVFJVU1QgTmV0 d29yazEhMB8GA1UECxMYaHR0cDovL3d3dy51c2VydHJ1c3QuY29tMTYwNAYDVQQDEy1VVE4tVVNF UkZpcnN0LUNsaWVudCBBdXRoZW50aWNhdGlvbiBhbmQgRW1haWwwHhcNMTAwNjIxMDAwMDAwWhcN MTMwNjIwMjM1OTU5WjCCAToxCzAJBgNVBAYTAlVTMQ4wDAYDVQQREwU5NDMwMTETMBEGA1UECBMK Q2FsaWZvcm5pYTESMBAGA1UEBxMJUGFsbyBBbHRvMRIwEAYDVQQJEwlTdWl0ZSAzMDAxGTAXBgNV BAkTEDEwMCBIYW1pbHRvbiBBdmUxHjAcBgNVBAoTFVBhbGFudGlyIFRlY2hub2xvZ2llczELMAkG A1UECxMCSVQxOzA5BgNVBAsTMklzc3VlZCB0aHJvdWdoIFBhbGFudGlyIFRlY2hub2xvZ2llcyBF LVBLSSBNYW5hZ2VyMR8wHQYDVQQLExZDb3Jwb3JhdGUgU2VjdXJlIEVtYWlsMRUwEwYDVQQDEwxK ZWZmcmV5IFdhbmcxITAfBgkqhkiG9w0BCQEWEmp3YW5nQHBhbGFudGlyLmNvbTCCASIwDQYJKoZI hvcNAQEBBQADggEPADCCAQoCggEBALiovezdrKoQI/GrBxajvik5PkYE1kM0UvkWRV1UKyyeiaxL GQ9ubaZ5jeUd8jyWqWYzTl+do5+CMqgMlOxTH895939jYxLKoXQCp/oByQqaX/Eet02psNLFyyH7 HNSJiOA7v5SQKrYFY7T2Fub37FSBXlK5hodAvKTf0TCYkHh1M6xZ7J9gwI3lfcEsENlnt54qW95m nw/16M3zkzZGI9jTgNBdqRnTekNFqr/1oi/HO7zqE9BUGj5oBZwbfMMb1eDHB9B8PLe/WZmJE95H AW5z36E4QQYhnmbOQHBbQZu5gMnEYEdQbkX9XvfKFsje9msYIbQuIU9rJEtnNdZTrjMCAwEAAaOC Af4wggH6MB8GA1UdIwQYMBaAFImCZ33EnSZwAEu0UEh83j2uBG59MB0GA1UdDgQWBBQ7oOnNttsD PfvuWdDwNh1K9dwc6TAOBgNVHQ8BAf8EBAMCBaAwDAYDVR0TAQH/BAIwADAdBgNVHSUEFjAUBggr BgEFBQcDBAYIKwYBBQUHAwIwRgYDVR0gBD8wPTA7BgwrBgEEAbIxAQIBAwUwKzApBggrBgEFBQcC ARYdaHR0cHM6Ly9zZWN1cmUuY29tb2RvLm5ldC9DUFMwgaUGA1UdHwSBnTCBmjBMoEqgSIZGaHR0 cDovL2NybC5jb21vZG9jYS5jb20vVVROLVVTRVJGaXJzdC1DbGllbnRBdXRoZW50aWNhdGlvbmFu ZEVtYWlsLmNybDBKoEigRoZEaHR0cDovL2NybC5jb21vZG8ubmV0L1VUTi1VU0VSRmlyc3QtQ2xp ZW50QXV0aGVudGljYXRpb25hbmRFbWFpbC5jcmwwbAYIKwYBBQUHAQEEYDBeMDYGCCsGAQUFBzAC hipodHRwOi8vY3J0LmNvbW9kb2NhLmNvbS9VVE5BQUFDbGllbnRDQS5jcnQwJAYIKwYBBQUHMAGG GGh0dHA6Ly9vY3NwLmNvbW9kb2NhLmNvbTAdBgNVHREEFjAUgRJqd2FuZ0BwYWxhbnRpci5jb20w DQYJKoZIhvcNAQEFBQADggEBAG3u36M7zrIP/s8uR69W2Y/kiOOXM3h16NbQN68gXCkGfCEhz9Tz rKK21r1ZkWSQtH4ppezbpVvmo/uCWyKorzQvc8phXNPtD5qMO0sSqlsbZgso377x24OQP6wVCybL fjH+lFV9ofnFxMBgbDsjJxZnPJd6p1iPT+W8+89RkVje/PZtv7HjDkbwMxZckDlXF6S+xV9v956G eN1sZ5g9K25VMbmd8U9QNgqNmQhNCztrcU6rOBt7vvY1fMOSua5g0s/l2ORsvzrcd7+pDOLjgqRO KS9nHYT9zdW/7ZGQCdDk2R8vXfe39pEC72PJwZ1hXyJjmInMaAqCDLO6HzFrfyQxggRoMIIEZAIB ATCBxDCBrjELMAkGA1UEBhMCVVMxCzAJBgNVBAgTAlVUMRcwFQYDVQQHEw5TYWx0IExha2UgQ2l0 eTEeMBwGA1UEChMVVGhlIFVTRVJUUlVTVCBOZXR3b3JrMSEwHwYDVQQLExhodHRwOi8vd3d3LnVz ZXJ0cnVzdC5jb20xNjA0BgNVBAMTLVVUTi1VU0VSRmlyc3QtQ2xpZW50IEF1dGhlbnRpY2F0aW9u IGFuZCBFbWFpbAIRAPam6MORK2txy64lXxarT88wCQYFKw4DAhoFAKCCAngwGAYJKoZIhvcNAQkD MQsGCSqGSIb3DQEHATAcBgkqhkiG9w0BCQUxDxcNMTEwNzEyMDMyODI0WjAjBgkqhkiG9w0BCQQx FgQU/oeZh/rn37EWzKA/HJFHYz/qE+IwZwYJKoZIhvcNAQkPMVowWDAKBggqhkiG9w0DBzAOBggq hkiG9w0DAgICAIAwDQYIKoZIhvcNAwICAUAwBwYFKw4DAgcwDQYIKoZIhvcNAwICASgwBwYFKw4D AhowCgYIKoZIhvcNAgUwgdUGCSsGAQQBgjcQBDGBxzCBxDCBrjELMAkGA1UEBhMCVVMxCzAJBgNV BAgTAlVUMRcwFQYDVQQHEw5TYWx0IExha2UgQ2l0eTEeMBwGA1UEChMVVGhlIFVTRVJUUlVTVCBO ZXR3b3JrMSEwHwYDVQQLExhodHRwOi8vd3d3LnVzZXJ0cnVzdC5jb20xNjA0BgNVBAMTLVVUTi1V U0VSRmlyc3QtQ2xpZW50IEF1dGhlbnRpY2F0aW9uIGFuZCBFbWFpbAIRAPam6MORK2txy64lXxar T88wgdcGCyqGSIb3DQEJEAILMYHHoIHEMIGuMQswCQYDVQQGEwJVUzELMAkGA1UECBMCVVQxFzAV BgNVBAcTDlNhbHQgTGFrZSBDaXR5MR4wHAYDVQQKExVUaGUgVVNFUlRSVVNUIE5ldHdvcmsxITAf BgNVBAsTGGh0dHA6Ly93d3cudXNlcnRydXN0LmNvbTE2MDQGA1UEAxMtVVROLVVTRVJGaXJzdC1D bGllbnQgQXV0aGVudGljYXRpb24gYW5kIEVtYWlsAhEA9qbow5Era3HLriVfFqtPzzANBgkqhkiG 9w0BAQEFAASCAQAPM2Z922gytj82tkhJ70sr1i6wRJgVEuM4j1pucDjHUpQ+Am99DUs2arM/GOqH vAzalBTldi4goEoDM+fZZhUZZxAysTwp7mob74ZPev0vWRNnh9t7LJcdEK/NbZQ3YxOFTIxt8HiE HffEH7uvBOfejpLnDzXDGZE6S3O6ekk9DLXrwf4KhnKeNAtLuojk1wd6AGHrRvv1nLCVIambYtfZ P+LeW3XS3vyehqHEGUoLtXA2tzEpMQUskX/vaTs0wQuAr3aScfvRHayVUCQwbK07gN27WtxTbkAo xqzMugouSvg9lGy0f3YTdJPu3eTSlScmVQvbGYi3t9Zj6XDF4H/SAAAAAAAA ------=_NextPart_000_09AA_01CC4009.156A67C0--