Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 35E744C9C for ; Thu, 23 Jun 2011 21:55:52 +0000 (UTC) Received: (qmail 84348 invoked by uid 500); 23 Jun 2011 21:55:49 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 84312 invoked by uid 500); 23 Jun 2011 21:55:49 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 84304 invoked by uid 99); 23 Jun 2011 21:55:49 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 21:55:49 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of jwang@palantir.com designates 206.188.26.34 as permitted sender) Received: from [206.188.26.34] (HELO mx2.palantir.com) (206.188.26.34) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 23 Jun 2011 21:55:41 +0000 Received: from EX01-WEST.YOJOE.local ([::1]) by ex01-west.YOJOE.local ([::1]) with mapi id 14.01.0289.008; Thu, 23 Jun 2011 14:55:19 -0700 From: Jeffrey Wang To: "user@cassandra.apache.org" Subject: hinted handoff sleeping Thread-Topic: hinted handoff sleeping Thread-Index: Acwx8Dz17xbw00A4T12jP5OM0RmmOw== Date: Thu, 23 Jun 2011 21:55:19 +0000 Message-ID: <008BB2800AE49940817156C538647C9303FD2957@ex01-west.YOJOE.local> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: yes X-MS-TNEF-Correlator: x-originating-ip: [10.100.23.216] Content-Type: multipart/signed; protocol="application/x-pkcs7-signature"; micalg=SHA1; boundary="----=_NextPart_000_0097_01CC31B5.909B9A20" MIME-Version: 1.0 X-Virus-Checked: Checked by ClamAV on apache.org ------=_NextPart_000_0097_01CC31B5.909B9A20 Content-Type: multipart/alternative; boundary="----=_NextPart_001_0098_01CC31B5.909B9A20" ------=_NextPart_001_0098_01CC31B5.909B9A20 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Hey all, We're running a slightly patched version of 0.7.3 on a cluster of 5 nodes. I've been noticing a number of messages in our logs which look like this (after a node goes "down" and comes back up, usually just due to a GC): 2011-06-23 14:46:35,381 INFO [HintedHandoff:1] org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHa ndOffManager.java:290) {USER='',IP=''} - Sleeping 32649ms to stagger hint delivery The interesting thing is that we have hinted_handoff_enabled = false in the YAML configuration, so it always says 0 rows are handed off (later, after the sleep). Thus this sleeping seems quite wasteful. Is this part of the code supposed to be reach even with hinted handoff disabled? Thanks. -Jeffrey ------=_NextPart_001_0098_01CC31B5.909B9A20 Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

Hey = all,

We’re running a slightly patched version of = 0.7.3 on a cluster of 5 nodes. I’ve been noticing a number of = messages in our logs which look like this (after a node goes = “down” and comes back up, usually just due to a = GC):

2011-06-23 14:46:35,381 INFO [HintedHandoff:1] = org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(Hinte= dHandOffManager.java:290) {USER=3D'',IP=3D''} - Sleeping 32649ms to = stagger hint delivery

The = interesting thing is that we have hinted_handoff_enabled =3D false in = the YAML configuration, so it always says 0 rows are handed off (later, = after the sleep). Thus this sleeping seems quite wasteful. Is this part = of the code supposed to be reach even with hinted handoff disabled? = Thanks.

-Jeffrey

------=_NextPart_001_0098_01CC31B5.909B9A20-- ------=_NextPart_000_0097_01CC31B5.909B9A20 Content-Type: application/x-pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIPkTCCBDIw ggMaoAMCAQICAQEwDQYJKoZIhvcNAQEFBQAwezELMAkGA1UEBhMCR0IxGzAZBgNVBAgMEkdyZWF0 ZXIgTWFuY2hlc3RlcjEQMA4GA1UEBwwHU2FsZm9yZDEaMBgGA1UECgwRQ29tb2RvIENBIExpbWl0 ZWQxITAfBgNVBAMMGEFBQSBDZXJ0aWZpY2F0ZSBTZXJ2aWNlczAeFw0wNDAxMDEwMDAwMDBaFw0y ODEyMzEyMzU5NTlaMHsxCzAJBgNVBAYTAkdCMRswGQYDVQQIDBJHcmVhdGVyIE1hbmNoZXN0ZXIx EDAOBgNVBAcMB1NhbGZvcmQxGjAYBgNVBAoMEUNvbW9kbyBDQSBMaW1pdGVkMSEwHwYDVQQDDBhB QUEgQ2VydGlmaWNhdGUgU2VydmljZXMwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC+ QJ30buHqdoccTUVEjr5GyIMGncEq/hgfjuQC+vOrXVCKFjELmgbQxXAizUktVGPMtm5oRgtT6stM JMC8ck7q8RWu9FSaEgrDerIzYOLaiVXzIljz3tzP74OGooyUT59o8piQRoQnx3a/48w1LIteB2Rl gsBIsKiR+WGfdiBQqJHHZrXreGIDVvCKGhPqMaMeoJn9OPb2JzJYbwf1a7j7FCuvt6rM1mNfc4za BZmoOKjLF3g2UazpnvR4Oo3PD9lC4pgMqy+fDgHe75+ZSfEt36x0TRuYtUfF5SnR+ZAYx2KcvoPH Jns+iiXHwN2d5jVoECCdj9je0sOEnA1e6C/JAgMBAAGjgcAwgb0wHQYDVR0OBBYEFKARCiM+lvEH 7OKvKe+CpX/QMKS0MA4GA1UdDwEB/wQEAwIBBjAPBgNVHRMBAf8EBTADAQH/MHsGA1UdHwR0MHIw OKA2oDSGMmh0dHA6Ly9jcmwuY29tb2RvY2EuY29tL0FBQUNlcnRpZmljYXRlU2VydmljZXMuY3Js MDagNKAyhjBodHRwOi8vY3JsLmNvbW9kby5uZXQvQUFBQ2VydGlmaWNhdGVTZXJ2aWNlcy5jcmww DQYJKoZIhvcNAQEFBQADggEBAAhW/ALwm+j/pPrWe8ZEgM5PxMX2AFjMpra8FEloBHbo5u5d7AIP YNaNUBhPJk4B4+awpe6/vHRUQb/9/BK4x09a9IlgBX9gtwVK8/bxwr/EuXSGti19a8zS80bdL8bg asPDNAMsfZbdWsIOpwqZwQWLqwwv81w6z2w3VQmH3lNAbFjv/LarZW4E9hvcPOBaFcae2fFZSDAh ZQNs7Okhc+ybA6HgN62gFRiP+roCzqcsqRATLNTlCCarIpdg+JBedNSimlO98qlo4KJuwtdssaMP nr/raOdW8q7y4ys4OgmBtWuF174t7T8at7Jj4vViLILUagBBUPE5g5+V6TaWmG4wggTdMIIDxaAD AgECAhBxkvvmGV+sTRKFdHE0ohinMA0GCSqGSIb3DQEBBQUAMHsxCzAJBgNVBAYTAkdCMRswGQYD VQQIDBJHcmVhdGVyIE1hbmNoZXN0ZXIxEDAOBgNVBAcMB1NhbGZvcmQxGjAYBgNVBAoMEUNvbW9k byBDQSBMaW1pdGVkMSEwHwYDVQQDDBhBQUEgQ2VydGlmaWNhdGUgU2VydmljZXMwHhcNMDQwMTAx MDAwMDAwWhcNMjgxMjMxMjM1OTU5WjCBrjELMAkGA1UEBhMCVVMxCzAJBgNVBAgTAlVUMRcwFQYD VQQHEw5TYWx0IExha2UgQ2l0eTEeMBwGA1UEChMVVGhlIFVTRVJUUlVTVCBOZXR3b3JrMSEwHwYD VQQLExhodHRwOi8vd3d3LnVzZXJ0cnVzdC5jb20xNjA0BgNVBAMTLVVUTi1VU0VSRmlyc3QtQ2xp ZW50IEF1dGhlbnRpY2F0aW9uIGFuZCBFbWFpbDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoC ggEBALI5haTyfatBO2JGN67NwWB1vDll+UoaR6K5zEjMapjVTTUZuaRC5c5J4oovHnzSMQfHTrSD ZJ0uKdWiZMSFvYVRNXmkTmiQexx6pJKoF/KYFfKTzMmkMpW7DE8wvZigC4vlbhuiRvp4vKJvq1le pS/Pytptqi/rrKGzaqq3Lmc1i3nhHmmI4uZGzaCl6r4LznY6eg6b6vzaJ1s9cx8i5khhxkzzabGo Lhu21DEgLLyCio6kDqXXiUP8FlqvHXHXEVnauocNr/rz4cLwpMVnjNbWVDreCqS6A3ezZcj9HtN0 YqoYymiTHqGFfvVHZcv4TVcodNI0/zC27vZiMBSMLOsCAwEAAaOCAScwggEjMB8GA1UdIwQYMBaA FKARCiM+lvEH7OKvKe+CpX/QMKS0MB0GA1UdDgQWBBSJgmd9xJ0mcABLtFBIfN49rgRufTAOBgNV HQ8BAf8EBAMCAQYwDwYDVR0TAQH/BAUwAwEB/zAdBgNVHSUEFjAUBggrBgEFBQcDAgYIKwYBBQUH AwQwEQYDVR0gBAowCDAGBgRVHSAAMHsGA1UdHwR0MHIwOKA2oDSGMmh0dHA6Ly9jcmwuY29tb2Rv Y2EuY29tL0FBQUNlcnRpZmljYXRlU2VydmljZXMuY3JsMDagNKAyhjBodHRwOi8vY3JsLmNvbW9k by5uZXQvQUFBQ2VydGlmaWNhdGVTZXJ2aWNlcy5jcmwwEQYJYIZIAYb4QgEBBAQDAgEGMA0GCSqG SIb3DQEBBQUAA4IBAQCdlcs8uH6lCcQevwvCx3aOOTyUxhCqTwzJ4KuEXYlU4GU7820cfDcsJVRf liH8N4SRnRXcFE+Bz1Qda2xFYMct+ZdRTPlmyjyggoymyPDi6dRK+ew/VsnddozDggFPbADzHhph dARHA6nGQFeRvGUixSdnT1fbZFrZjR+6hi/0Bq6cae3p9M8pF9jgSp8aIC+XTFG7RgfEijdOIOMJ MWjHnsSLneh+EbwyaBCWEZhE2CpRYE2I63Q630MGMsg5Vow6EVLTQaRDA/Tt7zMn2zngFE4mydj1 OeKJuJNdtykmQeqzm66D/Hd1yujKtf7iZUpjPkTE0MNeh3OpmByvfxV/MIIGdjCCBV6gAwIBAgIR APam6MORK2txy64lXxarT88wDQYJKoZIhvcNAQEFBQAwga4xCzAJBgNVBAYTAlVTMQswCQYDVQQI EwJVVDEXMBUGA1UEBxMOU2FsdCBMYWtlIENpdHkxHjAcBgNVBAoTFVRoZSBVU0VSVFJVU1QgTmV0 d29yazEhMB8GA1UECxMYaHR0cDovL3d3dy51c2VydHJ1c3QuY29tMTYwNAYDVQQDEy1VVE4tVVNF UkZpcnN0LUNsaWVudCBBdXRoZW50aWNhdGlvbiBhbmQgRW1haWwwHhcNMTAwNjIxMDAwMDAwWhcN MTMwNjIwMjM1OTU5WjCCAToxCzAJBgNVBAYTAlVTMQ4wDAYDVQQREwU5NDMwMTETMBEGA1UECBMK Q2FsaWZvcm5pYTESMBAGA1UEBxMJUGFsbyBBbHRvMRIwEAYDVQQJEwlTdWl0ZSAzMDAxGTAXBgNV BAkTEDEwMCBIYW1pbHRvbiBBdmUxHjAcBgNVBAoTFVBhbGFudGlyIFRlY2hub2xvZ2llczELMAkG A1UECxMCSVQxOzA5BgNVBAsTMklzc3VlZCB0aHJvdWdoIFBhbGFudGlyIFRlY2hub2xvZ2llcyBF LVBLSSBNYW5hZ2VyMR8wHQYDVQQLExZDb3Jwb3JhdGUgU2VjdXJlIEVtYWlsMRUwEwYDVQQDEwxK ZWZmcmV5IFdhbmcxITAfBgkqhkiG9w0BCQEWEmp3YW5nQHBhbGFudGlyLmNvbTCCASIwDQYJKoZI hvcNAQEBBQADggEPADCCAQoCggEBALiovezdrKoQI/GrBxajvik5PkYE1kM0UvkWRV1UKyyeiaxL GQ9ubaZ5jeUd8jyWqWYzTl+do5+CMqgMlOxTH895939jYxLKoXQCp/oByQqaX/Eet02psNLFyyH7 HNSJiOA7v5SQKrYFY7T2Fub37FSBXlK5hodAvKTf0TCYkHh1M6xZ7J9gwI3lfcEsENlnt54qW95m nw/16M3zkzZGI9jTgNBdqRnTekNFqr/1oi/HO7zqE9BUGj5oBZwbfMMb1eDHB9B8PLe/WZmJE95H AW5z36E4QQYhnmbOQHBbQZu5gMnEYEdQbkX9XvfKFsje9msYIbQuIU9rJEtnNdZTrjMCAwEAAaOC Af4wggH6MB8GA1UdIwQYMBaAFImCZ33EnSZwAEu0UEh83j2uBG59MB0GA1UdDgQWBBQ7oOnNttsD PfvuWdDwNh1K9dwc6TAOBgNVHQ8BAf8EBAMCBaAwDAYDVR0TAQH/BAIwADAdBgNVHSUEFjAUBggr BgEFBQcDBAYIKwYBBQUHAwIwRgYDVR0gBD8wPTA7BgwrBgEEAbIxAQIBAwUwKzApBggrBgEFBQcC ARYdaHR0cHM6Ly9zZWN1cmUuY29tb2RvLm5ldC9DUFMwgaUGA1UdHwSBnTCBmjBMoEqgSIZGaHR0 cDovL2NybC5jb21vZG9jYS5jb20vVVROLVVTRVJGaXJzdC1DbGllbnRBdXRoZW50aWNhdGlvbmFu ZEVtYWlsLmNybDBKoEigRoZEaHR0cDovL2NybC5jb21vZG8ubmV0L1VUTi1VU0VSRmlyc3QtQ2xp ZW50QXV0aGVudGljYXRpb25hbmRFbWFpbC5jcmwwbAYIKwYBBQUHAQEEYDBeMDYGCCsGAQUFBzAC hipodHRwOi8vY3J0LmNvbW9kb2NhLmNvbS9VVE5BQUFDbGllbnRDQS5jcnQwJAYIKwYBBQUHMAGG GGh0dHA6Ly9vY3NwLmNvbW9kb2NhLmNvbTAdBgNVHREEFjAUgRJqd2FuZ0BwYWxhbnRpci5jb20w DQYJKoZIhvcNAQEFBQADggEBAG3u36M7zrIP/s8uR69W2Y/kiOOXM3h16NbQN68gXCkGfCEhz9Tz rKK21r1ZkWSQtH4ppezbpVvmo/uCWyKorzQvc8phXNPtD5qMO0sSqlsbZgso377x24OQP6wVCybL fjH+lFV9ofnFxMBgbDsjJxZnPJd6p1iPT+W8+89RkVje/PZtv7HjDkbwMxZckDlXF6S+xV9v956G eN1sZ5g9K25VMbmd8U9QNgqNmQhNCztrcU6rOBt7vvY1fMOSua5g0s/l2ORsvzrcd7+pDOLjgqRO KS9nHYT9zdW/7ZGQCdDk2R8vXfe39pEC72PJwZ1hXyJjmInMaAqCDLO6HzFrfyQxggRoMIIEZAIB ATCBxDCBrjELMAkGA1UEBhMCVVMxCzAJBgNVBAgTAlVUMRcwFQYDVQQHEw5TYWx0IExha2UgQ2l0 eTEeMBwGA1UEChMVVGhlIFVTRVJUUlVTVCBOZXR3b3JrMSEwHwYDVQQLExhodHRwOi8vd3d3LnVz ZXJ0cnVzdC5jb20xNjA0BgNVBAMTLVVUTi1VU0VSRmlyc3QtQ2xpZW50IEF1dGhlbnRpY2F0aW9u IGFuZCBFbWFpbAIRAPam6MORK2txy64lXxarT88wCQYFKw4DAhoFAKCCAngwGAYJKoZIhvcNAQkD MQsGCSqGSIb3DQEHATAcBgkqhkiG9w0BCQUxDxcNMTEwNjIzMjE1NTE3WjAjBgkqhkiG9w0BCQQx FgQUYe1MClcjDbZvp8s06KOCeRqM39AwZwYJKoZIhvcNAQkPMVowWDAKBggqhkiG9w0DBzAOBggq hkiG9w0DAgICAIAwDQYIKoZIhvcNAwICAUAwBwYFKw4DAgcwDQYIKoZIhvcNAwICASgwBwYFKw4D AhowCgYIKoZIhvcNAgUwgdUGCSsGAQQBgjcQBDGBxzCBxDCBrjELMAkGA1UEBhMCVVMxCzAJBgNV BAgTAlVUMRcwFQYDVQQHEw5TYWx0IExha2UgQ2l0eTEeMBwGA1UEChMVVGhlIFVTRVJUUlVTVCBO ZXR3b3JrMSEwHwYDVQQLExhodHRwOi8vd3d3LnVzZXJ0cnVzdC5jb20xNjA0BgNVBAMTLVVUTi1V U0VSRmlyc3QtQ2xpZW50IEF1dGhlbnRpY2F0aW9uIGFuZCBFbWFpbAIRAPam6MORK2txy64lXxar T88wgdcGCyqGSIb3DQEJEAILMYHHoIHEMIGuMQswCQYDVQQGEwJVUzELMAkGA1UECBMCVVQxFzAV BgNVBAcTDlNhbHQgTGFrZSBDaXR5MR4wHAYDVQQKExVUaGUgVVNFUlRSVVNUIE5ldHdvcmsxITAf BgNVBAsTGGh0dHA6Ly93d3cudXNlcnRydXN0LmNvbTE2MDQGA1UEAxMtVVROLVVTRVJGaXJzdC1D bGllbnQgQXV0aGVudGljYXRpb24gYW5kIEVtYWlsAhEA9qbow5Era3HLriVfFqtPzzANBgkqhkiG 9w0BAQEFAASCAQCkNjhZGL1iJhX7u4/7xf4yQPBAFTMXPgHT8abOj8TxsY/Wb27Je1x8SXhnld5N Ifk0aXT6EGvFm/FJnk9VQmin6DBN+waKBo7gTgRSwH91VvmcSYJM6pIRDxLTuAY93IqJonfddj+O YwxIUaF7xfqMrbXOeMbWGjyYUipNFVJiIuwBmrBvTTaw5I8bffA8BGviwIk+tmfe8weEBo9m6SlF YBsoIZRYBX6EpTGJ/GRjWv3i632VLl6/gI/ZEoxbI78d8g+kZNKvIS7QmBu2fA3ws9wst+bdwaYS zd8xNeyu+gyJlRBPPz93nDCq2EKG2rqGLUpp4odGXION/iVFgjQEAAAAAAAA ------=_NextPart_000_0097_01CC31B5.909B9A20--