From user-return-25117-apmail-cassandra-user-archive=cassandra.apache.org@cassandra.apache.org Tue Mar 27 17:10:37 2012 Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id DC1689BE6 for ; Tue, 27 Mar 2012 17:10:37 +0000 (UTC) Received: (qmail 20809 invoked by uid 500); 27 Mar 2012 17:10:35 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 20778 invoked by uid 500); 27 Mar 2012 17:10:35 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 20768 invoked by uid 99); 27 Mar 2012 17:10:35 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Mar 2012 17:10:35 +0000 X-ASF-Spam-Status: No, hits=-0.0 required=5.0 tests=SPF_HELO_PASS,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of andre.cruz@co.sapo.pt designates 212.55.140.20 as permitted sender) Received: from [212.55.140.20] (HELO sl.pt) (212.55.140.20) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Mar 2012 17:10:26 +0000 Received: (qmail 6960 invoked from network); 27 Mar 2012 17:10:05 -0000 X-AntiVirus: PTMail-AV 0.3-0.95.1 X-Scan-Status: AV clean (0.02252 seconds); Received: from unknown (HELO [10.134.132.86]) (andre.cruz@co.sapo.pt@[10.134.132.86]) (envelope-sender ) by mail-sl-pt02 (qmail-ldap-1.03) with AES128-SHA encrypted SMTP for ; 27 Mar 2012 17:10:05 -0000 From: =?iso-8859-1?Q?Andr=E9_Cruz?= Content-Type: multipart/signed; boundary="Apple-Mail=_BA5BCB88-DD3D-41FC-8800-20CB9B67925D"; protocol="application/pkcs7-signature"; micalg=sha1 Subject: Advice on architecture Date: Tue, 27 Mar 2012 18:10:04 +0100 Message-Id: <006D0692-9F40-4BEB-8FEC-DE3D441E09D3@co.sapo.pt> To: user@cassandra.apache.org Mime-Version: 1.0 (Apple Message framework v1257) X-Mailer: Apple Mail (2.1257) --Apple-Mail=_BA5BCB88-DD3D-41FC-8800-20CB9B67925D Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=iso-8859-1 Hello. I'm developing a system that will require me to store large (<=3D4MB) = columns in Cassandra. Right now I'm storing 1 column per row, in a = single CF. The machines I have at my disposal are 32GB RAM machines with = 10 SATA drives each. I would prefer to have a larger number of smaller = nodes, but this is what I have to work with. Some issues that I have = are: RAID0 Vs separate data dirs, and SizeTiered compaction Vs Leveled = compaction. I will have approximately 2 times more writes than reads. RAID0 would help me use more efficiently the total disk space available = at each node, but tests have shown that under write load it behaves much = worse than using separate data dirs, one per disk. I used a 3-node = cluster, and the node with RAID0 kept getting behind the other two nodes = which had separate data dirs. The problem with separate data dirs is = that it seems to be difficult for Cassandra to use the space efficiently = due to the compactions. I first tried the new Leveled compactions = scheme, which seemed promising since it would create "small" files that = could be scattered by the data dirs, but the IO necessary for this = compaction scheme is enormous under write load. It was constantly = working and it affected the write throughput because it slowed the = flushing of memtables. I then tried tiered compaction and it performed = better, but as it tends to create large SSTables they cannot be split = across the multiple data dirs. What I'm thinking of doing now is using multiple data dirs, with tiered = compaction, and dividing the input data in several (64) different CFs. = This way smaller SSTables will be created and these can be split across = the multiple data dirs. This will allow me to better use the available = capacity and I will not need as much free space for compactions than I = would if the SSTables were larger. Am I missing something here? Is this the best way to deal with this = (abnormal) use case? Thanks and best regards, Andr=E9 Cruz= --Apple-Mail=_BA5BCB88-DD3D-41FC-8800-20CB9B67925D Content-Disposition: attachment; filename=smime.p7s Content-Type: application/pkcs7-signature; name=smime.p7s Content-Transfer-Encoding: base64 MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIPLjCCBN0w ggPFoAMCAQICEHGS++YZX6xNEoV0cTSiGKcwDQYJKoZIhvcNAQEFBQAwezELMAkGA1UEBhMCR0Ix GzAZBgNVBAgMEkdyZWF0ZXIgTWFuY2hlc3RlcjEQMA4GA1UEBwwHU2FsZm9yZDEaMBgGA1UECgwR Q29tb2RvIENBIExpbWl0ZWQxITAfBgNVBAMMGEFBQSBDZXJ0aWZpY2F0ZSBTZXJ2aWNlczAeFw0w NDAxMDEwMDAwMDBaFw0yODEyMzEyMzU5NTlaMIGuMQswCQYDVQQGEwJVUzELMAkGA1UECBMCVVQx FzAVBgNVBAcTDlNhbHQgTGFrZSBDaXR5MR4wHAYDVQQKExVUaGUgVVNFUlRSVVNUIE5ldHdvcmsx ITAfBgNVBAsTGGh0dHA6Ly93d3cudXNlcnRydXN0LmNvbTE2MDQGA1UEAxMtVVROLVVTRVJGaXJz dC1DbGllbnQgQXV0aGVudGljYXRpb24gYW5kIEVtYWlsMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8A MIIBCgKCAQEAsjmFpPJ9q0E7YkY3rs3BYHW8OWX5ShpHornMSMxqmNVNNRm5pELlzkniii8efNIx B8dOtINknS4p1aJkxIW9hVE1eaROaJB7HHqkkqgX8pgV8pPMyaQylbsMTzC9mKALi+VuG6JG+ni8 om+rWV6lL8/K2m2qL+usobNqqrcuZzWLeeEeaYji5kbNoKXqvgvOdjp6Dpvq/NonWz1zHyLmSGHG TPNpsaguG7bUMSAsvIKKjqQOpdeJQ/wWWq8dcdcRWdq6hw2v+vPhwvCkxWeM1tZUOt4KpLoDd7Nl yP0e03RiqhjKaJMeoYV+9Udly/hNVyh00jT/MLbu9mIwFIws6wIDAQABo4IBJzCCASMwHwYDVR0j BBgwFoAUoBEKIz6W8Qfs4q8p74Klf9AwpLQwHQYDVR0OBBYEFImCZ33EnSZwAEu0UEh83j2uBG59 MA4GA1UdDwEB/wQEAwIBBjAPBgNVHRMBAf8EBTADAQH/MB0GA1UdJQQWMBQGCCsGAQUFBwMCBggr BgEFBQcDBDARBgNVHSAECjAIMAYGBFUdIAAwewYDVR0fBHQwcjA4oDagNIYyaHR0cDovL2NybC5j b21vZG9jYS5jb20vQUFBQ2VydGlmaWNhdGVTZXJ2aWNlcy5jcmwwNqA0oDKGMGh0dHA6Ly9jcmwu Y29tb2RvLm5ldC9BQUFDZXJ0aWZpY2F0ZVNlcnZpY2VzLmNybDARBglghkgBhvhCAQEEBAMCAQYw DQYJKoZIhvcNAQEFBQADggEBAJ2Vyzy4fqUJxB6/C8LHdo45PJTGEKpPDMngq4RdiVTgZTvzbRx8 NywlVF+WIfw3hJGdFdwUT4HPVB1rbEVgxy35l1FM+WbKPKCCjKbI8OLp1Er57D9Wyd12jMOCAU9s APMeGmF0BEcDqcZAV5G8ZSLFJ2dPV9tkWtmNH7qGL/QGrpxp7en0zykX2OBKnxogL5dMUbtGB8SK N04g4wkxaMeexIud6H4RvDJoEJYRmETYKlFgTYjrdDrfQwYyyDlWjDoRUtNBpEMD9O3vMyfbOeAU TibJ2PU54om4k123KSZB6rObroP8d3XK6Mq1/uJlSmM+RMTQw16Hc6mYHK9/FX8wggUaMIIEAqAD AgECAhBtGeqnGU9qMyLmIjJ6qnHeMA0GCSqGSIb3DQEBBQUAMIGuMQswCQYDVQQGEwJVUzELMAkG A1UECBMCVVQxFzAVBgNVBAcTDlNhbHQgTGFrZSBDaXR5MR4wHAYDVQQKExVUaGUgVVNFUlRSVVNU IE5ldHdvcmsxITAfBgNVBAsTGGh0dHA6Ly93d3cudXNlcnRydXN0LmNvbTE2MDQGA1UEAxMtVVRO LVVTRVJGaXJzdC1DbGllbnQgQXV0aGVudGljYXRpb24gYW5kIEVtYWlsMB4XDTExMDQyODAwMDAw MFoXDTIwMDUzMDEwNDgzOFowgZMxCzAJBgNVBAYTAkdCMRswGQYDVQQIExJHcmVhdGVyIE1hbmNo ZXN0ZXIxEDAOBgNVBAcTB1NhbGZvcmQxGjAYBgNVBAoTEUNPTU9ETyBDQSBMaW1pdGVkMTkwNwYD VQQDEzBDT01PRE8gQ2xpZW50IEF1dGhlbnRpY2F0aW9uIGFuZCBTZWN1cmUgRW1haWwgQ0EwggEi MA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCShIRbS1eY1F4vi6ThQMijU1hfZmXxMk73nzJ9 VdB4TFW3QpTg+SdxB8XGaaS5MsTxQBqQzCdWYn8XtXFpruUgG+TLY15gyqJB9mrho/+43x9IbWVD jCouK2M4d9+xF6zC2oIC1tQyatRnbyATj1w1+uVUgK/YcQodNwoCUFNslR2pEBS0mZVZEjH/CaLS TNxS297iQAFbSGjdxUq04O0kHzqvcV8H46y/FDuwJXFoPfQP1hdYRhWBPGiLi4MPbXohV+Y0sNsy fuNK4aVScmQmkU6lkg//4LFg/RpvaFGZY40ai6XMQpubfSJj06mg/M6ekN9EGfRcWzW6FvOnm//B AgMBAAGjggFLMIIBRzAfBgNVHSMEGDAWgBSJgmd9xJ0mcABLtFBIfN49rgRufTAdBgNVHQ4EFgQU ehNOAHRbxnhjZCfBL+KgW7x5xXswDgYDVR0PAQH/BAQDAgEGMBIGA1UdEwEB/wQIMAYBAf8CAQAw EQYDVR0gBAowCDAGBgRVHSAAMFgGA1UdHwRRME8wTaBLoEmGR2h0dHA6Ly9jcmwudXNlcnRydXN0 LmNvbS9VVE4tVVNFUkZpcnN0LUNsaWVudEF1dGhlbnRpY2F0aW9uYW5kRW1haWwuY3JsMHQGCCsG AQUFBwEBBGgwZjA9BggrBgEFBQcwAoYxaHR0cDovL2NydC51c2VydHJ1c3QuY29tL1VUTkFkZFRy dXN0Q2xpZW50X0NBLmNydDAlBggrBgEFBQcwAYYZaHR0cDovL29jc3AudXNlcnRydXN0LmNvbTAN BgkqhkiG9w0BAQUFAAOCAQEAhda+eFdVbTN/RFL+QtUGqAEDgIr7DbL9Sr/2r0FJ9RtaxdKtG3Nu PukmfOZMmMEwKN/L+0I8oSU+CnXW0D05hmbRoZu1TZtvryhsHa/l6nRaqNqxwPF1ei+eupN5yv7i kR5WdLL4jdPgQ3Ib7Y/9YDkgR/uLrzplSDyYPaUlv73vYOBJ5RbI6z9Dg/Dg7g3B080zX5vQvWBq szv++tTJOjwf7Zv/m0kzvkIpOYPuM2kugp1FTahp2oAbHj3SGl18R5mlmwhtEpmG1l1XBxunML5L SUS4kH7K0Xk467Qz+qA6XSZYnmFVGLQh1ZnV4ENAQjC+6qXnlNKw/vN1+X9u5zCCBSswggQToAMC AQICEHwRFDWcApBpCERr1nJep4gwDQYJKoZIhvcNAQEFBQAwgZMxCzAJBgNVBAYTAkdCMRswGQYD VQQIExJHcmVhdGVyIE1hbmNoZXN0ZXIxEDAOBgNVBAcTB1NhbGZvcmQxGjAYBgNVBAoTEUNPTU9E TyBDQSBMaW1pdGVkMTkwNwYDVQQDEzBDT01PRE8gQ2xpZW50IEF1dGhlbnRpY2F0aW9uIGFuZCBT ZWN1cmUgRW1haWwgQ0EwHhcNMTEwODI5MDAwMDAwWhcNMTIwODI4MjM1OTU5WjAmMSQwIgYJKoZI hvcNAQkBFhVhbmRyZS5jcnV6QGNvLnNhcG8ucHQwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEK AoIBAQDd+5q7j0XhK1B3mziTDsxMcWIBYPjCzqxJvRmUI3NQyrl2gbdTC2FzNsiCLiXmAmF7nPVw f0KGte8YrQqJcv3RGm+OxEjjOemZ3jH+pUWoq1NZzkMz8rkDU6hIlxsQ1xNsniVYU4nftk8ihSwX 780aBSDYB9An2DxnoWDw0JsEalWM3w7VuRLZZpWJSi2SWnBNvqFv95GY2/iMLISH+19GeD2vfmP6 hop1qnbEUILB2CC2HwBxWpNp8IivypbPhGXxSqDpPn00O0+XjGXzaFXTKguLH7hxaSg2E11vuZzM HzEnePHQT9+G2QFtXVuVEW9NtmolA0pyLbCfSawAik3ZAgMBAAGjggHlMIIB4TAfBgNVHSMEGDAW gBR6E04AdFvGeGNkJ8Ev4qBbvHnFezAdBgNVHQ4EFgQUEPYfzWmFvUHLNSqc2MZWdzvMmjYwDgYD VR0PAQH/BAQDAgWgMAwGA1UdEwEB/wQCMAAwIAYDVR0lBBkwFwYIKwYBBQUHAwQGCysGAQQBsjEB AwUCMBEGCWCGSAGG+EIBAQQEAwIFIDBGBgNVHSAEPzA9MDsGDCsGAQQBsjEBAgEBATArMCkGCCsG AQUFBwIBFh1odHRwczovL3NlY3VyZS5jb21vZG8ubmV0L0NQUzBXBgNVHR8EUDBOMEygSqBIhkZo dHRwOi8vY3JsLmNvbW9kb2NhLmNvbS9DT01PRE9DbGllbnRBdXRoZW50aWNhdGlvbmFuZFNlY3Vy ZUVtYWlsQ0EuY3JsMIGIBggrBgEFBQcBAQR8MHowUgYIKwYBBQUHMAKGRmh0dHA6Ly9jcnQuY29t b2RvY2EuY29tL0NPTU9ET0NsaWVudEF1dGhlbnRpY2F0aW9uYW5kU2VjdXJlRW1haWxDQS5jcnQw JAYIKwYBBQUHMAGGGGh0dHA6Ly9vY3NwLmNvbW9kb2NhLmNvbTAgBgNVHREEGTAXgRVhbmRyZS5j cnV6QGNvLnNhcG8ucHQwDQYJKoZIhvcNAQEFBQADggEBADPV/9DWnRrBJeZZUk1d5H1z8ey/YmJ2 FfbzrwHLO4IizENmQFfp12Igcy9hvg7FWt6jqUrbS5oTK0zEZqONQRupbaaBLIYdK/3tD3n3FgOt daBRzgOkVoi/w266zZ97JVml6uqZ9WUl11gRo67NuBMEgMbhqEHF9G4xL0G3wbuM0RqGI7vkjYiY HLPfiIxnIdHyJRJqsWLOvQXhUmB+jx94CF17mmqf61qPGuLpilyNUIM86VfbETrt4Icr2qyEUimc vgpmw5aXnCSyqATMOtuis2ET8tl2/reJBMeKaTaor4bW/mLIH6aJctv/VWFUII1OapDL3F9uhzwF c6vhY2YxggOrMIIDpwIBATCBqDCBkzELMAkGA1UEBhMCR0IxGzAZBgNVBAgTEkdyZWF0ZXIgTWFu Y2hlc3RlcjEQMA4GA1UEBxMHU2FsZm9yZDEaMBgGA1UEChMRQ09NT0RPIENBIExpbWl0ZWQxOTA3 BgNVBAMTMENPTU9ETyBDbGllbnQgQXV0aGVudGljYXRpb24gYW5kIFNlY3VyZSBFbWFpbCBDQQIQ fBEUNZwCkGkIRGvWcl6niDAJBgUrDgMCGgUAoIIB1zAYBgkqhkiG9w0BCQMxCwYJKoZIhvcNAQcB MBwGCSqGSIb3DQEJBTEPFw0xMjAzMjcxNzEwMDVaMCMGCSqGSIb3DQEJBDEWBBSrUyMfpLH+MZtK mBO9ADfXiPiEKjCBuQYJKwYBBAGCNxAEMYGrMIGoMIGTMQswCQYDVQQGEwJHQjEbMBkGA1UECBMS R3JlYXRlciBNYW5jaGVzdGVyMRAwDgYDVQQHEwdTYWxmb3JkMRowGAYDVQQKExFDT01PRE8gQ0Eg TGltaXRlZDE5MDcGA1UEAxMwQ09NT0RPIENsaWVudCBBdXRoZW50aWNhdGlvbiBhbmQgU2VjdXJl IEVtYWlsIENBAhB8ERQ1nAKQaQhEa9ZyXqeIMIG7BgsqhkiG9w0BCRACCzGBq6CBqDCBkzELMAkG A1UEBhMCR0IxGzAZBgNVBAgTEkdyZWF0ZXIgTWFuY2hlc3RlcjEQMA4GA1UEBxMHU2FsZm9yZDEa MBgGA1UEChMRQ09NT0RPIENBIExpbWl0ZWQxOTA3BgNVBAMTMENPTU9ETyBDbGllbnQgQXV0aGVu dGljYXRpb24gYW5kIFNlY3VyZSBFbWFpbCBDQQIQfBEUNZwCkGkIRGvWcl6niDANBgkqhkiG9w0B AQEFAASCAQAsW70Jli2ZhF/EAqbDaw4h4buQjPezsmAehAAljMevHzmX8YaySwKyliWbOys5vSCW 4h2Fuo0+v2xvNGGFrfWkqpQTE9lhEKy4iiJ6kyfyNEneQ0aq5d85OSh5rUwhc423bY1wOMgRQJXj NAQ8J8sa11GYT5Z+mslYuZHXrpBAbHr0Wt94yn8ERZwJyDwbd2dAMJxqnytXe6O/Acr48ljm+un7 FdTnL0HiixGJeLBBq4oe/hVg6mTukBLL27vvmqP/X9/WiRonI/HeLmsQl/iNUUBlNlnVGHeWEtK+ RUA4g/g45kd/X6kPNgDwEvOkp2h+6XxzFeHgN31Juc1iL4aVAAAAAAAA --Apple-Mail=_BA5BCB88-DD3D-41FC-8800-20CB9B67925D--