Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id EEE7C81B4 for ; Tue, 30 Aug 2011 22:57:45 +0000 (UTC) Received: (qmail 34059 invoked by uid 500); 30 Aug 2011 22:57:43 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 33988 invoked by uid 500); 30 Aug 2011 22:57:43 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 33978 invoked by uid 99); 30 Aug 2011 22:57:42 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 30 Aug 2011 22:57:42 +0000 X-ASF-Spam-Status: No, hits=-0.0 required=5.0 tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (nike.apache.org: local policy) Received: from [209.85.210.48] (HELO mail-pz0-f48.google.com) (209.85.210.48) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 30 Aug 2011 22:57:33 +0000 Received: by pzk34 with SMTP id 34so224577pzk.21 for ; Tue, 30 Aug 2011 15:57:11 -0700 (PDT) Received: by 10.142.185.10 with SMTP id i10mr1314193wff.22.1314745031545; Tue, 30 Aug 2011 15:57:11 -0700 (PDT) Received: from happy.lab (173-160-196-241-Washington.hfc.comcastbusiness.net. [173.160.196.241]) by mx.google.com with ESMTPS id b7sm26211884pbm.0.2011.08.30.15.57.10 (version=TLSv1/SSLv3 cipher=OTHER); Tue, 30 Aug 2011 15:57:10 -0700 (PDT) From: Derek Andree Mime-Version: 1.0 (Apple Message framework v1244.3) Content-Type: multipart/signed; boundary="Apple-Mail=_EEE30627-A4F9-400A-89C6-8B3C895B9BFE"; protocol="application/pkcs7-signature"; micalg=sha1 Subject: Re: Disk usage for CommitLog Date: Tue, 30 Aug 2011 15:57:09 -0700 In-Reply-To: <4e5d052a.42bbec0a.759d.0034@mx.google.com> To: user@cassandra.apache.org References: <4e5d052a.42bbec0a.759d.0034@mx.google.com> Message-Id: <83073BA6-32D5-479D-990D-4E4A0D7B481D@lacunasystems.com> X-Mailer: Apple Mail (2.1244.3) X-Virus-Checked: Checked by ClamAV on apache.org --Apple-Mail=_EEE30627-A4F9-400A-89C6-8B3C895B9BFE Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii > > 86GB in commitlog and 42GB in data > =20 > Whoa, that seems really wrong, particularly given your data spans 13 = months. Have you changed any of the default cassandra.yaml setting? What = is the maximum memtable_flush_after across all your CFs? Any = warnings/errors in the Cassandra log? > =20 It seems wrong to me too. It got so bad that /var/lib/cassandra looked = like this: $ du -hs ./* 122G ./commitlog 55G ./data 17M ./saved_caches I restarted cassandra, and it took a while to chew through all the = commitlog files, then disk utilization was like so: du -hs ./* 1.1M ./commitlog 56G ./data 17M ./saved_caches This isn't with 13 months of data, only with a couple months of data. Upon going through the cassandra logs, I saw a ton of "too many open = files" warnings: WARN [Thread-4] 2011-08-30 12:07:27,601 CustomTThreadPoolServer.java = (line 112) Transport error occurred during acceptance of message. org.apache.thrift.transport.TTransportException: = java.net.SocketException: Too many open files at = org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:11= 8) at = org.apache.cassandra.thrift.TCustomServerSocket.acceptImpl(TCustomServerSo= cket.java:68) at = org.apache.cassandra.thrift.TCustomServerSocket.acceptImpl(TCustomServerSo= cket.java:39) at = org.apache.thrift.transport.TServerTransport.accept(TServerTransport.java:= 31) at = org.apache.cassandra.thrift.CustomTThreadPoolServer.serve(CustomTThreadPoo= lServer.java:102) at = org.apache.cassandra.thrift.CassandraDaemon$ThriftServer.run(CassandraDaem= on.java:198) Caused by: java.net.SocketException: Too many open files at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408) at java.net.ServerSocket.implAccept(ServerSocket.java:462) at java.net.ServerSocket.accept(ServerSocket.java:430) at = org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:11= 3) I guess I should set the number of allowed files to some big number with = ulimit. Anyone have a suggestion for how big? I was thinking ulimit -n = 10000, but first I'm going to try to reproduce the "too many files open" = condition and then have a look at lsof to see just how many files are = really open. On a side note, why does cassandra seem to log to /var/log/cassandra.log = no matter what's in log4j.properties? I ended up having to link that to = /dev/null to keep from filling up my root partition with cassandra logs = that I already have elsewhere on another filesystem. -Derek --Apple-Mail=_EEE30627-A4F9-400A-89C6-8B3C895B9BFE Content-Disposition: attachment; filename=smime.p7s Content-Type: application/pkcs7-signature; name=smime.p7s Content-Transfer-Encoding: base64 MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIKZTCCBN0w ggPFoAMCAQICEHGS++YZX6xNEoV0cTSiGKcwDQYJKoZIhvcNAQEFBQAwezELMAkGA1UEBhMCR0Ix GzAZBgNVBAgMEkdyZWF0ZXIgTWFuY2hlc3RlcjEQMA4GA1UEBwwHU2FsZm9yZDEaMBgGA1UECgwR Q29tb2RvIENBIExpbWl0ZWQxITAfBgNVBAMMGEFBQSBDZXJ0aWZpY2F0ZSBTZXJ2aWNlczAeFw0w NDAxMDEwMDAwMDBaFw0yODEyMzEyMzU5NTlaMIGuMQswCQYDVQQGEwJVUzELMAkGA1UECBMCVVQx FzAVBgNVBAcTDlNhbHQgTGFrZSBDaXR5MR4wHAYDVQQKExVUaGUgVVNFUlRSVVNUIE5ldHdvcmsx ITAfBgNVBAsTGGh0dHA6Ly93d3cudXNlcnRydXN0LmNvbTE2MDQGA1UEAxMtVVROLVVTRVJGaXJz dC1DbGllbnQgQXV0aGVudGljYXRpb24gYW5kIEVtYWlsMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8A MIIBCgKCAQEAsjmFpPJ9q0E7YkY3rs3BYHW8OWX5ShpHornMSMxqmNVNNRm5pELlzkniii8efNIx B8dOtINknS4p1aJkxIW9hVE1eaROaJB7HHqkkqgX8pgV8pPMyaQylbsMTzC9mKALi+VuG6JG+ni8 om+rWV6lL8/K2m2qL+usobNqqrcuZzWLeeEeaYji5kbNoKXqvgvOdjp6Dpvq/NonWz1zHyLmSGHG TPNpsaguG7bUMSAsvIKKjqQOpdeJQ/wWWq8dcdcRWdq6hw2v+vPhwvCkxWeM1tZUOt4KpLoDd7Nl yP0e03RiqhjKaJMeoYV+9Udly/hNVyh00jT/MLbu9mIwFIws6wIDAQABo4IBJzCCASMwHwYDVR0j BBgwFoAUoBEKIz6W8Qfs4q8p74Klf9AwpLQwHQYDVR0OBBYEFImCZ33EnSZwAEu0UEh83j2uBG59 MA4GA1UdDwEB/wQEAwIBBjAPBgNVHRMBAf8EBTADAQH/MB0GA1UdJQQWMBQGCCsGAQUFBwMCBggr BgEFBQcDBDARBgNVHSAECjAIMAYGBFUdIAAwewYDVR0fBHQwcjA4oDagNIYyaHR0cDovL2NybC5j b21vZG9jYS5jb20vQUFBQ2VydGlmaWNhdGVTZXJ2aWNlcy5jcmwwNqA0oDKGMGh0dHA6Ly9jcmwu Y29tb2RvLm5ldC9BQUFDZXJ0aWZpY2F0ZVNlcnZpY2VzLmNybDARBglghkgBhvhCAQEEBAMCAQYw DQYJKoZIhvcNAQEFBQADggEBAJ2Vyzy4fqUJxB6/C8LHdo45PJTGEKpPDMngq4RdiVTgZTvzbRx8 NywlVF+WIfw3hJGdFdwUT4HPVB1rbEVgxy35l1FM+WbKPKCCjKbI8OLp1Er57D9Wyd12jMOCAU9s APMeGmF0BEcDqcZAV5G8ZSLFJ2dPV9tkWtmNH7qGL/QGrpxp7en0zykX2OBKnxogL5dMUbtGB8SK N04g4wkxaMeexIud6H4RvDJoEJYRmETYKlFgTYjrdDrfQwYyyDlWjDoRUtNBpEMD9O3vMyfbOeAU TibJ2PU54om4k123KSZB6rObroP8d3XK6Mq1/uJlSmM+RMTQw16Hc6mYHK9/FX8wggWAMIIEaKAD AgECAhAHed03fGlPt2nV4Ztl4nCuMA0GCSqGSIb3DQEBBQUAMIGuMQswCQYDVQQGEwJVUzELMAkG A1UECBMCVVQxFzAVBgNVBAcTDlNhbHQgTGFrZSBDaXR5MR4wHAYDVQQKExVUaGUgVVNFUlRSVVNU IE5ldHdvcmsxITAfBgNVBAsTGGh0dHA6Ly93d3cudXNlcnRydXN0LmNvbTE2MDQGA1UEAxMtVVRO LVVTRVJGaXJzdC1DbGllbnQgQXV0aGVudGljYXRpb24gYW5kIEVtYWlsMB4XDTEwMTIyOTAwMDAw MFoXDTExMTIyOTIzNTk1OVowKjEoMCYGCSqGSIb3DQEJARYZZGFuZHJlZUBsYWN1bmFzeXN0ZW1z LmNvbTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAN8AUGZU5j3OjRr/ZHftcdrSdFHC W9Ds21da8VWWO0+C52uCWvXvM7bI2y7L4lYbuHY+qs+zG70dRk/PuXYPaxrJOPGtA9qcQ/cUULBw Ilmvv1lqSGvHXhjg025blS+u7sQlkycY5+g4cuqGoez4XCmEnwdGpzCVmgNkYzPZOFdS2rajVEwg qYyK9Eo9lWHX2a/HXMzHQmCyDqjJSGQUUiHk55Jwg+pWtykhhnQQM9BQrgvdQOJCL2dyezTJJlNf hY54lCfKD4hwd3fOVZLDUj3Fd1t00q1xg0AF+GDYLkWRpviLWuHZCWKL99R1IshQjKG6jU4q7iFr ecmrVw5o56MCAwEAAaOCAhswggIXMB8GA1UdIwQYMBaAFImCZ33EnSZwAEu0UEh83j2uBG59MB0G A1UdDgQWBBTfpxv0dUFtmuR/ZQJz2jQ2qt3aTDAOBgNVHQ8BAf8EBAMCBaAwDAYDVR0TAQH/BAIw ADAgBgNVHSUEGTAXBggrBgEFBQcDBAYLKwYBBAGyMQEDBQIwEQYJYIZIAYb4QgEBBAQDAgUgMEYG A1UdIAQ/MD0wOwYMKwYBBAGyMQECAQEBMCswKQYIKwYBBQUHAgEWHWh0dHBzOi8vc2VjdXJlLmNv bW9kby5uZXQvQ1BTMIGlBgNVHR8EgZ0wgZowTKBKoEiGRmh0dHA6Ly9jcmwuY29tb2RvY2EuY29t L1VUTi1VU0VSRmlyc3QtQ2xpZW50QXV0aGVudGljYXRpb25hbmRFbWFpbC5jcmwwSqBIoEaGRGh0 dHA6Ly9jcmwuY29tb2RvLm5ldC9VVE4tVVNFUkZpcnN0LUNsaWVudEF1dGhlbnRpY2F0aW9uYW5k RW1haWwuY3JsMGwGCCsGAQUFBwEBBGAwXjA2BggrBgEFBQcwAoYqaHR0cDovL2NydC5jb21vZG9j YS5jb20vVVROQUFBQ2xpZW50Q0EuY3J0MCQGCCsGAQUFBzABhhhodHRwOi8vb2NzcC5jb21vZG9j YS5jb20wJAYDVR0RBB0wG4EZZGFuZHJlZUBsYWN1bmFzeXN0ZW1zLmNvbTANBgkqhkiG9w0BAQUF AAOCAQEAR1HPJz0+rvkf6efQ/La72DkmYCeH/fNUaPMkLesu4ilWWHDT3ROY7HO7JtxPf5r6PNAv KihHmYdfVxpaJ/J47NTXiEmuMTuXOnhx0tbd5kqi3gDEJcOcr4FNA/AFhz0Uxcd8t4FXLXuSGsrt DmTFmswbDYJa1INeTwqoMmLZYsWj85qOOxz4GyIMQCuNFPNwTrC5TLOLw9CarLbw/3Sm6hIqYiaq 4e44jb7C4fBXhVAG0olSBI4embs2gW9I2JCuhEKgtFlbdlV8bn9IKPNCuxlm/zfWj1M5bVCW5pjJ 2Bid+CVQfiEFNqUJidt+HolYpWdEVoxl/z/zcu0BsAPp3DGCA/wwggP4AgEBMIHDMIGuMQswCQYD VQQGEwJVUzELMAkGA1UECBMCVVQxFzAVBgNVBAcTDlNhbHQgTGFrZSBDaXR5MR4wHAYDVQQKExVU aGUgVVNFUlRSVVNUIE5ldHdvcmsxITAfBgNVBAsTGGh0dHA6Ly93d3cudXNlcnRydXN0LmNvbTE2 MDQGA1UEAxMtVVROLVVTRVJGaXJzdC1DbGllbnQgQXV0aGVudGljYXRpb24gYW5kIEVtYWlsAhAH ed03fGlPt2nV4Ztl4nCuMAkGBSsOAwIaBQCgggINMBgGCSqGSIb3DQEJAzELBgkqhkiG9w0BBwEw HAYJKoZIhvcNAQkFMQ8XDTExMDgzMDIyNTcwOVowIwYJKoZIhvcNAQkEMRYEFM0NSMaFNJYxwq/r k2HsYsDkg3MNMIHUBgkrBgEEAYI3EAQxgcYwgcMwga4xCzAJBgNVBAYTAlVTMQswCQYDVQQIEwJV VDEXMBUGA1UEBxMOU2FsdCBMYWtlIENpdHkxHjAcBgNVBAoTFVRoZSBVU0VSVFJVU1QgTmV0d29y azEhMB8GA1UECxMYaHR0cDovL3d3dy51c2VydHJ1c3QuY29tMTYwNAYDVQQDEy1VVE4tVVNFUkZp cnN0LUNsaWVudCBBdXRoZW50aWNhdGlvbiBhbmQgRW1haWwCEAd53Td8aU+3adXhm2XicK4wgdYG CyqGSIb3DQEJEAILMYHGoIHDMIGuMQswCQYDVQQGEwJVUzELMAkGA1UECBMCVVQxFzAVBgNVBAcT DlNhbHQgTGFrZSBDaXR5MR4wHAYDVQQKExVUaGUgVVNFUlRSVVNUIE5ldHdvcmsxITAfBgNVBAsT GGh0dHA6Ly93d3cudXNlcnRydXN0LmNvbTE2MDQGA1UEAxMtVVROLVVTRVJGaXJzdC1DbGllbnQg QXV0aGVudGljYXRpb24gYW5kIEVtYWlsAhAHed03fGlPt2nV4Ztl4nCuMA0GCSqGSIb3DQEBAQUA BIIBAKZhisL92pvLWw+gkntluq3UGSI/93AP1fzPaRJGYS94CwFGPPBsmtE54OpUXqMHkxbTHMWe edyKuyZHzC5TxQB06+iwFzPIckYtu8TGUK6EZegmiIA3VjMMyZI00g3mJ5WPzeBwJoPyjQMJHepg 6PLpnnqCMrff4uGl+dW2IS1zzDllzxNtZ3so1f4xlOjE1gPr0h8MLwcJPZyrJ5RSmEGm36vYQ+ot rsHylWcU7LMkIWM4FIak+0QXFtMuVb3Qmnfo3RKjmcLqAcwsiyu347bL1AKOe4y2KU3d4T25sbjH XZr5RWC2sb/JGiKoGgM5LbFQPQO0cQ5TtFAVdiikOTkAAAAAAAA= --Apple-Mail=_EEE30627-A4F9-400A-89C6-8B3C895B9BFE--