Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 0CB42200B13 for ; Wed, 1 Jun 2016 00:24:42 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 0B656160A46; Tue, 31 May 2016 22:24:42 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 2944C160A44 for ; Wed, 1 Jun 2016 00:24:41 +0200 (CEST) Received: (qmail 44856 invoked by uid 500); 31 May 2016 22:24:40 -0000 Mailing-List: contact user-help@accumulo.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@accumulo.apache.org Delivered-To: mailing list user@accumulo.apache.org Received: (qmail 44846 invoked by uid 99); 31 May 2016 22:24:40 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 May 2016 22:24:40 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id D5727C0AB5 for ; Tue, 31 May 2016 22:24:39 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 0.572 X-Spam-Level: X-Spam-Status: No, score=0.572 tagged_above=-999 required=6.31 tests=[HTML_MESSAGE=2, RP_MATCHES_RCVD=-1.426, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id 87uWXYvVHfWr for ; Tue, 31 May 2016 22:24:38 +0000 (UTC) Received: from mail.keywcorp.com (mail.keywcorp.com [50.224.70.23]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id D09915F252 for ; Tue, 31 May 2016 22:24:37 +0000 (UTC) From: Jayesh Patel To: "user@accumulo.apache.org" Subject: walog consumes all the disk space on power failure Thread-Topic: walog consumes all the disk space on power failure Thread-Index: AdG7hXnRf9xDSfa5Q+iX3FFNKNffcg== Date: Tue, 31 May 2016 22:18:32 +0000 Message-ID: <177B0C59EA9CBE41BCDDDE7465344525AB36E333@IHQ-ExchMS01.resources.keywcorp.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: yes X-MS-TNEF-Correlator: Content-Type: multipart/signed; protocol="application/x-pkcs7-signature"; micalg=SHA1; boundary="----=_NextPart_000_03DC_01D1BB69.ABABD3A0" MIME-Version: 1.0 archived-at: Tue, 31 May 2016 22:24:42 -0000 ------=_NextPart_000_03DC_01D1BB69.ABABD3A0 Content-Type: multipart/alternative; boundary="----=_NextPart_001_03DD_01D1BB69.ABABD3A0" ------=_NextPart_001_03DD_01D1BB69.ABABD3A0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit We have a 3 node Accumulo 1.7 cluster running as VMWare VMs with minute amount of data compared to Accumulo standards. We have run into a situation multiple times now where all the nodes have a power failure and when they are trying to recover from it simultaneously, walog grows exponentially and fills up all the available disk space. We have confirmed that the walog folder under /accumulo in hdfs is consuming 99% of the disk space. We have tried freeing enough space to be able to run Accumulo processes in the hopes of it burning through walog without success. Walog just grew to take up the freed space. Given that we need to better manage the power situation, we're trying to understand what could be causing this and if there's anything we can do to avoid this situation. We have some heartbeat data being written to a table at a very small constant rate which is not sufficient to cause a such large write-ahead log even if HDFS was pulled from under Accumulo's feet, so to speak during the power failure in case you're wondering. Thank you, Jayesh ------=_NextPart_001_03DD_01D1BB69.ABABD3A0 Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

We have a 3 node Accumulo 1.7 cluster running as = VMWare VMs with minute amount of data compared to Accumulo standards. =  

 

We have run into a situation multiple times now where = all the nodes have a power failure and when they are trying to recover = from it simultaneously, walog grows exponentially and fills up all the = available disk space.  We have confirmed that the walog folder = under /accumulo in hdfs is consuming 99% of the disk = space.

 

We have tried freeing enough space to be able to run = Accumulo processes in the hopes of it burning through walog without = success.  Walog just grew to take up the freed = space.

 

Given that we need to better manage the power = situation, we’re trying to understand what could be causing this = and if there’s anything we can do to avoid this situation.  =

We have some heartbeat data being = written to a table at a very small constant rate which is not sufficient = to cause a such large write-ahead log even if HDFS was pulled from under = Accumulo’s feet, so to speak during the power failure in case = you’re wondering.

 

Thank = you,

Jayesh

 

 

------=_NextPart_001_03DD_01D1BB69.ABABD3A0-- ------=_NextPart_000_03DC_01D1BB69.ABABD3A0 Content-Type: application/pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIMzzCCA3Uw ggJdoAMCAQICCwQAAAAAARVLWsOUMA0GCSqGSIb3DQEBBQUAMFcxCzAJBgNVBAYTAkJFMRkwFwYD VQQKExBHbG9iYWxTaWduIG52LXNhMRAwDgYDVQQLEwdSb290IENBMRswGQYDVQQDExJHbG9iYWxT aWduIFJvb3QgQ0EwHhcNOTgwOTAxMTIwMDAwWhcNMjgwMTI4MTIwMDAwWjBXMQswCQYDVQQGEwJC RTEZMBcGA1UEChMQR2xvYmFsU2lnbiBudi1zYTEQMA4GA1UECxMHUm9vdCBDQTEbMBkGA1UEAxMS R2xvYmFsU2lnbiBSb290IENBMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA2g7mmY3O o+NPin778YuDJWvqSB/xKrC5lREEvfBj0eJnZs8c3c8bSCvujYmOmq8pgGWr6cctEsurHExwB6E9 CjDNFY1P+N3UjFAVHO9Q7sQu9/zpUvKRfeBt1TUwjl5Dc/JB6dVq47KJOlY5OG8GPIhpWypNxadU uGyJzJv5PMrl/Yn1EjySeJbW3HRuk0Rh0Y3HRrJ1DoboGYrVbWzVeBaVounICjjr8iQTT3NUkxOF Ohu8HjS1iwWMuXeLsdsfIJGrCVNukM57N3S5cEeRIlFjFnmusa5BJgjIGSvRRqpI1mQq14M0/ywq wWwZQ0oHhefTfPYhaO/q8lKff5OQzwIDAQABo0IwQDAOBgNVHQ8BAf8EBAMCAQYwDwYDVR0TAQH/ BAUwAwEB/zAdBgNVHQ4EFgQUYHtmGkUNl8qJUC99BM00qP/8/UswDQYJKoZIhvcNAQEFBQADggEB ANZz53xPdtCNv+y6or40xSgytXz8bJwsK70JnlO/a16qEUi25Qijs8o9YU3TRgmzPsOg42NVG/K6 76054UO5OKPmL4omO++gUFb5xgr9OM3EC3BRlJeYBN/DX5TVFckUQZzEXXVkFQ3/VTDsho//De8s uWNG9qr837xp/S4SSGSa4JXwpu8pjwGxFbUMHaX+aSxpJHges6cccWLuysiXrBddisL4R4ZuKsRW MZXQZ4mFK/lspl1GnQyqguSZUd1wt9tWPWHkauFc1vb+Pd5BzAeuY1K/U1P0K+nH/bb3gl+F0kEY 24GzBBzFH6SAbxUgyd4MiAod1mZV4vxIySkmaeAwggRXMIIDP6ADAgECAgsEAAAAAAEvTuExRjAN BgkqhkiG9w0BAQUFADBXMQswCQYDVQQGEwJCRTEZMBcGA1UEChMQR2xvYmFsU2lnbiBudi1zYTEQ MA4GA1UECxMHUm9vdCBDQTEbMBkGA1UEAxMSR2xvYmFsU2lnbiBSb290IENBMB4XDTExMDQxMzEw MDAwMFoXDTE5MDQxMzEwMDAwMFowVDELMAkGA1UEBhMCQkUxGTAXBgNVBAoTEEdsb2JhbFNpZ24g bnYtc2ExKjAoBgNVBAMTIUdsb2JhbFNpZ24gUGVyc29uYWxTaWduIDIgQ0EgLSBHMjCCASIwDQYJ KoZIhvcNAQEBBQADggEPADCCAQoCggEBAMFrQfk17PgSfd0iUWlft7kTRid3FDvjE4FvPuR0F34M tfQs5AyNU9TcMCAov26OEf5mEeRRFsfdf/3hNBJQv/PYmOxpC9LQ2rJlceEzl566q7M4lHMRDz6h 0RPMeDYbQSu/vKNJ7DCCTANYMmdhQOU6NhMNQQbr6L7wyfjbmt6jgjQTbvvAPnjaSZVZ5bv6ge/l 1mj17VDJbCIpMQ/oERBVVIGBOFcwbi2tpJINFS3dPV5BNnHkQ5umIEQE7g5OqIFMl+Di8QhiCRfM oumd+zNMHpgwOlH39BLqncA0HeR8Bv63q51I7dYKy3QMavAcMsEUYNHhR5hPkoYacjtxYvsCAwEA AaOCASUwggEhMA4GA1UdDwEB/wQEAwIBBjASBgNVHRMBAf8ECDAGAQH/AgEAMB0GA1UdDgQWBBQ/ FdJtfC/nMZ5DCgaolGwsO8XuZTBHBgNVHSAEQDA+MDwGBFUdIAAwNDAyBggrBgEFBQcCARYmaHR0 cHM6Ly93d3cuZ2xvYmFsc2lnbi5jb20vcmVwb3NpdG9yeS8wMwYDVR0fBCwwKjAooCagJIYiaHR0 cDovL2NybC5nbG9iYWxzaWduLm5ldC9yb290LmNybDA9BggrBgEFBQcBAQQxMC8wLQYIKwYBBQUH MAGGIWh0dHA6Ly9vY3NwLmdsb2JhbHNpZ24uY29tL3Jvb3RyMTAfBgNVHSMEGDAWgBRge2YaRQ2X yolQL30EzTSo//z9SzANBgkqhkiG9w0BAQUFAAOCAQEAyFjhfKEB6SONcDttXFL4xPNfY7rhmtOR ecivxEqlvvbYfb34RvdnmgUi1YkyCVe4HwhW7ddVnsXyv+ODXoiTH4ucF+rCu157fECCuZR+2V9V oP8ytXF464EWmFDl/zAPj5rkZuIFldMxcGDitrr+DhYGjJla/vHp+ytWkGNnnCPWIRGg1jnEho+k jfA78z3ROBuLQZOP2iLFIHnfbBg6kWMHipWty0zY1z6bNWPs7FW4By7Y3TiAdGGdYa5QEHGNi/2/ lfmMqIuwjHTrZptUZuiYeSC28EE45VApILuclXX7PSwynDNEj3gpXZI32hlUnfp1ETDlMt4X8bDF bsffhTCCBPcwggPfoAMCAQICEQCWaWbA3qWpL+QmmSNuCrYvMA0GCSqGSIb3DQEBBQUAMFQxCzAJ BgNVBAYTAkJFMRkwFwYDVQQKExBHbG9iYWxTaWduIG52LXNhMSowKAYDVQQDEyFHbG9iYWxTaWdu IFBlcnNvbmFsU2lnbiAyIENBIC0gRzIwHhcNMTQwMzExMTUyMzUzWhcNMTcwMzExMTUyMzUzWjCB jDELMAkGA1UEBhMCVVMxETAPBgNVBAgTCE1hcnlsYW5kMRAwDgYDVQQHEwdIYW5vdmVyMR0wGwYD VQQKExRUaGUgS0VZVyBDb3Jwb3JhdGlvbjEVMBMGA1UEAxMMSmF5ZXNoIFBhdGVsMSIwIAYJKoZI hvcNAQkBFhNqcGF0ZWxAa2V5d2NvcnAuY29tMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKC AQEAviJ2YJa9fQYjlzp07KUG9t9T/mv12LO05QN0qMrRuwtFnzCyeeMR+LqNP8UEP8X2DsZ0vUuh U/nsUVMf9Al+sBnBCXrlGJiGv5olWPbuHu5Ai7xwPgOWdCzLCf83S4JhmN7Jg3kwKZh2QmeKn/l6 ShpsC3lq4bhT4QRagiFSbCWxGcUHU+0dp+offm1As+xK10jnHuHdQqJ2xkbhoQFgd8F9YWB9bOys OJ482IfGMsF9xi218u7BD8S83RvFfXt2jNuiskRxEsuhw++3n2wsd+sDGEdxHxx9ZdwFgiq/lEn0 lvRe+jManRoM5IAm95/b3OA9RkRrlt2k5Z4NkLCMJQIDAQABo4IBiTCCAYUwDgYDVR0PAQH/BAQD AgWgME0GA1UdIARGMEQwQgYKKwYBBAGgMgEoCjA0MDIGCCsGAQUFBwIBFiZodHRwczovL3d3dy5n bG9iYWxzaWduLmNvbS9yZXBvc2l0b3J5LzAeBgNVHREEFzAVgRNqcGF0ZWxAa2V5d2NvcnAuY29t MAkGA1UdEwQCMAAwHQYDVR0lBBYwFAYIKwYBBQUHAwIGCCsGAQUFBwMEMEMGA1UdHwQ8MDowOKA2 oDSGMmh0dHA6Ly9jcmwuZ2xvYmFsc2lnbi5jb20vZ3MvZ3NwZXJzb25hbHNpZ24yZzIuY3JsMFUG CCsGAQUFBwEBBEkwRzBFBggrBgEFBQcwAoY5aHR0cDovL3NlY3VyZS5nbG9iYWxzaWduLmNvbS9j YWNlcnQvZ3NwZXJzb25hbHNpZ24yZzIuY3J0MB0GA1UdDgQWBBRK+1gM3fWyol7yk5ta3Imdrlkt WTAfBgNVHSMEGDAWgBQ/FdJtfC/nMZ5DCgaolGwsO8XuZTANBgkqhkiG9w0BAQUFAAOCAQEAEElx YCqGUrYLTgKqr2c7fBz+Tot6CBR+MhiJi5bmoqyF7X16dfZ0RZ4+MxDNRFfMAu5rpVbN8wAGQlmf EJOK695rT4GMh8i1v81aBDeLggCCUCDlbDSjExocwCJPum1VbJ3EnoRI9ScT8ZHagV6Ngb6ePo4D XY1rvLSRHjcqnwlzMjSuyJNrKO2qAML/Hfs5FcLCb2yRQxdWSM+AVNxWY8KLE0pNeutwKoukJvle 6mLjC2fWws+rmb23dZwUczZBh6CskoKRxvTfdVAOWqJXXUXVlMHkQHo9DypLt3kDjJVek7pe96Ql 6vCmqSM9OkVX7Zq1qAoCWRjFXcxRJ7FejDGCA30wggN5AgEBMGkwVDELMAkGA1UEBhMCQkUxGTAX BgNVBAoTEEdsb2JhbFNpZ24gbnYtc2ExKjAoBgNVBAMTIUdsb2JhbFNpZ24gUGVyc29uYWxTaWdu IDIgQ0EgLSBHMgIRAJZpZsDepakv5CaZI24Kti8wCQYFKw4DAhoFAKCCAekwGAYJKoZIhvcNAQkD MQsGCSqGSIb3DQEHATAcBgkqhkiG9w0BCQUxDxcNMTYwNTMxMjIyNDI5WjAjBgkqhkiG9w0BCQQx FgQUFIpaImtL8/qVZ3rKczDqPIU94KUweAYJKwYBBAGCNxAEMWswaTBUMQswCQYDVQQGEwJCRTEZ MBcGA1UEChMQR2xvYmFsU2lnbiBudi1zYTEqMCgGA1UEAxMhR2xvYmFsU2lnbiBQZXJzb25hbFNp Z24gMiBDQSAtIEcyAhEAlmlmwN6lqS/kJpkjbgq2LzB6BgsqhkiG9w0BCRACCzFroGkwVDELMAkG A1UEBhMCQkUxGTAXBgNVBAoTEEdsb2JhbFNpZ24gbnYtc2ExKjAoBgNVBAMTIUdsb2JhbFNpZ24g UGVyc29uYWxTaWduIDIgQ0EgLSBHMgIRAJZpZsDepakv5CaZI24Kti8wgZMGCSqGSIb3DQEJDzGB hTCBgjALBglghkgBZQMEASowCwYJYIZIAWUDBAEWMAoGCCqGSIb3DQMHMAsGCWCGSAFlAwQBAjAO BggqhkiG9w0DAgICAIAwDQYIKoZIhvcNAwICAUAwBwYFKw4DAhowCwYJYIZIAWUDBAIDMAsGCWCG SAFlAwQCAjALBglghkgBZQMEAgEwDQYJKoZIhvcNAQEBBQAEggEAfypuUlkTOzSA59VaXvucCoLn u7zwOvOSJ882/w8dQRCImmkg8gDX9a4zcolhSI5SrNz2wb/0yJCtV19kXEeFB9B4ScFen0SoUDV+ g7wzKXa/cR65wTOGFrXnaBlqfxuh7rScAGwfxnLhd073xclL0P6iPXhdIayavShhvaEABUoNYAmk TjETq42OI3QHZDs2MsvO7Cb54iXchNbGZOrzzArndsgdraxGgRBat5f7j7gETP1PZmIO5qpFARYC EwcAmxBg3F8aXEadWSbR2ruNITNw04CZZhcfXbzHeFx9aVyLpsR+0yuCCnnhDFgYwmPCX3zXVyn/ w7VNMh2lZU4CvwAAAAAAAA== ------=_NextPart_000_03DC_01D1BB69.ABABD3A0--