Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id AFF849C7A for ; Tue, 21 May 2013 09:27:23 +0000 (UTC) Received: (qmail 11996 invoked by uid 500); 21 May 2013 09:27:21 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 11643 invoked by uid 500); 21 May 2013 09:27:21 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 11609 invoked by uid 99); 21 May 2013 09:27:19 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 May 2013 09:27:19 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,NORMAL_HTTP_TO_IP,RCVD_IN_DNSWL_LOW,SPF_PASS,TO_NO_BRKTS_PCNT X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of vlad.volkov@gmail.com designates 209.85.220.50 as permitted sender) Received: from [209.85.220.50] (HELO mail-pa0-f50.google.com) (209.85.220.50) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 May 2013 09:27:13 +0000 Received: by mail-pa0-f50.google.com with SMTP id fb10so510567pad.9 for ; Tue, 21 May 2013 02:26:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:from:date:message-id:subject:to:content-type; bh=eEiczq+hHTfzKMMt/XxMkKBHxFT2MDbgbVUisT4beP8=; b=gQsgZkl6Fswoc0TI8jebc44uXMt/dla3ywZ9+z61ng/QmnFp49SsN3MMYzlFE44CLi R92jRknpY+bqW6HTzloQCx1j5TiHp/aXacUlZl+1mOslZsuMsBaRFHCjHF84R54gs2iR +/9SvTys7hXX8INEjNqu76gpdIiA6IDSQxdTgk6/UtfNGQ4DpeOL7NuA7ulP+5J2O54S 0zZHS8pvBTUR2Rfb5dp/Ynd5TfcKrtNkMhk3w3Aafk/LYjtcaZe/zN63PKS1cGWuVwnF RHU+e2QoPh69lVAiLv2UATG9ELbUaQcKYD7xdYLmF3iqUrQ5kBhQ44bLQOk7MGCfeetB lGeQ== X-Received: by 10.68.40.200 with SMTP id z8mr1612882pbk.210.1369128411646; Tue, 21 May 2013 02:26:51 -0700 (PDT) MIME-Version: 1.0 Received: by 10.68.203.73 with HTTP; Tue, 21 May 2013 02:26:31 -0700 (PDT) From: Vladimir Volkov Date: Tue, 21 May 2013 13:26:31 +0400 Message-ID: Subject: Cassandra hangs on large hinted handoffs To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=bcaec51a74aca0d35004dd370c38 X-Virus-Checked: Checked by ClamAV on apache.org --bcaec51a74aca0d35004dd370c38 Content-Type: text/plain; charset=UTF-8 Hello. I'm stress-testing our Cassandra (version 1.0.9) cluster, and tried turning off two of the four nodes for half an hour under heavy load. As a result I got a large volume of hints on the alive nodes - HintsColumnFamily takes about 1.5 GB disk space on each of the nodes. It seems, these hints are never replayed successfully. After I bring other nodes back online, tpstats shows active handoffs, but I can't see any writes on the target nodes. The log indicates memory pressure - the heap is >80% full (heap size is 8GB total, 1GB young). A fragment of the log: INFO 18:34:05,513 Started hinted handoff for token: 1 with IP: / 84.201.162.144 INFO 18:34:06,794 GC for ParNew: 300 ms for 1 collections, 5974181760 used; max is 8588951552 INFO 18:34:07,795 GC for ParNew: 263 ms for 1 collections, 6226018744 used; max is 8588951552 INFO 18:34:08,795 GC for ParNew: 256 ms for 1 collections, 6559918392 used; max is 8588951552 INFO 18:34:09,796 GC for ParNew: 231 ms for 1 collections, 6846133712 used; max is 8588951552 WARN 18:34:09,805 Heap is 0.7978131149667941 full. You may need to reduce memtable and/or cache sizes. Cassandra will now flush up to the two largest memtables to free up memory. WARN 18:34:09,805 Flushing CFS(Keyspace='test', ColumnFamily='t2') to relieve memory pressure INFO 18:34:09,806 Enqueuing flush of Memtable-t2@639524673(60608588/571839171 serialized/live bytes, 743266 ops) INFO 18:34:09,807 Writing Memtable-t2@639524673(60608588/571839171 serialized/live bytes, 743266 ops) INFO 18:34:11,018 GC for ParNew: 449 ms for 2 collections, 6573394480 used; max is 8588951552 INFO 18:34:12,019 GC for ParNew: 265 ms for 1 collections, 6820930056 used; max is 8588951552 INFO 18:34:13,112 GC for ParNew: 331 ms for 1 collections, 6900566728 used; max is 8588951552 INFO 18:34:14,181 GC for ParNew: 269 ms for 1 collections, 7101358936 used; max is 8588951552 INFO 18:34:14,691 Completed flushing /mnt/raid/cassandra/data/test/t2-hc-244-Data.db (56156246 bytes) INFO 18:34:15,381 GC for ParNew: 280 ms for 1 collections, 7268441248 used; max is 8588951552 INFO 18:34:35,306 InetAddress /84.201.162.144 is now dead. INFO 18:34:35,306 GC for ConcurrentMarkSweep: 19223 ms for 1 collections, 3774714808 used; max is 8588951552 INFO 18:34:35,309 InetAddress /84.201.162.144 is now UP After taking off the load and restatring the service, I still see pending handoffs: $ nodetool -h localhost tpstats Pool Name Active Pending Completed Blocked All time blocked ReadStage 0 0 1004257 0 0 RequestResponseStage 0 0 92555 0 0 MutationStage 0 0 6 0 0 ReadRepairStage 0 0 57773 0 0 ReplicateOnWriteStage 0 0 0 0 0 GossipStage 0 0 143332 0 0 AntiEntropyStage 0 0 0 0 0 MigrationStage 0 0 0 0 0 MemtablePostFlusher 0 0 2 0 0 StreamStage 0 0 0 0 0 FlushWriter 0 0 2 0 0 MiscStage 0 0 0 0 0 InternalResponseStage 0 0 0 0 0 HintedHandoff 1 3 15 0 0 These 3 handoffs remain pending for a long time (>12 hours). Most of the time Cassandra uses 100% of one CPU core, the stack trace of the busy thread is: "HintedHandoff:1" daemon prio=10 tid=0x0000000001220800 nid=0x3843 runnable [0x00007fa1e1146000] java.lang.Thread.State: RUNNABLE at java.util.ArrayList$Itr.remove(ArrayList.java:808) at org.apache.cassandra.db.ColumnFamilyStore.removeDeletedSuper(ColumnFamilyStore.java:908) at org.apache.cassandra.db.ColumnFamilyStore.removeDeletedColumnsOnly(ColumnFamilyStore.java:857) at org.apache.cassandra.db.ColumnFamilyStore.removeDeleted(ColumnFamilyStore.java:850) at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1195) at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1150) at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpointInternal(HintedHandOffManager.java:324) at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:256) at org.apache.cassandra.db.HintedHandOffManager.access$300(HintedHandOffManager.java:84) at org.apache.cassandra.db.HintedHandOffManager$3.runMayThrow(HintedHandOffManager.java:437) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:722) Heap usage is also rather high, though the node isn't doing anything, except the HH processing. Here is CMS output: 2013-05-20T22:22:59.812+0400: 4672.075: [GC[YG occupancy: 70070 K (943744 K)]4672.075: [Rescan (parallel) , 0.0224060 secs]4672.098: [weak refs processing, 0.0002900 secs]4672.098: [scrub string table, 0.0002670 secs] [1 CMS-remark: 5523830K(7340032K)] 5593901K(8283776K), 0.0231160 secs] [Times: user=0.28 sys=0.00, real=0.02 secs] Eventually, after a few service restarts, the hints suddenly disappear. Probably, the TTL expires and the hints get compacted away. Currently my best guess is the following. Hinted handoffs are stored as supercolumns, with one row per target node. The service tries to read them entirely into memory for replay and fails, because the volume is too large to fit in the heap at once. Then the TTL expires, and the service starts to delete old subcolumns during read. Since the underlying storage is a huge ArrayList, the deletion is inefficient and takes forever. So, it seems there're two problems here. 1) Hints are not paged correctly and cause significant memory pressure - that's actually strange, since the same issue was supposedly addressed in https://issues.apache.org/jira/browse/CASSANDRA-1327 and https://issues.apache.org/jira/browse/CASSANDRA-3624; 2) Deletion of outdated hints doesn't work well for large hint volumes. Any suggestions on how to make the cluster more tolerant to downtimes? If I turn off the hinted handoff entirely, and manually run a repair after a downtime, will it restore all the data correctly? -- Best regards, Vladimir --bcaec51a74aca0d35004dd370c38 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: base64 PGRpdiBkaXI9Imx0ciI+PGRpdj48ZGl2PjxkaXY+SGVsbG8uPGJyPjxicj48L2Rpdj5JJiMzOTtt IHN0cmVzcy10ZXN0aW5nIG91ciBDYXNzYW5kcmEgKHZlcnNpb24gMS4wLjkpIGNsdXN0ZXIsIGFu ZCB0cmllZCB0dXJuaW5nIG9mZiB0d28gb2YgdGhlIGZvdXIgbm9kZXMgZm9yIGhhbGYgYW4gaG91 ciB1bmRlciBoZWF2eSBsb2FkLiBBcyBhIHJlc3VsdCBJIGdvdCBhIGxhcmdlIHZvbHVtZSBvZiBo aW50cyBvbiB0aGUgYWxpdmUgbm9kZXMgLSBIaW50c0NvbHVtbkZhbWlseSB0YWtlcyBhYm91dCAx LjUgR0IgZGlzayBzcGFjZSBvbiBlYWNoIG9mIHRoZSBub2Rlcy4gSXQgc2VlbXMsIHRoZXNlIGhp bnRzIGFyZSBuZXZlciByZXBsYXllZCBzdWNjZXNzZnVsbHkuPGJyPg0KDQo8YnI+PC9kaXY+QWZ0 ZXIgSSBicmluZyBvdGhlciBub2RlcyBiYWNrIG9ubGluZSwgdHBzdGF0cyBzaG93cyBhY3RpdmUg aGFuZG9mZnMsIGJ1dCBJIGNhbiYjMzk7dCBzZWUgYW55IHdyaXRlcyBvbiB0aGUgdGFyZ2V0IG5v ZGVzLjxicj48L2Rpdj48ZGl2PlRoZSBsb2cgaW5kaWNhdGVzIG1lbW9yeSBwcmVzc3VyZSAtIHRo ZSBoZWFwIGlzICZndDs4MCUgZnVsbCAoaGVhcCBzaXplIGlzIDhHQiB0b3RhbCwgMUdCIHlvdW5n KS48YnI+DQoNCjxicj5BIGZyYWdtZW50IG9mIHRoZSBsb2c6PGJyPsKgSU5GTyAxODozNDowNSw1 MTMgU3RhcnRlZCBoaW50ZWQgaGFuZG9mZiBmb3IgdG9rZW46IDEgd2l0aCBJUDogLzxhIGhyZWY9 Imh0dHA6Ly84NC4yMDEuMTYyLjE0NCI+ODQuMjAxLjE2Mi4xNDQ8L2E+PGJyPsKgSU5GTyAxODoz NDowNiw3OTQgR0MgZm9yIFBhck5ldzogMzAwIG1zIGZvciAxIGNvbGxlY3Rpb25zLCA1OTc0MTgx NzYwIHVzZWQ7IG1heCBpcyA4NTg4OTUxNTUyPGJyPg0KDQrCoElORk8gMTg6MzQ6MDcsNzk1IEdD IGZvciBQYXJOZXc6IDI2MyBtcyBmb3IgMSBjb2xsZWN0aW9ucywgNjIyNjAxODc0NCB1c2VkOyBt YXggaXMgODU4ODk1MTU1Mjxicj7CoElORk8gMTg6MzQ6MDgsNzk1IEdDIGZvciBQYXJOZXc6IDI1 NiBtcyBmb3IgMSBjb2xsZWN0aW9ucywgNjU1OTkxODM5MiB1c2VkOyBtYXggaXMgODU4ODk1MTU1 Mjxicj7CoElORk8gMTg6MzQ6MDksNzk2IEdDIGZvciBQYXJOZXc6IDIzMSBtcyBmb3IgMSBjb2xs ZWN0aW9ucywgNjg0NjEzMzcxMiB1c2VkOyBtYXggaXMgODU4ODk1MTU1Mjxicj4NCg0KwqBXQVJO IDE4OjM0OjA5LDgwNSBIZWFwIGlzIDAuNzk3ODEzMTE0OTY2Nzk0MSBmdWxsLsKgIFlvdSBtYXkg bmVlZCB0byByZWR1Y2UgbWVtdGFibGUgYW5kL29yIGNhY2hlIHNpemVzLsKgIENhc3NhbmRyYSB3 aWxsIG5vdyBmbHVzaCB1cCB0byB0aGUgdHdvIGxhcmdlc3QgbWVtdGFibGVzIHRvIGZyZWUgdXAg bWVtb3J5Ljxicj7CoFdBUk4gMTg6MzQ6MDksODA1IEZsdXNoaW5nIENGUyhLZXlzcGFjZT0mIzM5 O3Rlc3QmIzM5OywgQ29sdW1uRmFtaWx5PSYjMzk7dDImIzM5OykgdG8gcmVsaWV2ZSBtZW1vcnkg cHJlc3N1cmU8YnI+DQoNCsKgSU5GTyAxODozNDowOSw4MDYgRW5xdWV1aW5nIGZsdXNoIG9mIE1l bXRhYmxlLXQyQDYzOTUyNDY3Myg2MDYwODU4OC81NzE4MzkxNzEgc2VyaWFsaXplZC9saXZlIGJ5 dGVzLCA3NDMyNjYgb3BzKTxicj7CoElORk8gMTg6MzQ6MDksODA3IFdyaXRpbmcgTWVtdGFibGUt dDJANjM5NTI0NjczKDYwNjA4NTg4LzU3MTgzOTE3MSBzZXJpYWxpemVkL2xpdmUgYnl0ZXMsIDc0 MzI2NiBvcHMpPGJyPg0KDQrCoElORk8gMTg6MzQ6MTEsMDE4IEdDIGZvciBQYXJOZXc6IDQ0OSBt cyBmb3IgMiBjb2xsZWN0aW9ucywgNjU3MzM5NDQ4MCB1c2VkOyBtYXggaXMgODU4ODk1MTU1Mjxi cj7CoElORk8gMTg6MzQ6MTIsMDE5IEdDIGZvciBQYXJOZXc6IDI2NSBtcyBmb3IgMSBjb2xsZWN0 aW9ucywgNjgyMDkzMDA1NiB1c2VkOyBtYXggaXMgODU4ODk1MTU1Mjxicj7CoElORk8gMTg6MzQ6 MTMsMTEyIEdDIGZvciBQYXJOZXc6IDMzMSBtcyBmb3IgMSBjb2xsZWN0aW9ucywgNjkwMDU2Njcy OCB1c2VkOyBtYXggaXMgODU4ODk1MTU1Mjxicj4NCg0KwqBJTkZPIDE4OjM0OjE0LDE4MSBHQyBm b3IgUGFyTmV3OiAyNjkgbXMgZm9yIDEgY29sbGVjdGlvbnMsIDcxMDEzNTg5MzYgdXNlZDsgbWF4 IGlzIDg1ODg5NTE1NTI8YnI+wqBJTkZPIDE4OjM0OjE0LDY5MSBDb21wbGV0ZWQgZmx1c2hpbmcg L21udC9yYWlkL2Nhc3NhbmRyYS9kYXRhL3Rlc3QvdDItaGMtMjQ0LURhdGEuZGIgKDU2MTU2MjQ2 IGJ5dGVzKTxicj7CoElORk8gMTg6MzQ6MTUsMzgxIEdDIGZvciBQYXJOZXc6IDI4MCBtcyBmb3Ig MSBjb2xsZWN0aW9ucywgNzI2ODQ0MTI0OCB1c2VkOyBtYXggaXMgODU4ODk1MTU1Mjxicj4NCg0K wqBJTkZPIDE4OjM0OjM1LDMwNiBJbmV0QWRkcmVzcyAvPGEgaHJlZj0iaHR0cDovLzg0LjIwMS4x NjIuMTQ0Ij44NC4yMDEuMTYyLjE0NDwvYT4gaXMgbm93IGRlYWQuPGJyPsKgSU5GTyAxODozNDoz NSwzMDYgR0MgZm9yIENvbmN1cnJlbnRNYXJrU3dlZXA6IDE5MjIzIG1zIGZvciAxIGNvbGxlY3Rp b25zLCAzNzc0NzE0ODA4IHVzZWQ7IG1heCBpcyA4NTg4OTUxNTUyPGJyPsKgSU5GTyAxODozNDoz NSwzMDkgSW5ldEFkZHJlc3MgLzxhIGhyZWY9Imh0dHA6Ly84NC4yMDEuMTYyLjE0NCI+ODQuMjAx LjE2Mi4xNDQ8L2E+IGlzIG5vdyBVUDxicj4NCg0KPGJyPjwvZGl2PjxkaXY+QWZ0ZXIgdGFraW5n IG9mZiB0aGUgbG9hZCBhbmQgcmVzdGF0cmluZyB0aGUgc2VydmljZSwgSSBzdGlsbCBzZWUgcGVu ZGluZyBoYW5kb2Zmczo8YnI+PC9kaXY+PGRpdj48ZGl2PjxkaXY+PGRpdj4kIG5vZGV0b29sIC1o IGxvY2FsaG9zdCB0cHN0YXRzPGJyPlBvb2wgTmFtZcKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKg wqDCoMKgwqDCoMKgIEFjdGl2ZcKgwqAgUGVuZGluZ8KgwqDCoMKgwqAgQ29tcGxldGVkwqDCoCBC bG9ja2VkwqAgQWxsIHRpbWUgYmxvY2tlZDxicj4NCg0KUmVhZFN0YWdlwqDCoMKgwqDCoMKgwqDC oMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgIDDCoMKgwqDCoMKgwqDCoMKgIDDCoMKg wqDCoMKgwqDCoCAxMDA0MjU3wqDCoMKgwqDCoMKgwqDCoCAwwqDCoMKgwqDCoMKgwqDCoMKgwqDC oMKgwqDCoMKgwqAgMDxicj5SZXF1ZXN0UmVzcG9uc2VTdGFnZcKgwqDCoMKgwqDCoMKgwqDCoMKg wqDCoMKgIDDCoMKgwqDCoMKgwqDCoMKgIDDCoMKgwqDCoMKgwqDCoMKgwqAgOTI1NTXCoMKgwqDC oMKgwqDCoMKgIDDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoCAwPGJyPk11dGF0aW9u U3RhZ2XCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgIDDCoMKgwqDCoMKg wqDCoMKgIDDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoCA2wqDCoMKgwqDCoMKgwqDCoCAwwqDC oMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqAgMDxicj4NCg0KUmVhZFJlcGFpclN0YWdlwqDC oMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgIDDCoMKgwqDCoMKgwqDCoMKgIDDCoMKg wqDCoMKgwqDCoMKgwqAgNTc3NzPCoMKgwqDCoMKgwqDCoMKgIDDCoMKgwqDCoMKgwqDCoMKgwqDC oMKgwqDCoMKgwqDCoCAwPGJyPlJlcGxpY2F0ZU9uV3JpdGVTdGFnZcKgwqDCoMKgwqDCoMKgwqDC oMKgwqDCoCAwwqDCoMKgwqDCoMKgwqDCoCAwwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqAgMMKg wqDCoMKgwqDCoMKgwqAgMMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgIDA8YnI+R29z c2lwU3RhZ2XCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoCAwwqDC oMKgwqDCoMKgwqDCoCAwwqDCoMKgwqDCoMKgwqDCoCAxNDMzMzLCoMKgwqDCoMKgwqDCoMKgIDDC oMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoCAwPGJyPg0KDQpBbnRpRW50cm9weVN0YWdl wqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoCAwwqDCoMKgwqDCoMKgwqDCoCAwwqDC oMKgwqDCoMKgwqDCoMKgwqDCoMKgwqAgMMKgwqDCoMKgwqDCoMKgwqAgMMKgwqDCoMKgwqDCoMKg wqDCoMKgwqDCoMKgwqDCoMKgIDA8YnI+TWlncmF0aW9uU3RhZ2XCoMKgwqDCoMKgwqDCoMKgwqDC oMKgwqDCoMKgwqDCoMKgwqDCoCAwwqDCoMKgwqDCoMKgwqDCoCAwwqDCoMKgwqDCoMKgwqDCoMKg wqDCoMKgwqAgMMKgwqDCoMKgwqDCoMKgwqAgMMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDC oMKgIDA8YnI+TWVtdGFibGVQb3N0Rmx1c2hlcsKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqAg MMKgwqDCoMKgwqDCoMKgwqAgMMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgIDLCoMKgwqDCoMKg wqDCoMKgIDDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoCAwPGJyPg0KDQpTdHJlYW1T dGFnZcKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgIDDCoMKgwqDC oMKgwqDCoMKgIDDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoCAwwqDCoMKgwqDCoMKgwqDCoCAw wqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqAgMDxicj5GbHVzaFdyaXRlcsKgwqDCoMKg wqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgIDDCoMKgwqDCoMKgwqDCoMKgIDDC oMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoCAywqDCoMKgwqDCoMKgwqDCoCAwwqDCoMKgwqDCoMKg wqDCoMKgwqDCoMKgwqDCoMKgwqAgMDxicj5NaXNjU3RhZ2XCoMKgwqDCoMKgwqDCoMKgwqDCoMKg wqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqAgMMKgwqDCoMKgwqDCoMKgwqAgMMKgwqDCoMKgwqDC oMKgwqDCoMKgwqDCoMKgIDDCoMKgwqDCoMKgwqDCoMKgIDDCoMKgwqDCoMKgwqDCoMKgwqDCoMKg wqDCoMKgwqDCoCAwPGJyPg0KDQpJbnRlcm5hbFJlc3BvbnNlU3RhZ2XCoMKgwqDCoMKgwqDCoMKg wqDCoMKgwqAgMMKgwqDCoMKgwqDCoMKgwqAgMMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgIDDC oMKgwqDCoMKgwqDCoMKgIDDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoCAwPGJyPkhp bnRlZEhhbmRvZmbCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgIDHCoMKg wqDCoMKgwqDCoMKgIDPCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqAgMTXCoMKgwqDCoMKgwqDCoMKg IDDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoCAwPGJyPjxicj5UaGVzZSAzIGhhbmRv ZmZzIHJlbWFpbiBwZW5kaW5nIGZvciBhIGxvbmcgdGltZSAoJmd0OzEyIGhvdXJzKS48YnI+DQoN CjwvZGl2PjxkaXY+TW9zdCBvZiB0aGUgdGltZSBDYXNzYW5kcmEgdXNlcyAxMDAlIG9mIG9uZSBD UFUgY29yZSwgdGhlIHN0YWNrIHRyYWNlIG9mIHRoZSBidXN5IHRocmVhZCBpczo8YnI+JnF1b3Q7 SGludGVkSGFuZG9mZjoxJnF1b3Q7IGRhZW1vbiBwcmlvPTEwIHRpZD0weDAwMDAwMDAwMDEyMjA4 MDAgbmlkPTB4Mzg0MyBydW5uYWJsZSBbMHgwMDAwN2ZhMWUxMTQ2MDAwXTxicj7CoMKgIGphdmEu bGFuZy5UaHJlYWQuU3RhdGU6IFJVTk5BQkxFPGJyPg0KDQrCoMKgwqDCoMKgwqDCoCBhdCBqYXZh LnV0aWwuQXJyYXlMaXN0JEl0ci5yZW1vdmUoQXJyYXlMaXN0LmphdmE6ODA4KTxicj7CoMKgwqDC oMKgwqDCoCBhdCBvcmcuYXBhY2hlLmNhc3NhbmRyYS5kYi5Db2x1bW5GYW1pbHlTdG9yZS5yZW1v dmVEZWxldGVkU3VwZXIoQ29sdW1uRmFtaWx5U3RvcmUuamF2YTo5MDgpPGJyPsKgwqDCoMKgwqDC oMKgIGF0IG9yZy5hcGFjaGUuY2Fzc2FuZHJhLmRiLkNvbHVtbkZhbWlseVN0b3JlLnJlbW92ZURl bGV0ZWRDb2x1bW5zT25seShDb2x1bW5GYW1pbHlTdG9yZS5qYXZhOjg1Nyk8YnI+DQoNCsKgwqDC oMKgwqDCoMKgIGF0IG9yZy5hcGFjaGUuY2Fzc2FuZHJhLmRiLkNvbHVtbkZhbWlseVN0b3JlLnJl bW92ZURlbGV0ZWQoQ29sdW1uRmFtaWx5U3RvcmUuamF2YTo4NTApPGJyPsKgwqDCoMKgwqDCoMKg IGF0IG9yZy5hcGFjaGUuY2Fzc2FuZHJhLmRiLkNvbHVtbkZhbWlseVN0b3JlLmdldENvbHVtbkZh bWlseShDb2x1bW5GYW1pbHlTdG9yZS5qYXZhOjExOTUpPGJyPsKgwqDCoMKgwqDCoMKgIGF0IG9y Zy5hcGFjaGUuY2Fzc2FuZHJhLmRiLkNvbHVtbkZhbWlseVN0b3JlLmdldENvbHVtbkZhbWlseShD b2x1bW5GYW1pbHlTdG9yZS5qYXZhOjExNTApPGJyPg0KDQrCoMKgwqDCoMKgwqDCoCBhdCBvcmcu YXBhY2hlLmNhc3NhbmRyYS5kYi5IaW50ZWRIYW5kT2ZmTWFuYWdlci5kZWxpdmVySGludHNUb0Vu ZHBvaW50SW50ZXJuYWwoSGludGVkSGFuZE9mZk1hbmFnZXIuamF2YTozMjQpPGJyPsKgwqDCoMKg wqDCoMKgIGF0IG9yZy5hcGFjaGUuY2Fzc2FuZHJhLmRiLkhpbnRlZEhhbmRPZmZNYW5hZ2VyLmRl bGl2ZXJIaW50c1RvRW5kcG9pbnQoSGludGVkSGFuZE9mZk1hbmFnZXIuamF2YToyNTYpPGJyPg0K DQrCoMKgwqDCoMKgwqDCoCBhdCBvcmcuYXBhY2hlLmNhc3NhbmRyYS5kYi5IaW50ZWRIYW5kT2Zm TWFuYWdlci5hY2Nlc3MkMzAwKEhpbnRlZEhhbmRPZmZNYW5hZ2VyLmphdmE6ODQpPGJyPsKgwqDC oMKgwqDCoMKgIGF0IG9yZy5hcGFjaGUuY2Fzc2FuZHJhLmRiLkhpbnRlZEhhbmRPZmZNYW5hZ2Vy JDMucnVuTWF5VGhyb3coSGludGVkSGFuZE9mZk1hbmFnZXIuamF2YTo0MzcpPGJyPsKgwqDCoMKg wqDCoMKgIGF0IG9yZy5hcGFjaGUuY2Fzc2FuZHJhLnV0aWxzLldyYXBwZWRSdW5uYWJsZS5ydW4o V3JhcHBlZFJ1bm5hYmxlLmphdmE6MzApPGJyPg0KDQrCoMKgwqDCoMKgwqDCoCBhdCBqYXZhLnV0 aWwuY29uY3VycmVudC5UaHJlYWRQb29sRXhlY3V0b3IucnVuV29ya2VyKFRocmVhZFBvb2xFeGVj dXRvci5qYXZhOjExMTApPGJyPsKgwqDCoMKgwqDCoMKgIGF0IGphdmEudXRpbC5jb25jdXJyZW50 LlRocmVhZFBvb2xFeGVjdXRvciRXb3JrZXIucnVuKFRocmVhZFBvb2xFeGVjdXRvci5qYXZhOjYw Myk8YnI+wqDCoMKgwqDCoMKgwqAgYXQgamF2YS5sYW5nLlRocmVhZC5ydW4oVGhyZWFkLmphdmE6 NzIyKTxicj4NCg0KPGJyPjwvZGl2PjxkaXY+SGVhcCB1c2FnZSBpcyBhbHNvIHJhdGhlciBoaWdo LCB0aG91Z2ggdGhlIG5vZGUgaXNuJiMzOTt0IGRvaW5nIGFueXRoaW5nLCBleGNlcHQgdGhlIEhI IHByb2Nlc3NpbmcuIEhlcmUgaXMgQ01TIG91dHB1dDo8YnI+MjAxMy0wNS0yMFQyMjoyMjo1OS44 MTIrMDQwMDogNDY3Mi4wNzU6IFtHQ1tZRyBvY2N1cGFuY3k6IDcwMDcwIEsgKDk0Mzc0NCBLKV00 NjcyLjA3NTogW1Jlc2NhbiAocGFyYWxsZWwpICwgMC4wMjI0MDYwIHNlY3NdNDY3Mi4wOTg6IFt3 ZWFrIHJlZnMgcHJvY2Vzc2luZywgMC4wMDAyOTAwIHNlY3NdNDY3Mi4wOTg6IFtzY3J1YiBzdHJp bmcgdGFibGUsIDAuMDAwMjY3MCBzZWNzXSBbMSBDTVMtcmVtYXJrOiA1NTIzODMwSyg3MzQwMDMy SyldIDU1OTM5MDFLKDgyODM3NzZLKSwgMC4wMjMxMTYwIHNlY3NdIFtUaW1lczogdXNlcj0wLjI4 IHN5cz0wLjAwLCByZWFsPTAuMDIgc2Vjc108YnI+DQoNCjxicj48L2Rpdj48ZGl2PjwvZGl2Pjxk aXY+RXZlbnR1YWxseSwgYWZ0ZXIgYSBmZXcgc2VydmljZSByZXN0YXJ0cywgdGhlIGhpbnRzIHN1 ZGRlbmx5IGRpc2FwcGVhci4gUHJvYmFibHksIHRoZSBUVEwgZXhwaXJlcyBhbmQgdGhlIGhpbnRz IGdldCBjb21wYWN0ZWQgYXdheS48YnI+PGJyPjxicj48L2Rpdj48ZGl2PkN1cnJlbnRseSBteSBi ZXN0IGd1ZXNzIGlzIHRoZSBmb2xsb3dpbmcuIEhpbnRlZCBoYW5kb2ZmcyBhcmUgc3RvcmVkIGFz IHN1cGVyY29sdW1ucywgd2l0aCBvbmUgcm93IHBlciB0YXJnZXQgbm9kZS4gVGhlIHNlcnZpY2Ug dHJpZXMgdG8gcmVhZCB0aGVtIGVudGlyZWx5IGludG8gbWVtb3J5IGZvciByZXBsYXkgYW5kIGZh aWxzLCBiZWNhdXNlIHRoZSB2b2x1bWUgaXMgdG9vIGxhcmdlIHRvIGZpdCBpbiB0aGUgaGVhcCBh dCBvbmNlLjxicj4NCg0KPC9kaXY+PGRpdj5UaGVuIHRoZSBUVEwgZXhwaXJlcywgYW5kIHRoZSBz ZXJ2aWNlIHN0YXJ0cyB0byBkZWxldGUgb2xkIHN1YmNvbHVtbnMgZHVyaW5nIHJlYWQuIFNpbmNl IHRoZSB1bmRlcmx5aW5nIHN0b3JhZ2UgaXMgYSBodWdlIEFycmF5TGlzdCwgdGhlIGRlbGV0aW9u IGlzIGluZWZmaWNpZW50IGFuZCB0YWtlcyBmb3JldmVyLjxicj48L2Rpdj48ZGl2Pjxicj5Tbywg aXQgc2VlbXMgdGhlcmUmIzM5O3JlIHR3byBwcm9ibGVtcyBoZXJlLjxicj4NCg0KPC9kaXY+PGRp dj4xKSBIaW50cyBhcmUgbm90IHBhZ2VkIGNvcnJlY3RseSBhbmQgY2F1c2Ugc2lnbmlmaWNhbnQg bWVtb3J5IHByZXNzdXJlIC0gdGhhdCYjMzk7cyBhY3R1YWxseSBzdHJhbmdlLCBzaW5jZSB0aGUg c2FtZSBpc3N1ZSB3YXMgc3VwcG9zZWRseSBhZGRyZXNzZWQgaW4gPGEgaHJlZj0iaHR0cHM6Ly9p c3N1ZXMuYXBhY2hlLm9yZy9qaXJhL2Jyb3dzZS9DQVNTQU5EUkEtMTMyNyI+aHR0cHM6Ly9pc3N1 ZXMuYXBhY2hlLm9yZy9qaXJhL2Jyb3dzZS9DQVNTQU5EUkEtMTMyNzwvYT4gYW5kIDxhIGhyZWY9 Imh0dHBzOi8vaXNzdWVzLmFwYWNoZS5vcmcvamlyYS9icm93c2UvQ0FTU0FORFJBLTM2MjQiPmh0 dHBzOi8vaXNzdWVzLmFwYWNoZS5vcmcvamlyYS9icm93c2UvQ0FTU0FORFJBLTM2MjQ8L2E+Ozxi cj4NCg0KPC9kaXY+PGRpdj4yKSBEZWxldGlvbiBvZiBvdXRkYXRlZCBoaW50cyBkb2VzbiYjMzk7 dCB3b3JrIHdlbGwgZm9yIGxhcmdlIGhpbnQgdm9sdW1lcy48YnI+PC9kaXY+PGRpdj48YnI+PC9k aXY+PGRpdj48YnI+QW55IHN1Z2dlc3Rpb25zIG9uIGhvdyB0byBtYWtlIHRoZSBjbHVzdGVyIG1v cmUgdG9sZXJhbnQgdG8gZG93bnRpbWVzPzxicj48YnI+PC9kaXY+PGRpdj5JZiBJIHR1cm4gb2Zm IHRoZSBoaW50ZWQgaGFuZG9mZiBlbnRpcmVseSwgYW5kIG1hbnVhbGx5IHJ1biBhIHJlcGFpciBh ZnRlciBhIGRvd250aW1lLCB3aWxsIGl0IHJlc3RvcmUgYWxsIHRoZSBkYXRhIGNvcnJlY3RseT88 YnI+DQoNCjwvZGl2PjxkaXY+PGJyIGNsZWFyPSJhbGwiPjxkaXY+PGRpdj4tLTxicj5CZXN0IHJl Z2FyZHMsIFZsYWRpbWlyPGJyPjxicj48L2Rpdj4NCjwvZGl2PjwvZGl2PjwvZGl2PjwvZGl2Pjwv ZGl2PjwvZGl2Pg0K --bcaec51a74aca0d35004dd370c38--