Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 806519F55 for ; Fri, 2 Dec 2011 13:35:47 +0000 (UTC) Received: (qmail 23044 invoked by uid 500); 2 Dec 2011 13:35:45 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 23012 invoked by uid 500); 2 Dec 2011 13:35:44 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 22998 invoked by uid 99); 2 Dec 2011 13:35:44 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Dec 2011 13:35:44 +0000 X-ASF-Spam-Status: No, hits=2.8 required=5.0 tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,URI_HEX X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of sicoe.alexandru@googlemail.com designates 209.85.216.44 as permitted sender) Received: from [209.85.216.44] (HELO mail-qw0-f44.google.com) (209.85.216.44) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Dec 2011 13:35:40 +0000 Received: by qadb15 with SMTP id b15so422456qad.10 for ; Fri, 02 Dec 2011 05:35:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlemail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=5nR5f6R+ak7OCDLotnjVA0Y9+cwcSl1hw9kEP0JbDuE=; b=vS0WlHUlkCz5dB4taQUkxylPrDnrPpyi2Ishti7XDNqEfWBs+YDyeDgvQsABK7b42M lEDdGT253N8I32t2JObuj6Hrr9KXefMizPDSh7CWwJPQ5Y5P1gC4rz23sGPzq1Xgr5Si yqMgIbxORKVXG37RFg6MGGGZlBSe/CbNiGOJk= MIME-Version: 1.0 Received: by 10.224.188.16 with SMTP id cy16mr2959609qab.32.1322832919733; Fri, 02 Dec 2011 05:35:19 -0800 (PST) Received: by 10.229.20.84 with HTTP; Fri, 2 Dec 2011 05:35:19 -0800 (PST) In-Reply-To: References: <4ED7C052.10802@morningstar.com> Date: Fri, 2 Dec 2011 14:35:19 +0100 Message-ID: Subject: Re: Insufficient disk space to flush From: Alexandru Dan Sicoe To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=20cf30334fef4723ec04b31c0acf --20cf30334fef4723ec04b31c0acf Content-Type: text/plain; charset=ISO-8859-1 Ok, so my problem persisted. On the node that is filling up the harddisk, I have a 230 GB disk. Right after I restart the node I it deletes tmp files and reaches 55GB of data on disk. Then it start to quickly fill up the disk - I see gigs added fast - it's not real data because other nodes don't have this. While all this is happening I am seeing the node do a minor compaction of the main data CF but extremely slowly. Today I saw the error: ERROR 09:44:57,605 Fatal exception in thread Thread[CompactionExecutor:15,1,main] java.io.IOException: File too large at java.io.RandomAccessFile.writeBytes(Native Method) at java.io.RandomAccessFile.write(RandomAccessFile.java:466) at org.apache.cassandra.io.util.BufferedRandomAccessFile.flush(BufferedRandomAccessFile.java:168) at org.apache.cassandra.io.util.BufferedRandomAccessFile.reBuffer(BufferedRandomAccessFile.java:242) at org.apache.cassandra.io.util.BufferedRandomAccessFile.writeAtMost(BufferedRandomAccessFile.java:369) at org.apache.cassandra.io.util.BufferedRandomAccessFile.write(BufferedRandomAccessFile.java:348) at org.apache.cassandra.db.compaction.PrecompactedRow.write(PrecompactedRow.java:114) at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:132) at org.apache.cassandra.db.compaction.CompactionManager.doCompactionWithoutSizeEstimation(CompactionManager.java:576) at org.apache.cassandra.db.compaction.CompactionManager.doCompaction(CompactionManager.java:507) at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:142) at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:108) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) which means that it cannot finish that compaction because it hit the max file size. So I checked the file system and block size and I got ext3 and 1K which means that the max file size is 16GB. I didn't know what to do in this case so I just decommisioned the node. Is there a way to get around this max file limit? Is there some Cassandra configuration that helps avoid this? I'm asking here because I couldn't find anything in the documentation about that. I'm waiting for new machines to run Cassandra on....what file systems are people using? Cheers, Alex On Thu, Dec 1, 2011 at 10:08 PM, Jahangir Mohammed wrote: > Yes, mostly sounds like it. In our case failed repairs were causing > accumulation of the tmp files. > > Thanks, > Jahangir Mohammed. > > On Thu, Dec 1, 2011 at 2:43 PM, Alexandru Dan Sicoe < > sicoe.alexandru@googlemail.com> wrote: > >> Hi Jeremiah, >> My commitlog was indeed on another disk. I did what you said and yes the >> node restart brings back the disk size to the around 50 GB I was expecting. >> Still I do not understand how the node managed to get itself in the >> situation of having these tmp files? Could you clarify what these are, how >> they are produced and why? I've tried to find a clear definition but all I >> could come up with is hints that they are produced during compaction. I >> also found a thread that described a similar problem: >> >> http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Errors-During-Compaction-td5953493.html >> as described there it seems like compaction fails and tmp files don't get >> cleaned up until they fill the disk. Is this what happened in my case? >> Compactions did not finish properly because the disk utilization was more >> than half and then more and more files tmp started getting accumulated at >> each other attempt. The Cassandra log would indicate this because I get >> many of these: >> ERROR [CompactionExecutor:22850] 2011-12-01 04:12:15,200 >> CompactionManager.java (line 513) insufficie >> nt space to compact even the two smallest files, aborting >> >> before I started getting many of these: >> ERROR [FlushWriter:283] 2011-12-01 04:12:22,917 >> AbstractCassandraDaemon.java (line 139) Fatal exception in thread >> Thread[FlushWriter:283,5,main] java.lang.RuntimeException: >> java.lang.RuntimeException: Insufficient disk space to flush 42531 bytes >> >> I just want to clearly understand what happened. >> >> Thanks, >> Alex >> >> >> On Thu, Dec 1, 2011 at 6:58 PM, Jeremiah Jordan < >> jeremiah.jordan@morningstar.com> wrote: >> >>> If you are writing data with QUORUM or ALL you should be safe to >>> restart cassandra on that node. If the extra space is all from *tmp* files >>> from compaction they will get deleted at startup. You will then need to >>> run repair on that node to get back any data that was missed while it was >>> full. If your commit log was on a different device you may not even have >>> lost much. >>> >>> -Jeremiah >>> >>> >>> On 12/01/2011 04:16 AM, Alexandru Dan Sicoe wrote: >>> >>> Hello everyone, >>> 4 node Cassandra 0.8.5 cluster with RF =2. >>> One node started throwing exceptions in its log: >>> >>> ERROR 10:02:46,837 Fatal exception in thread >>> Thread[FlushWriter:1317,5,main] >>> java.lang.RuntimeException: java.lang.RuntimeException: Insufficient >>> disk space to flush 17296 bytes >>> at >>> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >>> at java.lang.Thread.run(Thread.java:619) >>> Caused by: java.lang.RuntimeException: Insufficient disk space to flush >>> 17296 bytes >>> at >>> org.apache.cassandra.db.ColumnFamilyStore.getFlushPath(ColumnFamilyStore.java:714) >>> at >>> org.apache.cassandra.db.ColumnFamilyStore.createFlushWriter(ColumnFamilyStore.java:2301) >>> at >>> org.apache.cassandra.db.Memtable.writeSortedContents(Memtable.java:246) >>> at org.apache.cassandra.db.Memtable.access$400(Memtable.java:49) >>> at >>> org.apache.cassandra.db.Memtable$3.runMayThrow(Memtable.java:270) >>> at >>> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) >>> ... 3 more >>> >>> Checked disk and obviously it's 100% full. >>> >>> How do I recover from this without loosing the data? I've got plenty of >>> space on the other nodes, so I thought of doing a decommission which I >>> understand reassigns ranges to the other nodes and replicates data to them. >>> After that's done I plan on manually deleting the data on the node and then >>> joining in the same cluster position with auto-bootstrap turned off so that >>> I won't get back the old data and I can continue getting new data with the >>> node. >>> >>> Note, I would like to have 4 nodes in because the other three barely >>> take the input load alone. These are just long running tests until I get >>> some better machines. >>> >>> On strange thing I found is that the data folder on the ndoe that filled >>> up the disk is 150 GB (as measured with du) while the data folder on all >>> other 3 nodes is 50 GB. At the same time, DataStax OpsCenter shows a size >>> of around 50GB for all 4 nodes. I though that the node was making a major >>> compaction at which time it filled up the disk....but even that doesn't >>> make sense because shouldn't a major compaction just be capable of doubling >>> the size, not triple-ing it? Doesn anyone know how to explain this behavior? >>> >>> Thanks, >>> Alex >>> >>> >> > -- Alexandru Dan Sicoe MEng, CERN Marie Curie ACEOLE Fellow --20cf30334fef4723ec04b31c0acf Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: base64 T2ssIHNvIG15IHByb2JsZW0gcGVyc2lzdGVkLiBPbiB0aGUgbm9kZSB0aGF0IGlzIGZpbGxpbmcg dXAgdGhlIGhhcmRkaXNrLCBJIGhhdmUgYSAyMzAgR0IgZGlzay4gUmlnaHQgYWZ0ZXIgSSByZXN0 YXJ0IHRoZSBub2RlIEkgaXQgZGVsZXRlcyB0bXAgZmlsZXMgYW5kIHJlYWNoZXMgNTVHQiBvZiBk YXRhIG9uIGRpc2suIFRoZW4gaXQgc3RhcnQgdG8gcXVpY2tseSBmaWxsIHVwIHRoZSBkaXNrIC0g SSBzZWUgZ2lncyBhZGRlZCBmYXN0IC0gaXQmIzM5O3Mgbm90IHJlYWwgZGF0YSBiZWNhdXNlIG90 aGVyIG5vZGVzIGRvbiYjMzk7dCBoYXZlIHRoaXMuIDxicj4KPGJyPldoaWxlIGFsbCB0aGlzIGlz IGhhcHBlbmluZyBJIGFtIHNlZWluZyB0aGUgbm9kZSBkbyBhIG1pbm9yIGNvbXBhY3Rpb24gb2Yg dGhlIG1haW4gZGF0YSBDRiBidXQgZXh0cmVtZWx5IHNsb3dseS4gVG9kYXkgSSBzYXcgdGhlIGVy cm9yOjxicj4KPGJyPkVSUk9SIDA5OjQ0OjU3LDYwNSBGYXRhbCBleGNlcHRpb24gaW4gdGhyZWFk IFRocmVhZFtDb21wYWN0aW9uRXhlY3V0b3I6MTUsMSxtYWluXaCgoKCgoKCgoKCgoKCgoKCgoKCg oKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCg oKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgIDxicj5qYXZh LmlvLklPRXhjZXB0aW9uOiBGaWxlIHRvbyBsYXJnZaCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCg oKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCg oKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCg oKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoCA8YnI+CgqgoKCgoKCgIGF0IGphdmEuaW8u UmFuZG9tQWNjZXNzRmlsZS53cml0ZUJ5dGVzKE5hdGl2ZSBNZXRob2QpoKCgoKCgoKCgoKCgoKCg oKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCg oKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCg oKCgoKCgoKCgoKCgoKCgoCA8YnI+oKCgoKCgoCBhdCBqYXZhLmlvLlJhbmRvbUFjY2Vzc0ZpbGUu d3JpdGUoUmFuZG9tQWNjZXNzRmlsZS5qYXZhOjQ2NimgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCg oKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCg oKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKAg PGJyPgoKoKCgoKCgoCBhdCBvcmcuYXBhY2hlLmNhc3NhbmRyYS5pby51dGlsLkJ1ZmZlcmVkUmFu ZG9tQWNjZXNzRmlsZS5mbHVzaChCdWZmZXJlZFJhbmRvbUFjY2Vzc0ZpbGUuamF2YToxNjgpoKCg oKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCg oKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKAgPGJyPqCgoKCgoKAgYXQg b3JnLmFwYWNoZS5jYXNzYW5kcmEuaW8udXRpbC5CdWZmZXJlZFJhbmRvbUFjY2Vzc0ZpbGUucmVC dWZmZXIoQnVmZmVyZWRSYW5kb21BY2Nlc3NGaWxlLmphdmE6MjQyKaCgoKCgoKCgoKCgoKCgoKCg oKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCg oKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgIDxicj4KCqCgoKCgoKAgYXQgb3JnLmFwYWNoZS5jYXNz YW5kcmEuaW8udXRpbC5CdWZmZXJlZFJhbmRvbUFjY2Vzc0ZpbGUud3JpdGVBdE1vc3QoQnVmZmVy ZWRSYW5kb21BY2Nlc3NGaWxlLmphdmE6MzY5KaCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCg oKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCg oKCgoKCgoKCgIDxicj6goKCgoKCgIGF0IG9yZy5hcGFjaGUuY2Fzc2FuZHJhLmlvLnV0aWwuQnVm ZmVyZWRSYW5kb21BY2Nlc3NGaWxlLndyaXRlKEJ1ZmZlcmVkUmFuZG9tQWNjZXNzRmlsZS5qYXZh OjM0OCmgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCg oKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoCA8YnI+Cgqg oKCgoKCgIGF0IG9yZy5hcGFjaGUuY2Fzc2FuZHJhLmRiLmNvbXBhY3Rpb24uUHJlY29tcGFjdGVk Um93LndyaXRlKFByZWNvbXBhY3RlZFJvdy5qYXZhOjExNCmgoKCgoKCgoKCgoKCgoKCgoKCgoKCg oKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCg oKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoCA8YnI+oKCgoKCgoCBhdCBvcmcuYXBh Y2hlLmNhc3NhbmRyYS5pby5zc3RhYmxlLlNTVGFibGVXcml0ZXIuYXBwZW5kKFNTVGFibGVXcml0 ZXIuamF2YToxMzIpoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCg oKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCg oKCgoKCgoKCgoKCgoKCgoKAgPGJyPgoKoKCgoKCgoCBhdCBvcmcuYXBhY2hlLmNhc3NhbmRyYS5k Yi5jb21wYWN0aW9uLkNvbXBhY3Rpb25NYW5hZ2VyLmRvQ29tcGFjdGlvbldpdGhvdXRTaXplRXN0 aW1hdGlvbihDb21wYWN0aW9uTWFuYWdlci5qYXZhOjU3NimgoKCgoKCgoKCgoKCgoKCgoKCgoKCg oKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCg oKAgPGJyPqCgoKCgoKAgYXQgb3JnLmFwYWNoZS5jYXNzYW5kcmEuZGIuY29tcGFjdGlvbi5Db21w YWN0aW9uTWFuYWdlci5kb0NvbXBhY3Rpb24oQ29tcGFjdGlvbk1hbmFnZXIuamF2YTo1MDcpoKCg oKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCg oKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgIDxicj4KCqCgoKCgoKAg YXQgb3JnLmFwYWNoZS5jYXNzYW5kcmEuZGIuY29tcGFjdGlvbi5Db21wYWN0aW9uTWFuYWdlciQx LmNhbGwoQ29tcGFjdGlvbk1hbmFnZXIuamF2YToxNDIpoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCg oKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCg oKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgIDxicj6goKCgoKCgIGF0IG9yZy5hcGFjaGUuY2Fz c2FuZHJhLmRiLmNvbXBhY3Rpb24uQ29tcGFjdGlvbk1hbmFnZXIkMS5jYWxsKENvbXBhY3Rpb25N YW5hZ2VyLmphdmE6MTA4KaCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCg oKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCg oKCgoKCgoKCgoCA8YnI+CgqgoKCgoKCgIGF0IGphdmEudXRpbC5jb25jdXJyZW50LkZ1dHVyZVRh c2skU3luYy5pbm5lclJ1bihGdXR1cmVUYXNrLmphdmE6MzAzKaCgoKCgoKCgoKCgoKCgoKCgoKCg oKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCg oKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoCA8YnI+ oKCgoKCgoCBhdCBqYXZhLnV0aWwuY29uY3VycmVudC5GdXR1cmVUYXNrLnJ1bihGdXR1cmVUYXNr LmphdmE6MTM4KaCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCg oKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCg oKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKAgPGJyPgoKoKCgoKCgoCBhdCBqYXZh LnV0aWwuY29uY3VycmVudC5UaHJlYWRQb29sRXhlY3V0b3IkV29ya2VyLnJ1blRhc2soVGhyZWFk UG9vbEV4ZWN1dG9yLmphdmE6ODg2KaCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCg oKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCg oKCgoKCgoKCgoKCgoKCgoKCgoKAgPGJyPqCgoKCgoKAgYXQgamF2YS51dGlsLmNvbmN1cnJlbnQu VGhyZWFkUG9vbEV4ZWN1dG9yJFdvcmtlci5ydW4oVGhyZWFkUG9vbEV4ZWN1dG9yLmphdmE6OTA4 KaCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCg oKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCg oKCgIDxicj4KCqCgoKCgoKAgYXQgamF2YS5sYW5nLlRocmVhZC5ydW4oVGhyZWFkLmphdmE6NjE5 KSA8YnI+PGJyPndoaWNoIG1lYW5zIHRoYXQgaXQgY2Fubm90IGZpbmlzaCB0aGF0IGNvbXBhY3Rp b24gYmVjYXVzZSBpdCBoaXQgdGhlIG1heCBmaWxlIHNpemUuIFNvIEkgY2hlY2tlZCB0aGUgZmls ZSBzeXN0ZW0gYW5kIGJsb2NrIHNpemUgYW5kIEkgZ290IGV4dDMgYW5kIDFLIHdoaWNoIG1lYW5z IHRoYXQgdGhlIG1heCBmaWxlIHNpemUgaXMgMTZHQi48YnI+Cjxicj5JIGRpZG4mIzM5O3Qga25v dyB3aGF0IHRvIGRvIGluIHRoaXMgY2FzZSBzbyBJIGp1c3QgZGVjb21taXNpb25lZCB0aGUgbm9k ZS48YnI+PGJyPklzIHRoZXJlIGEgd2F5IHRvIGdldCBhcm91bmQgdGhpcyBtYXggZmlsZSBsaW1p dD8gSXMgdGhlcmUgc29tZSBDYXNzYW5kcmEgY29uZmlndXJhdGlvbiB0aGF0IGhlbHBzIGF2b2lk IHRoaXM/IEkmIzM5O20gYXNraW5nIGhlcmUgYmVjYXVzZSBJIGNvdWxkbiYjMzk7dCBmaW5kIGFu eXRoaW5nIGluIHRoZSBkb2N1bWVudGF0aW9uIGFib3V0IHRoYXQuPGJyPgo8YnI+SSYjMzk7bSB3 YWl0aW5nIGZvciBuZXcgbWFjaGluZXMgdG8gcnVuIENhc3NhbmRyYSBvbi4uLi53aGF0IGZpbGUg c3lzdGVtcyBhcmUgcGVvcGxlIHVzaW5nPzxicj48YnI+Q2hlZXJzLDxicj5BbGV4PGJyPjxicj48 YnI+PGJyPjxkaXYgY2xhc3M9ImdtYWlsX3F1b3RlIj5PbiBUaHUsIERlYyAxLCAyMDExIGF0IDEw OjA4IFBNLCBKYWhhbmdpciBNb2hhbW1lZCA8c3BhbiBkaXI9Imx0ciI+Jmx0OzxhIGhyZWY9Im1h aWx0bzptZC5qYWhhbmdpcjI3QGdtYWlsLmNvbSIgdGFyZ2V0PSJfYmxhbmsiPm1kLmphaGFuZ2ly MjdAZ21haWwuY29tPC9hPiZndDs8L3NwYW4+IHdyb3RlOjxicj4KCjxibG9ja3F1b3RlIGNsYXNz PSJnbWFpbF9xdW90ZSIgc3R5bGU9Im1hcmdpbjogMHB0IDBwdCAwcHQgMC44ZXg7IGJvcmRlci1s ZWZ0OiAxcHggc29saWQgcmdiKDIwNCwgMjA0LCAyMDQpOyBwYWRkaW5nLWxlZnQ6IDFleDsiPgo8 ZGl2PlllcywgbW9zdGx5IHNvdW5kcyBsaWtlIGl0LiBJbiBvdXIgY2FzZSBmYWlsZWQgcmVwYWly cyB3ZXJlIGNhdXNpbmcgYWNjdW11bGF0aW9uIG9mIHRoZSB0bXAgZmlsZXMuPC9kaXY+PGRpdj48 YnI+PC9kaXY+PGRpdj5UaGFua3MsPC9kaXY+PHNwYW4+PGZvbnQgY29sb3I9IiM4ODg4ODgiPjxk aXY+SmFoYW5naXIgTW9oYW1tZWQuPC9kaXY+PC9mb250Pjwvc3Bhbj48ZGl2Pgo8ZGl2Pjxicj48 ZGl2IGNsYXNzPSJnbWFpbF9xdW90ZSI+T24gVGh1LCBEZWMgMSwgMjAxMSBhdCAyOjQzIFBNLCBB bGV4YW5kcnUgRGFuIFNpY29lIDxzcGFuIGRpcj0ibHRyIj4mbHQ7PGEgaHJlZj0ibWFpbHRvOnNp Y29lLmFsZXhhbmRydUBnb29nbGVtYWlsLmNvbSIgdGFyZ2V0PSJfYmxhbmsiPnNpY29lLmFsZXhh bmRydUBnb29nbGVtYWlsLmNvbTwvYT4mZ3Q7PC9zcGFuPiB3cm90ZTo8YnI+CgoKCjxibG9ja3F1 b3RlIGNsYXNzPSJnbWFpbF9xdW90ZSIgc3R5bGU9Im1hcmdpbjogMHB0IDBwdCAwcHQgMC44ZXg7 IGJvcmRlci1sZWZ0OiAxcHggc29saWQgcmdiKDIwNCwgMjA0LCAyMDQpOyBwYWRkaW5nLWxlZnQ6 IDFleDsiPkhpIDxzcGFuPkplcmVtaWFoLDxicj6gTXkgY29tbWl0bG9nIHdhcyBpbmRlZWQgb24g YW5vdGhlciBkaXNrLiBJIGRpZCB3aGF0IHlvdSBzYWlkIGFuZCB5ZXMgdGhlIG5vZGUgcmVzdGFy dCBicmluZ3MgYmFjayB0aGUgZGlzayBzaXplIHRvIHRoZSBhcm91bmQgNTAgR0IgSSB3YXMgZXhw ZWN0aW5nLiBTdGlsbCBJIGRvIG5vdCB1bmRlcnN0YW5kIGhvdyB0aGUgbm9kZSBtYW5hZ2VkIHRv IGdldCBpdHNlbGYgaW4gdGhlIHNpdHVhdGlvbiBvZiBoYXZpbmcgdGhlc2UgdG1wIGZpbGVzPyBD b3VsZCB5b3UgY2xhcmlmeSB3aGF0IHRoZXNlIGFyZSwgaG93IHRoZXkgYXJlIHByb2R1Y2VkIGFu ZCB3aHk/IEkmIzM5O3ZlIHRyaWVkIHRvIGZpbmQgYSBjbGVhciBkZWZpbml0aW9uIGJ1dCBhbGwg SSBjb3VsZCBjb21lIHVwIHdpdGggaXMgaGludHMgdGhhdCB0aGV5IGFyZSBwcm9kdWNlZCBkdXJp bmcgY29tcGFjdGlvbi4gSSBhbHNvIGZvdW5kIGEgdGhyZWFkIHRoYXQgZGVzY3JpYmVkIGEgc2lt aWxhciBwcm9ibGVtOjxicj4KCgoKCjxhIGhyZWY9Imh0dHA6Ly9jYXNzYW5kcmEtdXNlci1pbmN1 YmF0b3ItYXBhY2hlLW9yZy4zMDY1MTQ2Lm4yLm5hYmJsZS5jb20vRXJyb3JzLUR1cmluZy1Db21w YWN0aW9uLXRkNTk1MzQ5My5odG1sIiB0YXJnZXQ9Il9ibGFuayI+aHR0cDovL2Nhc3NhbmRyYS11 c2VyLWluY3ViYXRvci1hcGFjaGUtb3JnLjMwNjUxNDYubjIubmFiYmxlLmNvbS9FcnJvcnMtRHVy aW5nLUNvbXBhY3Rpb24tdGQ1OTUzNDkzLmh0bWw8L2E+PGJyPgoKCgoKYXMgZGVzY3JpYmVkIHRo ZXJlIGl0IHNlZW1zIGxpa2UgY29tcGFjdGlvbiBmYWlscyBhbmQgdG1wIGZpbGVzIGRvbiYjMzk7 dCBnZXQgY2xlYW5lZCB1cCB1bnRpbCB0aGV5IGZpbGwgdGhlIGRpc2suIElzIHRoaXMgd2hhdCBo YXBwZW5lZCBpbiBteSBjYXNlPyBDb21wYWN0aW9ucyBkaWQgbm90IGZpbmlzaCBwcm9wZXJseSBi ZWNhdXNlIHRoZSBkaXNrIHV0aWxpemF0aW9uIHdhcyBtb3JlIHRoYW4gaGFsZiBhbmQgdGhlbiBt b3JlIGFuZCBtb3JlIGZpbGVzIHRtcCBzdGFydGVkIGdldHRpbmcgYWNjdW11bGF0ZWQgYXQgZWFj aCBvdGhlciBhdHRlbXB0LiBUaGUgQ2Fzc2FuZHJhIGxvZyB3b3VsZCBpbmRpY2F0ZSB0aGlzIGJl Y2F1c2UgSSBnZXQgbWFueSBvZiB0aGVzZTo8YnI+CgoKCgpFUlJPUiBbQ29tcGFjdGlvbkV4ZWN1 dG9yOjIyODUwXSAyMDExLTEyLTAxIDA0OjEyOjE1LDIwMCBDb21wYWN0aW9uTWFuYWdlci5qYXZh IChsaW5lIDUxMykgaW5zdWZmaWNpZTxicj5udCBzcGFjZSB0byBjb21wYWN0IGV2ZW4gdGhlIHR3 byBzbWFsbGVzdCBmaWxlcywgYWJvcnRpbmc8YnI+PGJyPmJlZm9yZSBJIHN0YXJ0ZWQgZ2V0dGlu ZyBtYW55IG9mIHRoZXNlOjxicj5FUlJPUiBbRmx1c2hXcml0ZXI6MjgzXSAyMDExLTEyLTAxIDA0 OjEyOjIyLDkxNyBBYnN0cmFjdENhc3NhbmRyYURhZW1vbi5qYXZhIChsaW5lIDEzOSkgRmF0YWwg ZXhjZXB0aW9uIGluIHRocmVhZCBUaHJlYWRbRmx1c2hXcml0ZXI6MjgzLDUsbWFpbl0gPC9zcGFu PgoKCgkKCQoJCglqYXZhLmxhbmcuUnVudGltZUV4Y2VwdGlvbjogamF2YS5sYW5nLlJ1bnRpbWVF eGNlcHRpb246IEluc3VmZmljaWVudCBkaXNrIHNwYWNlIHRvIGZsdXNoIDQyNTMxIGJ5dGVzPGJy PgoKPGJyPjxzcGFuPkkganVzdCB3YW50IHRvIGNsZWFybHkgdW5kZXJzdGFuZCB3aGF0IGhhcHBl bmVkLjxicj48YnI+VGhhbmtzLDxicj5BbGV4PGJyPjxicj48L3NwYW4+PGRpdj48ZGl2Pjxicj48 ZGl2IGNsYXNzPSJnbWFpbF9xdW90ZSI+T24gVGh1LCBEZWMgMSwgMjAxMSBhdCA2OjU4IFBNLCBK ZXJlbWlhaCBKb3JkYW4gPHNwYW4gZGlyPSJsdHIiPiZsdDs8YSBocmVmPSJtYWlsdG86amVyZW1p YWguam9yZGFuQG1vcm5pbmdzdGFyLmNvbSIgdGFyZ2V0PSJfYmxhbmsiPmplcmVtaWFoLmpvcmRh bkBtb3JuaW5nc3Rhci5jb208L2E+Jmd0Ozwvc3Bhbj4gd3JvdGU6PGJyPgoKCgoKPGJsb2NrcXVv dGUgY2xhc3M9ImdtYWlsX3F1b3RlIiBzdHlsZT0ibWFyZ2luOiAwcHQgMHB0IDBwdCAwLjhleDsg Ym9yZGVyLWxlZnQ6IDFweCBzb2xpZCByZ2IoMjA0LCAyMDQsIDIwNCk7IHBhZGRpbmctbGVmdDog MWV4OyI+CiAgCiAgICAKICAKICA8ZGl2IGJnY29sb3I9IiNGRkZGRkYiIHRleHQ9IiMwMDAwMDAi PgogICAgSWYgeW91IGFyZSB3cml0aW5nIGRhdGEgd2l0aCBRVU9SVU0gb3IgQUxMIHlvdSBzaG91 bGQgYmUgc2FmZSB0bwogICAgcmVzdGFydCBjYXNzYW5kcmEgb24gdGhhdCBub2RlLqAgSWYgdGhl IGV4dHJhIHNwYWNlIGlzIGFsbCBmcm9tCiAgICAqdG1wKiBmaWxlcyBmcm9tIGNvbXBhY3Rpb24g dGhleSB3aWxsIGdldCBkZWxldGVkIGF0IHN0YXJ0dXAuoCBZb3UKICAgIHdpbGwgdGhlbiBuZWVk IHRvIHJ1biByZXBhaXIgb24gdGhhdCBub2RlIHRvIGdldCBiYWNrIGFueSBkYXRhIHRoYXQKICAg IHdhcyBtaXNzZWQgd2hpbGUgaXQgd2FzIGZ1bGwuoCBJZiB5b3VyIGNvbW1pdCBsb2cgd2FzIG9u IGEgZGlmZmVyZW50CiAgICBkZXZpY2UgeW91IG1heSBub3QgZXZlbiBoYXZlIGxvc3QgbXVjaC48 c3Bhbj48Zm9udCBjb2xvcj0iIzg4ODg4OCI+PGJyPgogICAgPGJyPgogICAgLUplcmVtaWFoPC9m b250Pjwvc3Bhbj48ZGl2PjxkaXY+PGJyPgogICAgPGJyPgogICAgT24gMTIvMDEvMjAxMSAwNDox NiBBTSwgQWxleGFuZHJ1IERhbiBTaWNvZSB3cm90ZToKICAgIDxibG9ja3F1b3RlIHR5cGU9ImNp dGUiPkhlbGxvIGV2ZXJ5b25lLDxicj4KICAgICAgoDQgbm9kZSBDYXNzYW5kcmEgMC44LjUgY2x1 c3RlciB3aXRoIFJGID0yLjxicj4KICAgICAgoE9uZSBub2RlIHN0YXJ0ZWQgdGhyb3dpbmcgZXhj ZXB0aW9ucyBpbiBpdHMgbG9nOiA8YnI+CiAgICAgIDxicj4KICAgICAgRVJST1IgMTA6MDI6NDYs ODM3IEZhdGFsIGV4Y2VwdGlvbiBpbiB0aHJlYWQKICAgICAgVGhyZWFkW0ZsdXNoV3JpdGVyOjEz MTcsNSxtYWluXTxicj4KICAgICAgamF2YS5sYW5nLlJ1bnRpbWVFeGNlcHRpb246IGphdmEubGFu Zy5SdW50aW1lRXhjZXB0aW9uOgogICAgICBJbnN1ZmZpY2llbnQgZGlzayBzcGFjZSB0byBmbHVz aCAxNzI5NiBieXRlczxicj4KICAgICAgoKCgoKCgoCBhdAogICAgICBvcmcuYXBhY2hlLmNhc3Nh bmRyYS51dGlscy5XcmFwcGVkUnVubmFibGUucnVuKFdyYXBwZWRSdW5uYWJsZS5qYXZhOjM0KTxi cj4KICAgICAgoKCgoKCgoCBhdApqYXZhLnV0aWwuY29uY3VycmVudC5UaHJlYWRQb29sRXhlY3V0 b3IkV29ya2VyLnJ1blRhc2soVGhyZWFkUG9vbEV4ZWN1dG9yLmphdmE6ODg2KTxicj4KICAgICAg oKCgoKCgoCBhdApqYXZhLnV0aWwuY29uY3VycmVudC5UaHJlYWRQb29sRXhlY3V0b3IkV29ya2Vy LnJ1bihUaHJlYWRQb29sRXhlY3V0b3IuamF2YTo5MDgpPGJyPgogICAgICCgoKCgoKCgIGF0IGph dmEubGFuZy5UaHJlYWQucnVuKFRocmVhZC5qYXZhOjYxOSk8YnI+CiAgICAgIENhdXNlZCBieTog amF2YS5sYW5nLlJ1bnRpbWVFeGNlcHRpb246IEluc3VmZmljaWVudCBkaXNrIHNwYWNlIHRvCiAg ICAgIGZsdXNoIDE3Mjk2IGJ5dGVzPGJyPgogICAgICCgoKCgoKCgIGF0Cm9yZy5hcGFjaGUuY2Fz c2FuZHJhLmRiLkNvbHVtbkZhbWlseVN0b3JlLmdldEZsdXNoUGF0aChDb2x1bW5GYW1pbHlTdG9y ZS5qYXZhOjcxNCk8YnI+CiAgICAgIKCgoKCgoKAgYXQKb3JnLmFwYWNoZS5jYXNzYW5kcmEuZGIu Q29sdW1uRmFtaWx5U3RvcmUuY3JlYXRlRmx1c2hXcml0ZXIoQ29sdW1uRmFtaWx5U3RvcmUuamF2 YToyMzAxKTxicj4KICAgICAgoKCgoKCgoCBhdAogICAgICBvcmcuYXBhY2hlLmNhc3NhbmRyYS5k Yi5NZW10YWJsZS53cml0ZVNvcnRlZENvbnRlbnRzKE1lbXRhYmxlLmphdmE6MjQ2KTxicj4KICAg ICAgoKCgoKCgoCBhdAogICAgICBvcmcuYXBhY2hlLmNhc3NhbmRyYS5kYi5NZW10YWJsZS5hY2Nl c3MkNDAwKE1lbXRhYmxlLmphdmE6NDkpPGJyPgogICAgICCgoKCgoKCgIGF0CiAgICAgIG9yZy5h cGFjaGUuY2Fzc2FuZHJhLmRiLk1lbXRhYmxlJDMucnVuTWF5VGhyb3coTWVtdGFibGUuamF2YToy NzApPGJyPgogICAgICCgoKCgoKCgIGF0CiAgICAgIG9yZy5hcGFjaGUuY2Fzc2FuZHJhLnV0aWxz LldyYXBwZWRSdW5uYWJsZS5ydW4oV3JhcHBlZFJ1bm5hYmxlLmphdmE6MzApPGJyPgogICAgICCg oKCgoKCgIC4uLiAzIG1vcmU8YnI+CiAgICAgIDxicj4KICAgICAgQ2hlY2tlZCBkaXNrIGFuZCBv YnZpb3VzbHkgaXQmIzM5O3MgMTAwJSBmdWxsLjxicj4KICAgICAgPGJyPgogICAgICBIb3cgZG8g SSByZWNvdmVyIGZyb20gdGhpcyB3aXRob3V0IGxvb3NpbmcgdGhlIGRhdGE/IEkmIzM5O3ZlIGdv dAogICAgICBwbGVudHkgb2Ygc3BhY2Ugb24gdGhlIG90aGVyIG5vZGVzLCBzbyBJIHRob3VnaHQg b2YgZG9pbmcgYQogICAgICBkZWNvbW1pc3Npb24gd2hpY2ggSSB1bmRlcnN0YW5kIHJlYXNzaWdu cyByYW5nZXMgdG8gdGhlIG90aGVyCiAgICAgIG5vZGVzIGFuZCByZXBsaWNhdGVzIGRhdGEgdG8g dGhlbS4gQWZ0ZXIgdGhhdCYjMzk7cyBkb25lIEkgcGxhbiBvbgogICAgICBtYW51YWxseSBkZWxl dGluZyB0aGUgZGF0YSBvbiB0aGUgbm9kZSBhbmQgdGhlbiBqb2luaW5nIGluIHRoZQogICAgICBz YW1lIGNsdXN0ZXIgcG9zaXRpb24gd2l0aCBhdXRvLWJvb3RzdHJhcCB0dXJuZWQgb2ZmIHNvIHRo YXQgSQogICAgICB3b24mIzM5O3QgZ2V0IGJhY2sgdGhlIG9sZCBkYXRhIGFuZCBJIGNhbiBjb250 aW51ZSBnZXR0aW5nIG5ldyBkYXRhCiAgICAgIHdpdGggdGhlIG5vZGUuPGJyPgogICAgICA8YnI+ CiAgICAgIE5vdGUsIEkgd291bGQgbGlrZSB0byBoYXZlIDQgbm9kZXMgaW4gYmVjYXVzZSB0aGUg b3RoZXIgdGhyZWUKICAgICAgYmFyZWx5IHRha2UgdGhlIGlucHV0IGxvYWQgYWxvbmUuIFRoZXNl IGFyZSBqdXN0IGxvbmcgcnVubmluZwogICAgICB0ZXN0cyB1bnRpbCBJIGdldCBzb21lIGJldHRl ciBtYWNoaW5lcy48YnI+CiAgICAgIDxiciBjbGVhcj0iYWxsIj4KICAgICAgT24gc3RyYW5nZSB0 aGluZyBJIGZvdW5kIGlzIHRoYXQgdGhlIGRhdGEgZm9sZGVyIG9uIHRoZSBuZG9lIHRoYXQKICAg ICAgZmlsbGVkIHVwIHRoZSBkaXNrIGlzIDE1MCBHQiAoYXMgbWVhc3VyZWQgd2l0aCBkdSkgd2hp bGUgdGhlIGRhdGEKICAgICAgZm9sZGVyIG9uIGFsbCBvdGhlciAzIG5vZGVzIGlzIDUwIEdCLiBB dCB0aGUgc2FtZSB0aW1lLCBEYXRhU3RheAogICAgICBPcHNDZW50ZXIgc2hvd3MgYSBzaXplIG9m IGFyb3VuZCA1MEdCIGZvciBhbGwgNCBub2Rlcy4gSSB0aG91Z2gKICAgICAgdGhhdCB0aGUgbm9k ZSB3YXMgbWFraW5nIGEgbWFqb3IgY29tcGFjdGlvbiBhdCB3aGljaCB0aW1lIGl0CiAgICAgIGZp bGxlZCB1cCB0aGUgZGlzay4uLi5idXQgZXZlbiB0aGF0IGRvZXNuJiMzOTt0IG1ha2Ugc2Vuc2Ug YmVjYXVzZQogICAgICBzaG91bGRuJiMzOTt0IGEgbWFqb3IgY29tcGFjdGlvbiBqdXN0IGJlIGNh cGFibGUgb2YgZG91YmxpbmcgdGhlIHNpemUsCiAgICAgIG5vdCB0cmlwbGUtaW5nIGl0PyBEb2Vz biBhbnlvbmUga25vdyBob3cgdG8gZXhwbGFpbiB0aGlzIGJlaGF2aW9yPzxicj4KICAgICAgPGJy PgogICAgICBUaGFua3MsPGJyPgogICAgICBBbGV4PGZvbnQgc2l6ZT0iMiI+PHNwYW4gc3R5bGU9 ImZvbnQtc2l6ZTogMTBwdDsiPjxicj4KICAgICAgICA8L3NwYW4+PC9mb250Pjxicj4KICAgIDwv YmxvY2txdW90ZT4KICA8L2Rpdj48L2Rpdj48L2Rpdj4KCjwvYmxvY2txdW90ZT48L2Rpdj48Zm9u dCBzaXplPSIyIj48c3BhbiBzdHlsZT0iZm9udC1zaXplOiAxMHB0OyI+PC9zcGFuPjwvZm9udD48 YnI+CjwvZGl2PjwvZGl2PjwvYmxvY2txdW90ZT48L2Rpdj48YnI+CjwvZGl2PjwvZGl2PjwvYmxv Y2txdW90ZT48L2Rpdj48YnI+PGJyIGNsZWFyPSJhbGwiPjxicj4tLSA8YnI+PGZvbnQgc2l6ZT0i MiI+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZTogMTBwdDsiPkFsZXhhbmRydSBEYW4gU2ljb2U8YnI+ TUVuZywgQ0VSTiBNYXJpZSBDdXJpZSBBQ0VPTEUgRmVsbG93PGJyPjwvc3Bhbj48L2ZvbnQ+PGJy Pgo= --20cf30334fef4723ec04b31c0acf--