Return-Path: Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: (qmail 91016 invoked from network); 18 Oct 2010 20:23:19 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 18 Oct 2010 20:23:19 -0000 Received: (qmail 40473 invoked by uid 500); 18 Oct 2010 20:23:17 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 40452 invoked by uid 500); 18 Oct 2010 20:23:17 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 40444 invoked by uid 99); 18 Oct 2010 20:23:17 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 18 Oct 2010 20:23:17 +0000 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of jbellis@gmail.com designates 74.125.82.172 as permitted sender) Received: from [74.125.82.172] (HELO mail-wy0-f172.google.com) (74.125.82.172) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 18 Oct 2010 20:23:09 +0000 Received: by wyb29 with SMTP id 29so1672115wyb.31 for ; Mon, 18 Oct 2010 13:22:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:received:in-reply-to :references:date:message-id:subject:from:to:content-type; bh=Ew7Nww6OTEewIbdRjzQEHooXhJ84apNeYviVgiLg1cM=; b=K6TfBdMa5VLEMApuDnV923LSo+wUIm4TKp0B01ohJFl8LXeVi1V6a9kTm+VL6XXFYZ 6794QZ4p8XU2mXBWIQNYt7id/dbe3s5BRQi6WAN9/XgHlsMSTUGQmKBtkGpzRIgxI0hn ACc6Mb3z9ayHB9ZRAWzKU/N7kUPAIszy2gmQw= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; b=v40mPwYLW999btgrJd5E8K7TCfFTYu7Nth32jrFpUMMFD1O4xFN1R7utdnPmt0zNi3 Izg5sMIYLCyRVJcLKNglKaaOLawea+TA8Wl2Eg+mLzCCa2xS7CruaexiRiMOAUZvOTrE XgRq/Euc5BD2LVQ52N/keg4r2irTEJbZPlLt4= MIME-Version: 1.0 Received: by 10.216.11.205 with SMTP id 55mr5172213wex.51.1287433368893; Mon, 18 Oct 2010 13:22:48 -0700 (PDT) Received: by 10.216.52.16 with HTTP; Mon, 18 Oct 2010 13:22:29 -0700 (PDT) In-Reply-To: References: Date: Mon, 18 Oct 2010 15:22:29 -0500 Message-ID: Subject: Re: Read Latency From: Jonathan Ellis To: user Content-Type: multipart/mixed; boundary=0016364d2cbd9ff9c00492e9f097 X-Virus-Checked: Checked by ClamAV on apache.org --0016364d2cbd9ff9c00492e9f097 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable v3 attached to fix assertion failure. Those last seconds after "sending to client" aren't fixable except perhaps by using a faster client language. That is, the client is probably doing read-and-deserialize as one step rather than 2, so being slow in the deserialize part means the server is sitting there waiting to send the rest of a large response. But while that's happening it's using negligible resources on the server side so it's not the end of the world for a multithreaded app. On Mon, Oct 18, 2010 at 2:37 PM, Wayne wrote: > It is much faster again, but the last 3 seconds still linger. There a lot > more logout complete messages getting logged all of the time, but the one > below matches in time to the time the client registers it takes to get th= e > data back. It also produces errors sometimes...the error example is liste= d > below. > > DEBUG [pool-1-thread-64] 2010-10-18 19:25:28,866 CassandraServer.java (li= ne > 219) get_slice > DEBUG [pool-1-thread-64] 2010-10-18 19:25:28,867 StorageProxy.java (line > 471) strongread reading data for SliceFromReadCommand(table=3D'table', > key=3D'key1', column_parent=3D'QueryPath(columnFamilyName=3D'fact', > superColumnName=3D'null', columnName=3D'null')', start=3D'503a', finish= =3D'503a7c', > reversed=3Dfalse, count=3D10000000) from 698@/x.x.x.6 > DEBUG [pool-1-thread-64] 2010-10-18 19:25:28,867 StorageProxy.java (line > 471) strongread reading digest for SliceFromReadCommand(table=3D'table', > key=3D'key1', column_parent=3D'QueryPath(columnFamilyName=3D'fact', > superColumnName=3D'null', columnName=3D'null')', start=3D'503a', finish= =3D'503a7c', > reversed=3Dfalse, count=3D10000000) from 699@/x.x.x.7 > DEBUG [pool-1-thread-64] 2010-10-18 19:25:28,867 StorageProxy.java (line > 471) strongread reading digest for SliceFromReadCommand(table=3D'table', > key=3D'key1', column_parent=3D'QueryPath(columnFamilyName=3D'fact', > superColumnName=3D'null', columnName=3D'null')', start=3D'503a', finish= =3D'503a7c', > reversed=3Dfalse, count=3D10000000) from 699@/x.x.x.8 > DEBUG [pool-1-thread-63] 2010-10-18 19:25:29,410 CassandraServer.java (li= ne > 667) logout complete > DEBUG [RESPONSE-STAGE:5] 2010-10-18 19:25:30,864 ResponseVerbHandler.java > (line 42) Processing response on a callback from > 0952ED39-07CE-7971-8F06-0D611FCB5F34@/x.x.x.6 > DEBUG [RESPONSE-STAGE:6] 2010-10-18 19:25:31,449 ResponseVerbHandler.java > (line 42) Processing response on a callback from > 0952ED39-07CE-7971-8F06-0D611FCB5F34@/x.x.x.8 > DEBUG [pool-1-thread-64] 2010-10-18 19:25:31,449 ReadResponseResolver.jav= a > (line 71) resolving 2 responses > DEBUG [pool-1-thread-64] 2010-10-18 19:25:31,608 ReadResponseResolver.jav= a > (line 116) digests verified > DEBUG [pool-1-thread-64] 2010-10-18 19:25:31,609 ReadResponseResolver.jav= a > (line 133) resolve: 160 ms. > DEBUG [pool-1-thread-64] 2010-10-18 19:25:31,609 StorageProxy.java (line > 494) quorumResponseHandler: 2742 ms. > DEBUG [pool-1-thread-64] 2010-10-18 19:25:31,667 CassandraServer.java (li= ne > 191) Slice converted to thrift; sending to client > DEBUG [pool-1-thread-63] 2010-10-18 19:25:32,044 CassandraServer.java (li= ne > 667) logout complete > DEBUG [pool-1-thread-63] 2010-10-18 19:25:32,238 CassandraServer.java (li= ne > 667) logout complete > DEBUG [pool-1-thread-63] 2010-10-18 19:25:32,377 CassandraServer.java (li= ne > 667) logout complete > DEBUG [pool-1-thread-63] 2010-10-18 19:25:33,890 CassandraServer.java (li= ne > 667) logout complete > DEBUG [pool-1-thread-63] 2010-10-18 19:25:34,401 CassandraServer.java (li= ne > 667) logout complete > DEBUG [pool-1-thread-63] 2010-10-18 19:25:34,538 CassandraServer.java (li= ne > 667) logout complete > DEBUG [pool-1-thread-64] 2010-10-18 19:25:34,925 CassandraServer.java (li= ne > 667) logout complete > > > > Error: > > DEBUG [RESPONSE-STAGE:4] 2010-10-18 19:22:48,355 ResponseVerbHandler.java > (line 42) Processing response on a callback from > DDF1D615-BF1E-2CD1-34C6-96FA47D63259@/x.x.x.8 > ERROR [RESPONSE-STAGE:4] 2010-10-18 19:22:48,356 CassandraDaemon.java (li= ne > 87) Uncaught exception in thread Thread[RESPONSE-STAGE:4,5,main] > java.lang.AssertionError > =A0=A0=A0 at > org.apache.cassandra.service.ReadResponseResolver.getResult(ReadResponseR= esolver.java:218) > =A0=A0=A0 at > org.apache.cassandra.service.ReadResponseResolver.isDataPresent(ReadRespo= nseResolver.java:209) > =A0=A0=A0 at > org.apache.cassandra.service.QuorumResponseHandler.response(QuorumRespons= eHandler.java:93) > =A0=A0=A0 at > org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.j= ava:44) > =A0=A0=A0 at > org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java= :49) > =A0=A0=A0 at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor= .java:886) > =A0=A0=A0 at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.jav= a:908) > =A0=A0=A0 at java.lang.Thread.run(Thread.java:619) > DEBUG [pool-1-thread-33] 2010-10-18 19:22:48,380 ReadResponseResolver.jav= a > (line 71) resolving 2 responses > DEBUG [pool-1-thread-33] 2010-10-18 19:22:48,542 ReadResponseResolver.jav= a > (line 116) digests verified > DEBUG [pool-1-thread-33] 2010-10-18 19:22:48,542 ReadResponseResolver.jav= a > (line 133) resolve: 162 ms. > DEBUG [pool-1-thread-33] 2010-10-18 19:22:48,542 StorageProxy.java (line > 494) quorumResponseHandler: 2688 ms. > > > On Sat, Oct 16, 2010 at 9:18 PM, Jonathan Ellis wrote= : >> >> Thanks. =A0Take 2 attached. >> >> On Sat, Oct 16, 2010 at 3:37 PM, Wayne wrote: >> > ERROR [pool-1-thread-64] 2010-10-16 20:27:55,396 Cassandra.java (line >> > 1280) >> > Internal error processing get_slice >> > java.lang.AssertionError >> > =A0=A0=A0 at >> > >> > org.apache.cassandra.service.ReadResponseResolver.resolve(ReadResponse= Resolver.java:88) >> > =A0=A0=A0 at >> > >> > org.apache.cassandra.service.ReadResponseResolver.resolve(ReadResponse= Resolver.java:44) >> > =A0=A0=A0 at >> > >> > org.apache.cassandra.service.QuorumResponseHandler.get(QuorumResponseH= andler.java:86) >> > =A0=A0=A0 at >> > >> > org.apache.cassandra.service.StorageProxy.strongRead(StorageProxy.java= :489) >> > =A0=A0=A0 at >> > >> > org.apache.cassandra.service.StorageProxy.readProtocol(StorageProxy.ja= va:353) >> > =A0=A0=A0 at >> > >> > org.apache.cassandra.thrift.CassandraServer.readColumnFamily(Cassandra= Server.java:97) >> > =A0=A0=A0 at >> > >> > org.apache.cassandra.thrift.CassandraServer.getSlice(CassandraServer.j= ava:180) >> > =A0=A0=A0 at >> > >> > org.apache.cassandra.thrift.CassandraServer.multigetSliceInternal(Cass= andraServer.java:262) >> > =A0=A0=A0 at >> > >> > org.apache.cassandra.thrift.CassandraServer.get_slice(CassandraServer.= java:223) >> > =A0=A0=A0 at >> > >> > org.apache.cassandra.thrift.Cassandra$Processor$get_slice.process(Cass= andra.java:1272) >> > =A0=A0=A0 at >> > >> > org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java= :1166) >> > =A0=A0=A0 at >> > >> > org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(= CustomTThreadPoolServer.java:167) >> > =A0=A0=A0 at >> > >> > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecu= tor.java:886) >> > =A0=A0=A0 at >> > >> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.= java:908) >> > =A0=A0=A0 at java.lang.Thread.run(Thread.java:619) >> > >> > >> > On Sat, Oct 16, 2010 at 9:52 AM, Jonathan Ellis >> > wrote: >> >> >> >> Stack trace from cassandra log? >> >> >> >> On Sat, Oct 16, 2010 at 6:50 AM, Wayne wrote: >> >> > While doing a read I get a TApplicationException: Internal Error >> >> > processing >> >> > get_slice. >> >> > >> >> > On Fri, Oct 15, 2010 at 5:49 PM, Jonathan Ellis >> >> > wrote: >> >> >> >> >> >> On Fri, Oct 15, 2010 at 2:21 PM, Wayne wrote: >> >> >> > The optimization definitely shaved off some time. Now it is >> >> >> > running >> >> >> > about 3x >> >> >> > CFSTATS reported time. Below are the logs. >> >> >> > >> >> >> > There is a ~300ms time frame after the last ResponseVerbHandler >> >> >> > prior >> >> >> > to >> >> >> > the >> >> >> > resolver starting. Based on a quorum read the response resolver >> >> >> > should >> >> >> > kick >> >> >> > after 2 reads come in correct? That would mean it waited 500ms >> >> >> > before >> >> >> > starting. Where is this time going? >> >> >> >> >> >> It's going to deserialize the replies to see if it has both the da= ta >> >> >> and enough digests to call resolve(). =A0Then resolve deserializes >> >> >> them >> >> >> a 2nd time, so there's an easy win there caching the first >> >> >> deserialize, in the patch attached. =A0(Applies on top of the prev= ious >> >> >> one, which has been committed to the 0.6 svn branch at >> >> >> https://svn.apache.org/repos/asf/cassandra/branches/cassandra-0.6.= ) >> >> >> >> >> >> > There is still the 3+ second delay between the last 2 entries. I= s >> >> >> > this >> >> >> > the >> >> >> > thrift server? >> >> >> >> >> >> It's converting the reply from Cassandra's internal representation >> >> >> to >> >> >> thrift, and sending it to the client. =A0I suspect most of the tim= e is >> >> >> the actual sending/waiting for the client to read part. =A0Patch 2 >> >> >> also >> >> >> includes a debug statement after the convert-to-thrift stage to >> >> >> verify. >> >> >> >> >> >> -- >> >> >> Jonathan Ellis >> >> >> Project Chair, Apache Cassandra >> >> >> co-founder of Riptano, the source for professional Cassandra suppo= rt >> >> >> http://riptano.com >> >> > >> >> > >> >> >> >> >> >> >> >> -- >> >> Jonathan Ellis >> >> Project Chair, Apache Cassandra >> >> co-founder of Riptano, the source for professional Cassandra support >> >> http://riptano.com >> > >> > >> >> >> >> -- >> Jonathan Ellis >> Project Chair, Apache Cassandra >> co-founder of Riptano, the source for professional Cassandra support >> http://riptano.com > > --=20 Jonathan Ellis Project Chair, Apache Cassandra co-founder of Riptano, the source for professional Cassandra support http://riptano.com --0016364d2cbd9ff9c00492e9f097 Content-Type: text/plain; charset=US-ASCII; name="1620-2-v3.txt" Content-Disposition: attachment; filename="1620-2-v3.txt" Content-Transfer-Encoding: base64 X-Attachment-Id: f_gffsd2uy0 SW5kZXg6IHNyYy9qYXZhL29yZy9hcGFjaGUvY2Fzc2FuZHJhL3RocmlmdC9DYXNzYW5kcmFTZXJ2 ZXIuamF2YQo9PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09Ci0tLSBzcmMvamF2YS9vcmcvYXBhY2hlL2Nhc3NhbmRyYS90aHJp ZnQvQ2Fzc2FuZHJhU2VydmVyLmphdmEJKHJldmlzaW9uIDEwMjI5ODkpCisrKyBzcmMvamF2YS9v cmcvYXBhY2hlL2Nhc3NhbmRyYS90aHJpZnQvQ2Fzc2FuZHJhU2VydmVyLmphdmEJKHdvcmtpbmcg Y29weSkKQEAgLTE4Nyw2ICsxODcsOSBAQAogICAgICAgICAgICAgY29sdW1uRmFtaWxpZXNNYXAu cHV0KGNvbW1hbmQua2V5LCB0aHJpZnRpZmllZENvbHVtbnMpOwogICAgICAgICB9CiAKKyAgICAg ICAgaWYgKGxvZ2dlci5pc0RlYnVnRW5hYmxlZCgpKQorICAgICAgICAgICAgbG9nZ2VyLmRlYnVn KCJTbGljZSBjb252ZXJ0ZWQgdG8gdGhyaWZ0OyBzZW5kaW5nIHRvIGNsaWVudCIpOworCiAgICAg ICAgIHJldHVybiBjb2x1bW5GYW1pbGllc01hcDsKICAgICB9CiAKSW5kZXg6IHNyYy9qYXZhL29y Zy9hcGFjaGUvY2Fzc2FuZHJhL3NlcnZpY2UvUXVvcnVtUmVzcG9uc2VIYW5kbGVyLmphdmEKPT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PQotLS0gc3JjL2phdmEvb3JnL2FwYWNoZS9jYXNzYW5kcmEvc2VydmljZS9RdW9ydW1S ZXNwb25zZUhhbmRsZXIuamF2YQkocmV2aXNpb24gMTAyMjk4OSkKKysrIHNyYy9qYXZhL29yZy9h cGFjaGUvY2Fzc2FuZHJhL3NlcnZpY2UvUXVvcnVtUmVzcG9uc2VIYW5kbGVyLmphdmEJKHdvcmtp bmcgY29weSkKQEAgLTg5LDYgKzg5LDcgQEAKICAgICBwdWJsaWMgdm9pZCByZXNwb25zZShNZXNz YWdlIG1lc3NhZ2UpCiAgICAgewogICAgICAgICByZXNwb25zZXMuYWRkKG1lc3NhZ2UpOworICAg ICAgICByZXNwb25zZVJlc29sdmVyLnByZXByb2Nlc3MobWVzc2FnZSk7CiAgICAgICAgIGlmIChy ZXNwb25zZVJlc29sdmVyLmlzRGF0YVByZXNlbnQocmVzcG9uc2VzKSkKICAgICAgICAgewogICAg ICAgICAgICAgY29uZGl0aW9uLnNpZ25hbCgpOwpJbmRleDogc3JjL2phdmEvb3JnL2FwYWNoZS9j YXNzYW5kcmEvc2VydmljZS9SZWFkUmVzcG9uc2VSZXNvbHZlci5qYXZhCj09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT0KLS0t IHNyYy9qYXZhL29yZy9hcGFjaGUvY2Fzc2FuZHJhL3NlcnZpY2UvUmVhZFJlc3BvbnNlUmVzb2x2 ZXIuamF2YQkocmV2aXNpb24gMTAyMzA4OSkKKysrIHNyYy9qYXZhL29yZy9hcGFjaGUvY2Fzc2Fu ZHJhL3NlcnZpY2UvUmVhZFJlc3BvbnNlUmVzb2x2ZXIuamF2YQkod29ya2luZyBjb3B5KQpAQCAt MjIsMTAgKzIyLDcgQEAKIGltcG9ydCBqYXZhLmlvLkRhdGFJbnB1dFN0cmVhbTsKIGltcG9ydCBq YXZhLmlvLklPRXJyb3I7CiBpbXBvcnQgamF2YS5pby5JT0V4Y2VwdGlvbjsKLWltcG9ydCBqYXZh LnV0aWwuQXJyYXlMaXN0OwotaW1wb3J0IGphdmEudXRpbC5BcnJheXM7Ci1pbXBvcnQgamF2YS51 dGlsLkNvbGxlY3Rpb247Ci1pbXBvcnQgamF2YS51dGlsLkxpc3Q7CitpbXBvcnQgamF2YS51dGls Lio7CiAKIGltcG9ydCBvcmcuYXBhY2hlLmNhc3NhbmRyYS5kYi5Db2x1bW5GYW1pbHk7CiBpbXBv cnQgb3JnLmFwYWNoZS5jYXNzYW5kcmEuZGIuUmVhZFJlc3BvbnNlOwpAQCAtMzcsNiArMzQsNyBA QAogaW1wb3J0IG9yZy5hcGFjaGUuY2Fzc2FuZHJhLm5ldC5NZXNzYWdpbmdTZXJ2aWNlOwogaW1w b3J0IG9yZy5hcGFjaGUuY2Fzc2FuZHJhLnV0aWxzLkZCVXRpbGl0aWVzOwogaW1wb3J0IG9yZy5h cGFjaGUuY2Fzc2FuZHJhLmNvbmZpZy5EYXRhYmFzZURlc2NyaXB0b3I7CitpbXBvcnQgb3JnLmNs aWZmYy5oaWdoX3NjYWxlX2xpYi5Ob25CbG9ja2luZ0hhc2hNYXA7CiAKIGltcG9ydCBvcmcuYXBh Y2hlLmxvZzRqLkxvZ2dlcjsKIApAQCAtNDksNiArNDcsNyBAQAogCXByaXZhdGUgc3RhdGljIExv Z2dlciBsb2dnZXJfID0gTG9nZ2VyLmdldExvZ2dlcihSZWFkUmVzcG9uc2VSZXNvbHZlci5jbGFz cyk7CiAgICAgcHJpdmF0ZSBmaW5hbCBTdHJpbmcgdGFibGU7CiAgICAgcHJpdmF0ZSBmaW5hbCBp bnQgcmVzcG9uc2VDb3VudDsKKyAgICBwcml2YXRlIGZpbmFsIE1hcDxNZXNzYWdlLCBSZWFkUmVz cG9uc2U+IHJlc3VsdHMgPSBuZXcgTm9uQmxvY2tpbmdIYXNoTWFwPE1lc3NhZ2UsIFJlYWRSZXNw b25zZT4oKTsKIAogICAgIHB1YmxpYyBSZWFkUmVzcG9uc2VSZXNvbHZlcihTdHJpbmcgdGFibGUs IGludCByZXNwb25zZUNvdW50KQogICAgIHsKQEAgLTg0LDExICs4MywxMSBAQAogICAgICAgICAg KiBxdWVyeSBleGlzdHMgdGhlbiB3ZSBuZWVkIHRvIGNvbXBhcmUgdGhlIGRpZ2VzdCB3aXRoIAog ICAgICAgICAgKiB0aGUgZGlnZXN0IG9mIHRoZSBkYXRhIHRoYXQgaXMgcmVjZWl2ZWQuCiAgICAg ICAgICovCi0JCWZvciAoTWVzc2FnZSByZXNwb25zZSA6IHJlc3BvbnNlcykKLQkJewkJCQkJICAg ICAgICAgICAgCi0gICAgICAgICAgICBieXRlW10gYm9keSA9IHJlc3BvbnNlLmdldE1lc3NhZ2VC b2R5KCk7Ci0gICAgICAgICAgICBCeXRlQXJyYXlJbnB1dFN0cmVhbSBidWZJbiA9IG5ldyBCeXRl QXJyYXlJbnB1dFN0cmVhbShib2R5KTsKLSAgICAgICAgICAgIFJlYWRSZXNwb25zZSByZXN1bHQg PSBSZWFkUmVzcG9uc2Uuc2VyaWFsaXplcigpLmRlc2VyaWFsaXplKG5ldyBEYXRhSW5wdXRTdHJl YW0oYnVmSW4pKTsKKwkJZm9yIChNZXNzYWdlIG1lc3NhZ2UgOiByZXNwb25zZXMpCisJCXsKKyAg ICAgICAgICAgIFJlYWRSZXNwb25zZSByZXN1bHQgPSByZXN1bHRzLmdldChtZXNzYWdlKTsKKyAg ICAgICAgICAgIGlmIChyZXN1bHQgPT0gbnVsbCkKKyAgICAgICAgICAgICAgICBjb250aW51ZTsg Ly8gYXJyaXZlZCBhZnRlciBxdW9ydW0gYWxyZWFkeSBhY2hpZXZlZAogICAgICAgICAgICAgaWYg KHJlc3VsdC5pc0RpZ2VzdFF1ZXJ5KCkpCiAgICAgICAgICAgICB7CiAgICAgICAgICAgICAgICAg ZGlnZXN0ID0gcmVzdWx0LmRpZ2VzdCgpOwpAQCAtOTcsMTQgKzk2LDExIEBACiAgICAgICAgICAg ICBlbHNlCiAgICAgICAgICAgICB7CiAgICAgICAgICAgICAgICAgdmVyc2lvbnMuYWRkKHJlc3Vs dC5yb3coKS5jZik7Ci0gICAgICAgICAgICAgICAgZW5kUG9pbnRzLmFkZChyZXNwb25zZS5nZXRG cm9tKCkpOworICAgICAgICAgICAgICAgIGVuZFBvaW50cy5hZGQobWVzc2FnZS5nZXRGcm9tKCkp OwogICAgICAgICAgICAgICAgIGtleSA9IHJlc3VsdC5yb3coKS5rZXk7CiAgICAgICAgICAgICB9 CiAgICAgICAgIH0KIAotICAgICAgICBpZiAobG9nZ2VyXy5pc0RlYnVnRW5hYmxlZCgpKQotICAg ICAgICAgICAgbG9nZ2VyXy5kZWJ1ZygicmVzcG9uc2VzIGRlc2VyaWFsaXplZCIpOwotCiAJCS8v IElmIHRoZXJlIHdhcyBhIGRpZ2VzdCBxdWVyeSBjb21wYXJlIGl0IHdpdGggYWxsIHRoZSBkYXRh IGRpZ2VzdHMKIAkJLy8gSWYgdGhlcmUgaXMgYSBtaXNtYXRjaCB0aGVuIHRocm93IGFuIGV4Y2Vw dGlvbiBzbyB0aGF0IHJlYWQgcmVwYWlyIGNhbiBoYXBwZW4uCiAgICAgICAgIGlmIChpc0RpZ2Vz dFF1ZXJ5KQpAQCAtMTkwLDMwICsxODYsMzYgQEAKICAgICAgICAgcmV0dXJuIHJlc29sdmVkOwog ICAgIH0KIAotCXB1YmxpYyBib29sZWFuIGlzRGF0YVByZXNlbnQoQ29sbGVjdGlvbjxNZXNzYWdl PiByZXNwb25zZXMpCisgICAgcHVibGljIHZvaWQgcHJlcHJvY2VzcyhNZXNzYWdlIG1lc3NhZ2Up CisgICAgeworICAgICAgICBieXRlW10gYm9keSA9IG1lc3NhZ2UuZ2V0TWVzc2FnZUJvZHkoKTsK KyAgICAgICAgQnl0ZUFycmF5SW5wdXRTdHJlYW0gYnVmSW4gPSBuZXcgQnl0ZUFycmF5SW5wdXRT dHJlYW0oYm9keSk7CisgICAgICAgIHRyeQorICAgICAgICB7CisgICAgICAgICAgICBSZWFkUmVz cG9uc2UgcmVzdWx0ID0gUmVhZFJlc3BvbnNlLnNlcmlhbGl6ZXIoKS5kZXNlcmlhbGl6ZShuZXcg RGF0YUlucHV0U3RyZWFtKGJ1ZkluKSk7CisgICAgICAgICAgICByZXN1bHRzLnB1dChtZXNzYWdl LCByZXN1bHQpOworICAgICAgICB9CisgICAgICAgIGNhdGNoIChJT0V4Y2VwdGlvbiBlKQorICAg ICAgICB7CisgICAgICAgICAgICB0aHJvdyBuZXcgSU9FcnJvcihlKTsKKyAgICAgICAgfQorICAg IH0KKworICAgIHB1YmxpYyBib29sZWFuIGlzRGF0YVByZXNlbnQoQ29sbGVjdGlvbjxNZXNzYWdl PiByZXNwb25zZXMpCiAJewogICAgICAgICBpZiAocmVzcG9uc2VzLnNpemUoKSA8IHJlc3BvbnNl Q291bnQpCiAgICAgICAgICAgICByZXR1cm4gZmFsc2U7CiAKLSAgICAgICAgYm9vbGVhbiBpc0Rh dGFQcmVzZW50ID0gZmFsc2U7Ci0gICAgICAgIGZvciAoTWVzc2FnZSByZXNwb25zZSA6IHJlc3Bv bnNlcykKKyAgICAgICAgZm9yIChNZXNzYWdlIG1lc3NhZ2UgOiByZXNwb25zZXMpCiAgICAgICAg IHsKLSAgICAgICAgICAgIGJ5dGVbXSBib2R5ID0gcmVzcG9uc2UuZ2V0TWVzc2FnZUJvZHkoKTsK LSAgICAgICAgICAgIEJ5dGVBcnJheUlucHV0U3RyZWFtIGJ1ZkluID0gbmV3IEJ5dGVBcnJheUlu cHV0U3RyZWFtKGJvZHkpOwotICAgICAgICAgICAgdHJ5Ci0gICAgICAgICAgICB7Ci0gICAgICAg ICAgICAgICAgUmVhZFJlc3BvbnNlIHJlc3VsdCA9IFJlYWRSZXNwb25zZS5zZXJpYWxpemVyKCku ZGVzZXJpYWxpemUobmV3IERhdGFJbnB1dFN0cmVhbShidWZJbikpOwotICAgICAgICAgICAgICAg IGlmICghcmVzdWx0LmlzRGlnZXN0UXVlcnkoKSkKLSAgICAgICAgICAgICAgICB7Ci0gICAgICAg ICAgICAgICAgICAgIGlzRGF0YVByZXNlbnQgPSB0cnVlOwotICAgICAgICAgICAgICAgIH0KLSAg ICAgICAgICAgICAgICBidWZJbi5jbG9zZSgpOwotICAgICAgICAgICAgfQotICAgICAgICAgICAg Y2F0Y2ggKElPRXhjZXB0aW9uIGV4KQotICAgICAgICAgICAgewotICAgICAgICAgICAgICAgIHRo cm93IG5ldyBSdW50aW1lRXhjZXB0aW9uKGV4KTsKLSAgICAgICAgICAgIH0KKyAgICAgICAgICAg IFJlYWRSZXNwb25zZSByZXN1bHQgPSByZXN1bHRzLmdldChtZXNzYWdlKTsKKyAgICAgICAgICAg IGlmIChyZXN1bHQgPT0gbnVsbCkKKyAgICAgICAgICAgICAgICBjb250aW51ZTsgLy8gYXJyaXZl ZCBhZnRlciBxdW9ydW0gYWxyZWFkeSBhY2hpZXZlZAorICAgICAgICAgICAgaWYgKCFyZXN1bHQu aXNEaWdlc3RRdWVyeSgpKQorICAgICAgICAgICAgICAgIHJldHVybiB0cnVlOwogICAgICAgICB9 Ci0gICAgICAgIHJldHVybiBpc0RhdGFQcmVzZW50OworCisgICAgICAgIHJldHVybiBmYWxzZTsK ICAgICB9CisKIH0KSW5kZXg6IHNyYy9qYXZhL29yZy9hcGFjaGUvY2Fzc2FuZHJhL3NlcnZpY2Uv UmFuZ2VTbGljZVJlc3BvbnNlUmVzb2x2ZXIuamF2YQo9PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09Ci0tLSBzcmMvamF2YS9v cmcvYXBhY2hlL2Nhc3NhbmRyYS9zZXJ2aWNlL1JhbmdlU2xpY2VSZXNwb25zZVJlc29sdmVyLmph dmEJKHJldmlzaW9uIDEwMjI5ODkpCisrKyBzcmMvamF2YS9vcmcvYXBhY2hlL2Nhc3NhbmRyYS9z ZXJ2aWNlL1JhbmdlU2xpY2VSZXNwb25zZVJlc29sdmVyLmphdmEJKHdvcmtpbmcgY29weSkKQEAg LTEwOCw2ICsxMDgsMTAgQEAKICAgICAgICAgcmV0dXJuIHJlc29sdmVkUm93czsKICAgICB9CiAK KyAgICBwdWJsaWMgdm9pZCBwcmVwcm9jZXNzKE1lc3NhZ2UgbWVzc2FnZSkKKyAgICB7CisgICAg fQorCiAgICAgcHVibGljIGJvb2xlYW4gaXNEYXRhUHJlc2VudChDb2xsZWN0aW9uPE1lc3NhZ2U+ IHJlc3BvbnNlcykKICAgICB7CiAgICAgICAgIHJldHVybiByZXNwb25zZXMuc2l6ZSgpID49IHNv dXJjZXMuc2l6ZSgpOwpJbmRleDogc3JjL2phdmEvb3JnL2FwYWNoZS9jYXNzYW5kcmEvc2Vydmlj ZS9JUmVzcG9uc2VSZXNvbHZlci5qYXZhCj09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT0KLS0tIHNyYy9qYXZhL29yZy9hcGFj aGUvY2Fzc2FuZHJhL3NlcnZpY2UvSVJlc3BvbnNlUmVzb2x2ZXIuamF2YQkocmV2aXNpb24gMTAy Mjk4OSkKKysrIHNyYy9qYXZhL29yZy9hcGFjaGUvY2Fzc2FuZHJhL3NlcnZpY2UvSVJlc3BvbnNl UmVzb2x2ZXIuamF2YQkod29ya2luZyBjb3B5KQpAQCAtMzcsNCArMzcsNSBAQAogCXB1YmxpYyBU IHJlc29sdmUoQ29sbGVjdGlvbjxNZXNzYWdlPiByZXNwb25zZXMpIHRocm93cyBEaWdlc3RNaXNt YXRjaEV4Y2VwdGlvbiwgSU9FeGNlcHRpb247CiAJcHVibGljIGJvb2xlYW4gaXNEYXRhUHJlc2Vu dChDb2xsZWN0aW9uPE1lc3NhZ2U+IHJlc3BvbnNlcyk7CiAKKyAgICBwdWJsaWMgdm9pZCBwcmVw cm9jZXNzKE1lc3NhZ2UgbWVzc2FnZSk7CiB9Cg== --0016364d2cbd9ff9c00492e9f097--