Return-Path: Delivered-To: apmail-geronimo-activemq-users-archive@www.apache.org Received: (qmail 17907 invoked from network); 3 Jul 2006 13:34:57 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (209.237.227.199) by minotaur.apache.org with SMTP; 3 Jul 2006 13:34:57 -0000 Received: (qmail 9234 invoked by uid 500); 3 Jul 2006 13:34:56 -0000 Delivered-To: apmail-geronimo-activemq-users-archive@geronimo.apache.org Received: (qmail 9211 invoked by uid 500); 3 Jul 2006 13:34:56 -0000 Mailing-List: contact activemq-users-help@geronimo.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: activemq-users@geronimo.apache.org Delivered-To: mailing list activemq-users@geronimo.apache.org Received: (qmail 9202 invoked by uid 99); 3 Jul 2006 13:34:56 -0000 Received: from asf.osuosl.org (HELO asf.osuosl.org) (140.211.166.49) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 03 Jul 2006 06:34:56 -0700 X-ASF-Spam-Status: No, hits=2.5 required=10.0 tests=DNS_FROM_RFC_ABUSE,RCVD_IN_BL_SPAMCOP_NET,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (asf.osuosl.org: domain of caboom@gmail.com designates 72.14.214.194 as permitted sender) Received: from [72.14.214.194] (HELO hu-out-0102.google.com) (72.14.214.194) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 03 Jul 2006 06:34:54 -0700 Received: by hu-out-0102.google.com with SMTP id 28so812092hug for ; Mon, 03 Jul 2006 06:34:32 -0700 (PDT) DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=beta; d=gmail.com; h=received:message-id:date:from:to:subject:in-reply-to:mime-version:content-type:references; b=BRUSH1IVzFNsY9qZduw0Heh46HNQGWAzylomg7WZ6KFFkNbhNS4azQdRDypDrbC5+g/rv5UcFvd/wHN4ZnzyIdDFF0dD9OMAHfWvLkqvvgiyUCM+8UgbNqFvfSaAJ1b2fD3OSluiGaeVs4W9BMCLZG0/z8JcG7EqdNthEHA/ZWg= Received: by 10.70.80.1 with SMTP id d1mr5862877wxb; Mon, 03 Jul 2006 06:34:32 -0700 (PDT) Received: by 10.70.129.8 with HTTP; Mon, 3 Jul 2006 06:34:32 -0700 (PDT) Message-ID: <9de08efa0607030634r55c1ea4ev9ffae73d20b1ee5b@mail.gmail.com> Date: Mon, 3 Jul 2006 15:34:32 +0200 From: "Igor Bogicevic" To: activemq-users@geronimo.apache.org Subject: Re: problems with ActiveMQ with large number of producers/consumers In-Reply-To: MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="----=_Part_27007_21513279.1151933672081" References: <9de08efa0606210535n6987a05cu5e81d40483127fa7@mail.gmail.com> <9de08efa0606210912n2dccb64cqa4895d144c1ae7c8@mail.gmail.com> <9de08efa0606210940h445d2c95oc663895bbb9e3ba4@mail.gmail.com> <5126960.post@talk.nabble.com> <9de08efa0606301548v5c4912f9m488211a58dbf830a@mail.gmail.com> <9de08efa0607030151o4f88beefo7d35abdb5fdb07c0@mail.gmail.com> X-Virus-Checked: Checked by ClamAV on apache.org X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N ------=_Part_27007_21513279.1151933672081 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline Hi James, well, it doesn't depend on the prefetch value (I have tried with 1000 and 2000 prefetch messages, and without prefetching at all), and what happens is this: 1) create about 60 producers and 60 consumers (I have attached the small and ugly test I used - basically I've forked about 60 processes on each side, which means they had separate sessions and connections) 2) if you cut out the shutdown hook in this test, which handles cleanup of connections and sessions, and you kill large amount of producers/consumers this queue will be blocked in the terms that neither producers or consumers will be able to either send or receive messages on this queue - sometimes this happens for a period of a 20-30s and very often for good (>10h, that's the longest period i've left it before i've restarted broker). 3) in case there is shutdown hook with properly closing sessions and connections, this is very hard to repeat, and in case of 4.0.0 i've managed to block the queue with repetitve starting/killing large number of producers and consumers, but i've never managed to reproduce this with 4.0.1 4) i could see a large number of hanging connections (jmx console showed much more connections than there actually were) 4) after restarting broker all works like a charm Thanks, Igor On 7/3/06, James Strachan wrote: > By 'blocking the queue' do you just mean messages were not dispatched > to consumers? If so how many messages were on the queue, how many > connections & consumers did the broker think were open & what were the > prefetch values? If you are having trouble of dead connections not > being closed down agressively enough then your issue could just be an > effect of messages being placed into prefetch buffers for clients that > have not been detected as failed yet? Do you have any kind of JUnit > test case that can replicate your issue? > > On 7/3/06, Igor Bogicevic wrote: > > I am using activemq 4.0.1 (I've did tests on the both 4.0.0 and 4.0.1) > > and I have attached activemq.xml (basically, almost out of the box > > configuration). I am using persistent queues, and I have tried with > > prefetching and transactions, but that didn't really change the > > behavior. > > > > Thanks, > > Igor > > > > On 7/3/06, James Strachan wrote: > > > On 6/30/06, Igor Bogicevic wrote: > > > > I also forgot to answer on this thread... this was not the problem > > > > related to the lack of the memory, it's related to hanging > > > > connections, i've set up the broker with the large amount of memory > > > > (>2GB) in both broker configuration and via jconsole and problem was > > > > defintely in the connections which were not cleaned up - for unknown > > > > reason to me. it was easy to block queue with large number of > > > > un-properly closed sessions and relatively hard with properly closed > > > > connections in a shutdown hook. > > > > > > Using what version and what quality of service? e.g. are we talking > > > 4.0.1 and non-persistent queues? What was your XML config file? > > > > > > -- > > > > > > James > > > ------- > > > http://radio.weblogs.com/0112098/ > > > > > > > > > > > > -- > > James > ------- > http://radio.weblogs.com/0112098/ > ------=_Part_27007_21513279.1151933672081 Content-Type: text/x-java; name=ActivemqSafeJMSTest.java; charset=ISO-8859-1 Content-Transfer-Encoding: base64 X-Attachment-Id: f_ep6vfro8 Content-Disposition: attachment; filename="ActivemqSafeJMSTest.java" cGFja2FnZSB0ZXN0OwoKaW1wb3J0IG9yZy5hcGFjaGUuYWN0aXZlbXEuQWN0aXZlTVFDb25uZWN0 aW9uOwppbXBvcnQgb3JnLmFwYWNoZS5hY3RpdmVtcS5BY3RpdmVNUUNvbm5lY3Rpb25GYWN0b3J5 OwppbXBvcnQgb3JnLmFwYWNoZS5hY3RpdmVtcS51dGlsLkluZGVudFByaW50ZXI7CgppbXBvcnQg amF2YXguam1zLkNvbm5lY3Rpb247CmltcG9ydCBqYXZheC5qbXMuRGVzdGluYXRpb247CmltcG9y dCBqYXZheC5qbXMuRGVsaXZlcnlNb2RlOwppbXBvcnQgamF2YXguam1zLlNlc3Npb247CmltcG9y dCBqYXZheC5qbXMuRXhjZXB0aW9uTGlzdGVuZXI7CmltcG9ydCBqYXZheC5qbXMuSk1TRXhjZXB0 aW9uOwppbXBvcnQgamF2YXguam1zLk1lc3NhZ2VQcm9kdWNlcjsKaW1wb3J0IGphdmF4Lmptcy5N ZXNzYWdlQ29uc3VtZXI7CmltcG9ydCBqYXZheC5qbXMuTWVzc2FnZUxpc3RlbmVyOwppbXBvcnQg amF2YXguam1zLlRleHRNZXNzYWdlOwppbXBvcnQgamF2YXguam1zLk1lc3NhZ2U7CmltcG9ydCBq YXZheC5qbXMuVG9waWM7CgppbXBvcnQgamF2YS5pby5JT0V4Y2VwdGlvbjsKaW1wb3J0IGphdmEu dXRpbC5SYW5kb207CmltcG9ydCBqYXZhLnV0aWwuRGF0ZTsKCi8vIEFjdGl2ZW1xU2FmZUpNU1Rl c3QgY2xhc3MKCnB1YmxpYyBjbGFzcyBBY3RpdmVtcVNhZmVKTVNUZXN0IGltcGxlbWVudHMgTWVz c2FnZUxpc3RlbmVyLCBFeGNlcHRpb25MaXN0ZW5lciAKewogIHByb3RlY3RlZCBEZXN0aW5hdGlv biBkZXN0aW5hdGlvbjsKICBwcm90ZWN0ZWQgU3RyaW5nIHN1YmplY3QgPSAiIjsKICBwcm90ZWN0 ZWQgU3RyaW5nIHVybCA9ICIiOwogIC8vIHByb3BlcnRpZXMKICBwcm90ZWN0ZWQgYm9vbGVhbiB0 b3BpYyA9IGZhbHNlOwogIHByb3RlY3RlZCBib29sZWFuIHRyYW5zYWN0ZWQgPSBmYWxzZTsKICBw cm90ZWN0ZWQgYm9vbGVhbiBkdXJhYmxlID0gZmFsc2U7CiAgcHJvdGVjdGVkIGJvb2xlYW4gcGVy c2lzdGVudCA9IGZhbHNlOwogIHByb3RlY3RlZCBib29sZWFuIHZlcmJvc2UgPSBmYWxzZTsKICBw cml2YXRlIGxvbmcgc2xlZXBUaW1lPTBMOwogIHByb3RlY3RlZCBpbnQgYWNrTW9kZSA9IFNlc3Np b24uQ0xJRU5UX0FDS05PV0xFREdFOwogIC8vIGlkZW50aWZpY2F0aW9uCiAgcHJvdGVjdGVkIFN0 cmluZyB1c2VyID0gQWN0aXZlTVFDb25uZWN0aW9uLkRFRkFVTFRfVVNFUjsKICBwcm90ZWN0ZWQg U3RyaW5nIHB3ZCA9IEFjdGl2ZU1RQ29ubmVjdGlvbi5ERUZBVUxUX1BBU1NXT1JEOwogIHByb3Rl Y3RlZCBTdHJpbmcgY2xpZW50SUQgPSAiIjsKICBwcm90ZWN0ZWQgU3RyaW5nIGNvbnN1bWVyTmFt ZSA9ICIiOwoKICAvLyBjb25zdW1lcgogIHByb3RlY3RlZCBpbnQgY291bnQgPSAwOwogIHByb3Rl Y3RlZCBpbnQgZHVtcENvdW50ID0gMjAwMDA7CiAgcHJvdGVjdGVkIGludCBtYXhpdW1NZXNzYWdl cyA9IDA7CiAgcHJpdmF0ZSBib29sZWFuIHBhdXNlQmVmb3JlU2h1dGRvd247CiAgcHJpdmF0ZSBi b29sZWFuIHJ1bm5pbmc7CiAgcHJpdmF0ZSBTZXNzaW9uIGdsb2JTZXNzaW9uOwogIHByaXZhdGUg Q29ubmVjdGlvbiBnbG9iQ29ubmVjdGlvbjsKICBwcml2YXRlIGxvbmcgcmVjZWl2ZVRpbWVPdXQ9 MDsKCiAgLy8gcHJvZHVjZXIKICBwcm90ZWN0ZWQgaW50IG1lc3NhZ2VTaXplID0gMTAwMDsKICBw cm90ZWN0ZWQgaW50IG1lc3NhZ2VDb3VudCA9IDEwMDA7CiAgcHJvdGVjdGVkIGxvbmcgdGltZVRv TGl2ZSA9IDBMOwoKICBwdWJsaWMgQWN0aXZlbXFTYWZlSk1TVGVzdCgpIHsKICAgIFJ1bnRpbWUu Z2V0UnVudGltZSgpLmFkZFNodXRkb3duSG9vayhuZXcgVGhyZWFkKCkgeyAgCiAgICAgIHB1Ymxp YyB2b2lkIHJ1bigpewogICAgICAgIHRyeSB7CiAgICAgICAgICBpZihnbG9iU2Vzc2lvbiAhPSBu dWxsKSAKICAgICAgICAgICAgZ2xvYlNlc3Npb24uY2xvc2UoKTsKICAgICAgICAgIGlmKGdsb2JD b25uZWN0aW9uICE9IG51bGwpCiAgICAgICAgICAgIGdsb2JDb25uZWN0aW9uLmNsb3NlKCk7IAoK ICAgICAgICAgIFN5c3RlbS5vdXQucHJpbnRsbigiU2h1dGluZyBkb3duIG5pY2VseS4iKTsKICAg ICAgICB9CiAgICAgICAgY2F0Y2ggKEpNU0V4Y2VwdGlvbiBlKSB7CiAgICAgICAgICBTeXN0ZW0u b3V0LnByaW50bG4oIkZhaWxlZCB0byBzaHV0ZG93biBuaWNlbHkuIik7CiAgICAgICAgfQogICAg ICB9CiAgICB9KTsKCiAgfQoKICBwdWJsaWMgc3RhdGljIHZvaWQgbWFpbihTdHJpbmdbXSBhcmdz KSB7CiAgICBpbnQgY291bnQgPSAwOwogICAgU3RyaW5nIG1vZGUgPSAiIjsKICAgIEFjdGl2ZW1x U2FmZUpNU1Rlc3QgbmV3VGVzdCA9IG5ldyBBY3RpdmVtcVNhZmVKTVNUZXN0KCk7CgogICAgZm9y IChpbnQgaSA9IDA7IGkgPCBhcmdzLmxlbmd0aDsgaSsrKSB7ICAKICAgICAgLy8gcHJvcGVydGll cwogICAgICBpZiAoYXJnc1tpXS5tYXRjaGVzKCItLWR1cmFibGUiKSkgewogICAgICAgIG5ld1Rl c3QuZHVyYWJsZSA9IHRydWU7CiAgICAgIH0KICAgICAgZWxzZSBpZiAoYXJnc1tpXS5tYXRjaGVz KCItLXRyYW5zYWN0ZWQiKSkgewogICAgICAgIG5ld1Rlc3QudHJhbnNhY3RlZCA9IHRydWU7CiAg ICAgIH0KICAgICAgZWxzZSBpZiAoYXJnc1tpXS5tYXRjaGVzKCItLXBlcnNpc3RlbnQiKSkgewog ICAgICAgIG5ld1Rlc3QucGVyc2lzdGVudCA9IHRydWU7CiAgICAgIH0KICAgICAgZWxzZSBpZiAo YXJnc1tpXS5tYXRjaGVzKCItLXRvcGljIikpIHsKICAgICAgICBuZXdUZXN0LnRvcGljID0gdHJ1 ZTsKICAgICAgfQogICAgICBlbHNlIGlmIChhcmdzW2ldLm1hdGNoZXMoIi0tdmVyYm9zZSIpKSB7 CiAgICAgICAgbmV3VGVzdC52ZXJib3NlID0gdHJ1ZTsKICAgICAgfQogICAgICAvLyB2YWx1ZXMK ICAgICAgZWxzZSBpZiAoYXJnc1tpXS5tYXRjaGVzKCItLWNsaWVudGlkIikpIHsKICAgICAgICBp ZiAoaSA8IGFyZ3MubGVuZ3RoIC0gMSkgewogICAgICAgICAgbmV3VGVzdC5jbGllbnRJRCA9IGFy Z3NbKytpXTsKICAgICAgICB9CiAgICAgICAgZWxzZSB7CiAgICAgICAgICB1c2FnZSgpOwogICAg ICAgIH0KICAgICAgfQogICAgICBlbHNlIGlmIChhcmdzW2ldLm1hdGNoZXMoIi0tY29uc3VtZXJu YW1lIikpIHsKICAgICAgICBpZiAoaSA8IGFyZ3MubGVuZ3RoIC0gMSkgewogICAgICAgICAgbmV3 VGVzdC5jb25zdW1lck5hbWUgPSBhcmdzWysraV07CiAgICAgICAgfQogICAgICAgIGVsc2Ugewog ICAgICAgICAgdXNhZ2UoKTsKICAgICAgICB9CiAgICAgIH0KICAgICAgZWxzZSBpZiAoYXJnc1tp XS5tYXRjaGVzKCItLXN1YmplY3QiKSkgewogICAgICAgIGlmIChpIDwgYXJncy5sZW5ndGggLSAx KSB7CiAgICAgICAgICBuZXdUZXN0LnN1YmplY3QgPSBhcmdzWysraV07CiAgICAgICAgfQogICAg ICAgIGVsc2UgewogICAgICAgICAgdXNhZ2UoKTsKICAgICAgICB9CiAgICAgIH0KICAgICAgZWxz ZSBpZiAoYXJnc1tpXS5tYXRjaGVzKCItLXVybCIpKSB7CiAgICAgICAgaWYgKGkgPCBhcmdzLmxl bmd0aCAtIDEpIHsKICAgICAgICAgIG5ld1Rlc3QudXJsID0gYXJnc1srK2ldOwogICAgICAgIH0K ICAgICAgICBlbHNlIHsKICAgICAgICAgIHVzYWdlKCk7CiAgICAgICAgfQogICAgICB9CiAgICAg IGVsc2UgaWYgKGFyZ3NbaV0ubWF0Y2hlcygiLS1zaXplIikpIHsKICAgICAgICBpZiAoaSA8IGFy Z3MubGVuZ3RoIC0gMSkgewogICAgICAgICAgdHJ5IHsKICAgICAgICAgICAgbmV3VGVzdC5tZXNz YWdlU2l6ZSA9IEludGVnZXIucGFyc2VJbnQoYXJnc1srK2ldKTsKICAgICAgICAgIH0gY2F0Y2gg KE51bWJlckZvcm1hdEV4Y2VwdGlvbiBueCkgewogICAgICAgICAgICBTeXN0ZW0ub3V0LnByaW50 bG4oImJhZCBpbnRlZ2VyLi4uIik7CiAgICAgICAgICAgIFN5c3RlbS5leGl0KDEpOwogICAgICAg ICAgfQogICAgICAgIH0KICAgICAgICBlbHNlIHsKICAgICAgICAgIHVzYWdlKCk7CiAgICAgICAg fQogICAgICB9CiAgICAgIGVsc2UgaWYgKGFyZ3NbaV0ubWF0Y2hlcygiLS1tZXNzYWdlcyIpKSB7 CiAgICAgICAgaWYgKGkgPCBhcmdzLmxlbmd0aCAtIDEpIHsKICAgICAgICAgIHRyeSB7CiAgICAg ICAgICAgIG5ld1Rlc3QubWVzc2FnZUNvdW50ID0gSW50ZWdlci5wYXJzZUludChhcmdzWysraV0p OwogICAgICAgICAgfSBjYXRjaCAoTnVtYmVyRm9ybWF0RXhjZXB0aW9uIG54KSB7CiAgICAgICAg ICAgIFN5c3RlbS5vdXQucHJpbnRsbigiYmFkIGludGVnZXIuLi4iKTsKICAgICAgICAgICAgU3lz dGVtLmV4aXQoMSk7CiAgICAgICAgICB9CiAgICAgICAgfQogICAgICAgIGVsc2UgewogICAgICAg ICAgdXNhZ2UoKTsKICAgICAgICB9CiAgICAgIH0KICAgICAgLy8gbW9kZQogICAgICBlbHNlIGlm IChpID09IGFyZ3MubGVuZ3RoIC0gMSkgewogICAgICAgIG1vZGUgPSBhcmdzW2ldOwogICAgICB9 CiAgICAgIC8vIGYqY2t1cCAKICAgICAgZWxzZSB7CiAgICAgICAgdXNhZ2UoKTsKICAgICAgfQog ICAgfQoKICAgIC8vIHJhbmRvbWl6ZSBjb25zdW1lciBuYW1lIGFuZCBjbGllbnQgaWQgaWYgaXQn cyBub3QgYmVpbmcgc2V0IC0gYWthIGdpdmUgbW9qbwogICAgaWYgKG5ld1Rlc3QuY29uc3VtZXJO YW1lLm1hdGNoZXMoIiIpKSB7CiAgICAgIFJhbmRvbSByYW5kb21pemVyID0gbmV3IFJhbmRvbSgp OwogICAgICBuZXdUZXN0LmNvbnN1bWVyTmFtZSA9IExvbmcudG9TdHJpbmcoTWF0aC5hYnMocmFu ZG9taXplci5uZXh0TG9uZygpKSwgMzYpOwogICAgfQogICAgaWYgKG5ld1Rlc3QuY2xpZW50SUQu bWF0Y2hlcygiIikpIHsKICAgICAgUmFuZG9tIHJhbmRvbWl6ZXIgPSBuZXcgUmFuZG9tKCk7CiAg ICAgIG5ld1Rlc3QuY2xpZW50SUQgPSBMb25nLnRvU3RyaW5nKE1hdGguYWJzKHJhbmRvbWl6ZXIu bmV4dExvbmcoKSksIDM2KTsKICAgIH0KCiAgICAvLyBydW4gdGVzdAogICAgaWYgKG1vZGUuZXF1 YWxzKCJjb25zdW1lciIpKSB7CiAgICAgIG5ld1Rlc3QuY29uc3VtZSgpOwogICAgfQogICAgZWxz ZSBpZiAobW9kZS5lcXVhbHMoInByb2R1Y2VyIikpIHsKICAgICAgbmV3VGVzdC5wcm9kdWNlKCk7 CiAgICB9CiAgfQoKICAvLyAgICAgICAgICAgICAgICAvLwogIC8vICoqIEdFTkVSSUMgICoqIC8v CiAgLy8gICAgICAgICAgICAgICAgLy8KCgogIHByb3RlY3RlZCBTZXNzaW9uIGNyZWF0ZVNlc3Np b24oQ29ubmVjdGlvbiBjb25uZWN0aW9uKSB0aHJvd3MgRXhjZXB0aW9uIHsKICAgIFNlc3Npb24g c2Vzc2lvbiA9IGNvbm5lY3Rpb24uY3JlYXRlU2Vzc2lvbih0cmFuc2FjdGVkLCBhY2tNb2RlKTsK ICAgIGlmICh0b3BpYykgewogICAgICBkZXN0aW5hdGlvbiA9IHNlc3Npb24uY3JlYXRlVG9waWMo c3ViamVjdCk7CiAgICB9CiAgICBlbHNlIHsKICAgICAgZGVzdGluYXRpb24gPSBzZXNzaW9uLmNy ZWF0ZVF1ZXVlKHN1YmplY3QpOwogICAgfQogICAgcmV0dXJuIHNlc3Npb247CiAgfQoKICBwcm90 ZWN0ZWQgQ29ubmVjdGlvbiBjcmVhdGVDb25uZWN0aW9uKCkgdGhyb3dzIEpNU0V4Y2VwdGlvbiwg RXhjZXB0aW9uIHsKICAgIEFjdGl2ZU1RQ29ubmVjdGlvbkZhY3RvcnkgY29ubmVjdGlvbkZhY3Rv cnkgPSBuZXcgQWN0aXZlTVFDb25uZWN0aW9uRmFjdG9yeSh1c2VyLCBwd2QsIHVybCk7CiAgICBD b25uZWN0aW9uIGNvbm5lY3Rpb24gPSBjb25uZWN0aW9uRmFjdG9yeS5jcmVhdGVDb25uZWN0aW9u KCk7CiAgICBpZiAoZHVyYWJsZSAmJiBjbGllbnRJRCE9bnVsbCkgewogICAgICBjb25uZWN0aW9u LnNldENsaWVudElEKGNsaWVudElEKTsKICAgIH0KICAgIAogICAgY29ubmVjdGlvbi5zdGFydCgp OwogICAgcmV0dXJuIGNvbm5lY3Rpb247CiAgfQoKICBwcm90ZWN0ZWQgdm9pZCBjbG9zZShDb25u ZWN0aW9uIGNvbm5lY3Rpb24sIFNlc3Npb24gc2Vzc2lvbikgdGhyb3dzIEpNU0V4Y2VwdGlvbiB7 CiAgICAvLyBsZXRzIGR1bXAgdGhlIHN0YXRzCiAgICBkdW1wU3RhdHMoY29ubmVjdGlvbik7Cgog ICAgaWYgKHNlc3Npb24gIT0gbnVsbCkgewogICAgICBzZXNzaW9uLmNsb3NlKCk7CiAgICB9CiAg ICBpZiAoY29ubmVjdGlvbiAhPSBudWxsKSB7CiAgICAgIGNvbm5lY3Rpb24uY2xvc2UoKTsKICAg IH0KICB9CgogIHByb3RlY3RlZCB2b2lkIGR1bXBTdGF0cyhDb25uZWN0aW9uIGNvbm5lY3Rpb24p IHsKICAgIEFjdGl2ZU1RQ29ubmVjdGlvbiBjID0gKEFjdGl2ZU1RQ29ubmVjdGlvbikgY29ubmVj dGlvbjsKICAgIGMuZ2V0Q29ubmVjdGlvblN0YXRzKCkuZHVtcChuZXcgSW5kZW50UHJpbnRlcigp KTsKICB9CgogIC8vICAgICAgICAgICAgICAgIC8vCiAgLy8gKiogQ09OU1VNRVIgKiogLy8KICAv LyAgICAgICAgICAgICAgICAvLwoKICBwcml2YXRlIHZvaWQgY29uc3VtZSgpIHsKICAgIHRyeSB7 CiAgICAgIHJ1bm5pbmcgPSB0cnVlOwogICAgICAgICAgICAKICAgICAgU3lzdGVtLm91dC5wcmlu dGxuKCJDb25uZWN0aW5nIHRvIFVSTDogIiArIHVybCk7CiAgICAgIFN5c3RlbS5vdXQucHJpbnRs bigiQ29uc3VtaW5nICIgKyAodG9waWMgPyAidG9waWMiIDogInF1ZXVlIikgKyAiOiAiICsgc3Vi amVjdCk7CiAgICAgIFN5c3RlbS5vdXQucHJpbnRsbigiVXNpbmcgIiArIChkdXJhYmxlID8gImR1 cmFibGUiIDogIm5vbi1kdXJhYmxlIikgKyAiIHN1YnNjcmlwdGlvbiIpOwogICAgICBTeXN0ZW0u b3V0LnByaW50bG4oIlVzaW5nICIgKyAocGVyc2lzdGVudCA/ICJwZXJzaXN0ZW50IiA6ICJub24t cGVyc2lzdGVudCIpICsgIiBxdWV1ZSIpOwoKICAgICAgZ2xvYkNvbm5lY3Rpb24gPSBjcmVhdGVD b25uZWN0aW9uKCk7CiAgICAgIGdsb2JDb25uZWN0aW9uLnNldEV4Y2VwdGlvbkxpc3RlbmVyKHRo aXMpOwogICAgICBnbG9iU2Vzc2lvbiA9IGNyZWF0ZVNlc3Npb24oZ2xvYkNvbm5lY3Rpb24pOwog ICAgICBNZXNzYWdlQ29uc3VtZXIgY29uc3VtZXIgPSBudWxsOwogICAgICBpZiAoZHVyYWJsZSAm JiB0b3BpYykgewogICAgICAgIGNvbnN1bWVyID0gZ2xvYlNlc3Npb24uY3JlYXRlRHVyYWJsZVN1 YnNjcmliZXIoKFRvcGljKSBkZXN0aW5hdGlvbiwgY29uc3VtZXJOYW1lKTsKICAgICAgfQogICAg ICBlbHNlIHsKICAgICAgICBjb25zdW1lciA9IGdsb2JTZXNzaW9uLmNyZWF0ZUNvbnN1bWVyKGRl c3RpbmF0aW9uKTsKICAgICAgfQogICAgICBpZiAoIG1heGl1bU1lc3NhZ2VzID4gMCApIHsKICAg ICAgICBjb25zdW1lTWVzc2FnZXNBbmRDbG9zZShnbG9iQ29ubmVjdGlvbiwgZ2xvYlNlc3Npb24s IGNvbnN1bWVyKTsKICAgICAgfSBlbHNlICB7CiAgICAgICAgaWYocmVjZWl2ZVRpbWVPdXQ9PTAp IHsKICAgICAgICAgIGNvbnN1bWVyLnNldE1lc3NhZ2VMaXN0ZW5lcih0aGlzKTsKICAgICAgICB9 IGVsc2UgewogICAgICAgICAgY29uc3VtZU1lc3NhZ2VzQW5kQ2xvc2UoZ2xvYkNvbm5lY3Rpb24s IGdsb2JTZXNzaW9uLCBjb25zdW1lciwgcmVjZWl2ZVRpbWVPdXQpOwogICAgICAgIH0KICAgICAg fQogICAgfQogICAgY2F0Y2ggKEV4Y2VwdGlvbiBlKSB7CiAgICAgIFN5c3RlbS5vdXQucHJpbnRs bigiQ2F1Z2h0IHdoaWxlIHN0YXJ0aW5nOiAiICsgZSk7CiAgICAgIGUucHJpbnRTdGFja1RyYWNl KCk7CiAgICB9IAogIH0KCiAgcHVibGljIHZvaWQgb25NZXNzYWdlKE1lc3NhZ2UgbWVzc2FnZSkg ewogICAgdHJ5IHsKICAgICAgaWYgKG1lc3NhZ2UgaW5zdGFuY2VvZiBUZXh0TWVzc2FnZSkgewog ICAgICAgIFRleHRNZXNzYWdlIHR4dE1zZyA9IChUZXh0TWVzc2FnZSkgbWVzc2FnZTsKICAgICAg ICBpZiAodmVyYm9zZSkgewoKICAgICAgICAgIFN0cmluZyBtc2cgPSB0eHRNc2cuZ2V0VGV4dCgp OwogICAgICAgICAgaWYgKG1zZy5sZW5ndGgoKSA+IDUwKSB7CiAgICAgICAgICAgIG1zZyA9IG1z Zy5zdWJzdHJpbmcoMCwgNTApICsgIi4uLiI7CiAgICAgICAgICB9CgogICAgICAgICAgU3lzdGVt Lm91dC5wcmludGxuKCJSZWNlaXZlZDogIiArIG1zZyk7CiAgICAgICAgfQogICAgICB9CiAgICAg IGVsc2UgewogICAgICAgIGlmICh2ZXJib3NlKSB7CiAgICAgICAgICBTeXN0ZW0ub3V0LnByaW50 bG4oIlJlY2VpdmVkOiAiICsgbWVzc2FnZSk7CiAgICAgICAgfQogICAgICB9CiAgICAgIGlmKHRy YW5zYWN0ZWQpIHsKICAgICAgICBnbG9iU2Vzc2lvbi5jb21taXQoKTsKICAgICAgfQoKICAgICAg bWVzc2FnZS5hY2tub3dsZWRnZSgpOwogICAgfQogICAgY2F0Y2ggKEpNU0V4Y2VwdGlvbiBlKSB7 CiAgICAgIFN5c3RlbS5vdXQucHJpbnRsbigiQ2F1Z2h0OiAiICsgZSk7CiAgICAgIGUucHJpbnRT dGFja1RyYWNlKCk7CiAgICB9IGZpbmFsbHkgewogICAgICBpZiggc2xlZXBUaW1lPiAwICkgewog ICAgICAgIHRyeSB7CiAgICAgICAgICBUaHJlYWQuc2xlZXAoc2xlZXBUaW1lKTsKICAgICAgICB9 IGNhdGNoIChJbnRlcnJ1cHRlZEV4Y2VwdGlvbiBlKSB7CiAgICAgIAogICAgICAgIH0KICAgICAg fQogICAgfQogIH0KCiAgcHJvdGVjdGVkIHZvaWQgY29uc3VtZU1lc3NhZ2VzQW5kQ2xvc2UoQ29u bmVjdGlvbiBjb25uZWN0aW9uLCBTZXNzaW9uIHNlc3Npb24sIE1lc3NhZ2VDb25zdW1lciBjb25z dW1lcikgdGhyb3dzIEpNU0V4Y2VwdGlvbiwgSU9FeGNlcHRpb24gewogICAgU3lzdGVtLm91dC5w cmludGxuKCJXZSBhcmUgYWJvdXQgdG8gd2FpdCB1bnRpbCB3ZSBjb25zdW1lOiAiICsgbWF4aXVt TWVzc2FnZXMgKyAiIG1lc3NhZ2UocykgdGhlbiB3ZSB3aWxsIHNodXRkb3duIik7CgogICAgZm9y IChpbnQgaSA9IDA7IGkgPCBtYXhpdW1NZXNzYWdlcyAmJiBpc1J1bm5pbmcoKTsgKSB7CiAgICAg IE1lc3NhZ2UgbWVzc2FnZSA9IGNvbnN1bWVyLnJlY2VpdmUoMTAwMCk7CiAgICAgIGlmKCBtZXNz YWdlIT1udWxsICkgewogICAgICAgIGkrKzsKICAgICAgICBvbk1lc3NhZ2UobWVzc2FnZSk7CiAg ICAgIH0KICAgIH0KICAgIAogICAgU3lzdGVtLm91dC5wcmludGxuKCJDbG9zaW5nIGNvbm5lY3Rp b24iKTsKICAgIGNvbnN1bWVyLmNsb3NlKCk7CiAgICBzZXNzaW9uLmNsb3NlKCk7CiAgICBjb25u ZWN0aW9uLmNsb3NlKCk7CgogICAgY29uc3VtZXIgPSBudWxsOwogICAgc2Vzc2lvbiA9IG51bGw7 CiAgICBjb25uZWN0aW9uID0gbnVsbDsKCiAgICBpZiAocGF1c2VCZWZvcmVTaHV0ZG93bikgewog ICAgICBTeXN0ZW0ub3V0LnByaW50bG4oIlByZXNzIHJldHVybiB0byBzaHV0IGRvd24iKTsKICAg ICAgU3lzdGVtLmluLnJlYWQoKTsKICAgIH0KICB9CiAgICAKICBwcm90ZWN0ZWQgdm9pZCBjb25z dW1lTWVzc2FnZXNBbmRDbG9zZShDb25uZWN0aW9uIGNvbm5lY3Rpb24sIFNlc3Npb24gc2Vzc2lv biwgTWVzc2FnZUNvbnN1bWVyIGNvbnN1bWVyLCBsb25nIHRpbWVvdXQpIHRocm93cyBKTVNFeGNl cHRpb24sIElPRXhjZXB0aW9uIHsKICAgIFN5c3RlbS5vdXQucHJpbnRsbigiV2Ugd2lsbCBjb25z dW1lIG1lc3NhZ2VzIHdoaWxlIHRoZXkgY29udGludWUgdG8gYmUgZGVsaXZlcmVkIHdpdGhpbjog IiArIHRpbWVvdXQgKyAiIG1zLCBhbmQgdGhlbiB3ZSB3aWxsIHNodXRkb3duIik7CgogICAgTWVz c2FnZSBtZXNzYWdlOwogICAgd2hpbGUgKCAobWVzc2FnZSA9IGNvbnN1bWVyLnJlY2VpdmUodGlt ZW91dCkpICE9IG51bGwgKSB7CiAgICAgIG9uTWVzc2FnZShtZXNzYWdlKTsKICAgIH0KICAgICAg ICAKICAgIFN5c3RlbS5vdXQucHJpbnRsbigiQ2xvc2luZyBjb25uZWN0aW9uIik7CiAgICBjb25z dW1lci5jbG9zZSgpOwogICAgc2Vzc2lvbi5jbG9zZSgpOwogICAgY29ubmVjdGlvbi5jbG9zZSgp OwoKICAgIGNvbnN1bWVyID0gbnVsbDsKICAgIHNlc3Npb24gPSBudWxsOwogICAgY29ubmVjdGlv biA9IG51bGw7CgogICAgaWYgKHBhdXNlQmVmb3JlU2h1dGRvd24pIHsKICAgICAgU3lzdGVtLm91 dC5wcmludGxuKCJQcmVzcyByZXR1cm4gdG8gc2h1dCBkb3duIik7CiAgICAgIFN5c3RlbS5pbi5y ZWFkKCk7CiAgICB9CiB9CgoKICBzeW5jaHJvbml6ZWQgcHVibGljIHZvaWQgb25FeGNlcHRpb24o Sk1TRXhjZXB0aW9uIGV4KSB7CiAgICBTeXN0ZW0ub3V0LnByaW50bG4oIkpNUyBFeGNlcHRpb24g b2NjdXJlZC4gIFNodXR0aW5nIGRvd24gY2xpZW50LiIpOwogICAgcnVubmluZz1mYWxzZTsKICB9 CgogIHN5bmNocm9uaXplZCBib29sZWFuIGlzUnVubmluZygpIHsKICAgIHJldHVybiBydW5uaW5n OwogIH0KCiAgLy8gICAgICAgICAgICAgICAgLy8KICAvLyAqKiBQUk9EVUNFUiAqKiAvLwogIC8v ICAgICAgICAgICAgICAgIC8vCgogIHByaXZhdGUgdm9pZCBwcm9kdWNlKCkgewogICAgdHJ5IHsK ICAgICAgU3lzdGVtLm91dC5wcmludGxuKCJDb25uZWN0aW5nIHRvIFVSTDogIiArIHVybCk7CiAg ICAgIFN5c3RlbS5vdXQucHJpbnRsbigiUHVibGlzaGluZyBhIE1lc3NhZ2Ugd2l0aCBzaXplICIg KyBtZXNzYWdlU2l6ZSArICIgdG8gIiArICh0b3BpYyA/ICJ0b3BpYyIgOiAicXVldWUiKSArICI6 ICIgKyBzdWJqZWN0KTsKICAgICAgU3lzdGVtLm91dC5wcmludGxuKCJVc2luZyAiICsgKGR1cmFi bGUgPyAiZHVyYWJsZSIgOiAibm9uLWR1cmFibGUiKSArICIgcHVibGlzaGluZyIpOwogIAogICAg ICBnbG9iQ29ubmVjdGlvbiA9IGNyZWF0ZUNvbm5lY3Rpb24oKTsKICAgICAgZ2xvYlNlc3Npb24g PSBjcmVhdGVTZXNzaW9uKGdsb2JDb25uZWN0aW9uKTsKICAgICAgTWVzc2FnZVByb2R1Y2VyIHBy b2R1Y2VyID0gY3JlYXRlUHJvZHVjZXIoZ2xvYlNlc3Npb24pOwogICAgICBzZW5kTG9vcChnbG9i U2Vzc2lvbiwgcHJvZHVjZXIpOwoKICAgICAgU3lzdGVtLm91dC5wcmludGxuKCJEb25lLiIpOwog ICAgICBjbG9zZShnbG9iQ29ubmVjdGlvbiwgZ2xvYlNlc3Npb24pOyAKICAgIH0KICAgIGNhdGNo IChFeGNlcHRpb24gZSkgewogICAgICBTeXN0ZW0ub3V0LnByaW50bG4oIkNhdWdodDogIiArIGUp OwogICAgICBlLnByaW50U3RhY2tUcmFjZSgpOwogICAgfQogIH0KCiAgcHJvdGVjdGVkIHZvaWQg c2VuZExvb3AoU2Vzc2lvbiBzZXNzaW9uLCBNZXNzYWdlUHJvZHVjZXIgcHJvZHVjZXIpIHRocm93 cyBFeGNlcHRpb24gewogICAgZm9yIChpbnQgaSA9IDA7IGkgPCBtZXNzYWdlQ291bnQgfHwgbWVz c2FnZUNvdW50PT0wIDsgaSsrKSB7CiAgICAgIFRleHRNZXNzYWdlIG1lc3NhZ2UgPSBzZXNzaW9u LmNyZWF0ZVRleHRNZXNzYWdlKGNyZWF0ZU1lc3NhZ2VUZXh0KGkpKTsKCiAgICAgIGlmICh2ZXJi b3NlKSB7CiAgICAgICAgU3RyaW5nIG1zZyA9IG1lc3NhZ2UuZ2V0VGV4dCgpOwogICAgICAgIGlm IChtc2cubGVuZ3RoKCkgPiA1MCkgewogICAgICAgICAgbXNnID0gbXNnLnN1YnN0cmluZygwLCA1 MCkgKyAiLi4uIjsKICAgICAgICB9CiAgICAgICAgU3lzdGVtLm91dC5wcmludGxuKCJTZW5kaW5n IG1lc3NhZ2U6ICIgKyBtc2cpOwogICAgICB9CiAgICAgICAgICAgIAogICAgICBwcm9kdWNlci5z ZW5kKG1lc3NhZ2UpOwogICAgICBpZih0cmFuc2FjdGVkKSB7CiAgICAgICAgc2Vzc2lvbi5jb21t aXQoKTsKICAgICAgfQogICAgICAgICAgICAKICAgICAgVGhyZWFkLnNsZWVwKHNsZWVwVGltZSk7 CiAgICAgICAgICAgIAogICAgfQogIH0KCiAgcHJvdGVjdGVkIE1lc3NhZ2VQcm9kdWNlciBjcmVh dGVQcm9kdWNlcihTZXNzaW9uIHNlc3Npb24pIHRocm93cyBKTVNFeGNlcHRpb24gewogICAgTWVz c2FnZVByb2R1Y2VyIHByb2R1Y2VyID0gc2Vzc2lvbi5jcmVhdGVQcm9kdWNlcihkZXN0aW5hdGlv bik7CiAgICBpZiAoKGR1cmFibGUgJiYgdG9waWMpIHx8IChwZXJzaXN0ZW50ICYmICF0b3BpYykp IHsKICAgICAgcHJvZHVjZXIuc2V0RGVsaXZlcnlNb2RlKERlbGl2ZXJ5TW9kZS5QRVJTSVNURU5U KTsKICAgIH0KICAgIGVsc2UgewogICAgICBwcm9kdWNlci5zZXREZWxpdmVyeU1vZGUoRGVsaXZl cnlNb2RlLk5PTl9QRVJTSVNURU5UKTsKICAgIH0KICAgIGlmKCB0aW1lVG9MaXZlIT0wICkKICAg ICAgcHJvZHVjZXIuc2V0VGltZVRvTGl2ZSh0aW1lVG9MaXZlKTsKICAKICAgIHJldHVybiBwcm9k dWNlcjsKICB9CgoKICBwcml2YXRlIFN0cmluZyBjcmVhdGVNZXNzYWdlVGV4dChpbnQgaW5kZXgp IHsKICAgIFN0cmluZ0J1ZmZlciBidWZmZXIgPSBuZXcgU3RyaW5nQnVmZmVyKG1lc3NhZ2VTaXpl KTsKICAgIGJ1ZmZlci5hcHBlbmQoIk1lc3NhZ2U6ICIgKyBpbmRleCArICIgc2VudCBhdDogIiAr IG5ldyBEYXRlKCkpOwogICAgaWYgKGJ1ZmZlci5sZW5ndGgoKSA+IG1lc3NhZ2VTaXplKSB7CiAg ICAgIHJldHVybiBidWZmZXIuc3Vic3RyaW5nKDAsIG1lc3NhZ2VTaXplKTsKICAgIH0KICAgIGZv ciAoaW50IGkgPSBidWZmZXIubGVuZ3RoKCk7IGkgPCBtZXNzYWdlU2l6ZTsgaSsrKSB7CiAgICAg IGJ1ZmZlci5hcHBlbmQoJyAnKTsKICAgIH0KICAgIHJldHVybiBidWZmZXIudG9TdHJpbmcoKTsK ICB9CgogIC8vICAgICAgICAgICAgLy8KICAvLyAqKiBVVElMICoqIC8vCiAgLy8gICAgICAgICAg ICAvLwoKICBwcml2YXRlIHN0YXRpYyB2b2lkIHVzYWdlKCkgewogICAgU3lzdGVtLm91dC5wcmlu dGxuKCJ1c2FnZSA6IGphdmEgLWNwIHRlc3RNb2R1bGVzLmphciB0ZXN0LkFjdGl2ZW1xU2FmZUpN U1Rlc3QgW29wdGlvbnNdIG1vZGUiKTsKICAgIFN5c3RlbS5vdXQucHJpbnRsbigib3B0aW9uczog ICAgICAgICAgICAgICAgICAgICAgIik7CiAgICBTeXN0ZW0ub3V0LnByaW50bG4oIlx0LS1kdXJh YmxlICAgICAgICAgICAgICAgICAgICIpOwogICAgU3lzdGVtLm91dC5wcmludGxuKCJcdC0tdHJh bnNhY3RlZCAgICAgICAgICAgICAgICAiKTsKICAgIFN5c3RlbS5vdXQucHJpbnRsbigiXHQtLXBl cnNpc3RlbnQgICAgICAgICAgICAgICAgIik7CiAgICBTeXN0ZW0ub3V0LnByaW50bG4oIlx0LS10 b3BpYyAgICAgICAgICAgICAgICAgICAgICIpOwogICAgU3lzdGVtLm91dC5wcmludGxuKCJcdC0t Y2xpZW50aWQgY2xpZW50SUQgICAgICAgICAiKTsKICAgIFN5c3RlbS5vdXQucHJpbnRsbigiXHQt LWNvbnN1bWVybmFtZSBjb25zdW1lck5hbWUgIik7CiAgICBTeXN0ZW0ub3V0LnByaW50bG4oIlx0 LS1zdWJqZWN0IHN1YmplY3QgICAgICAgICAgICIpOwogICAgU3lzdGVtLm91dC5wcmludGxuKCJc dC0tc2l6ZSBzaXplICAgICAgICAgICAgICAgICAiKTsKICAgIFN5c3RlbS5vdXQucHJpbnRsbigi XHQtLW1lc3NhZ2VzIG1lc3NhZ2VDb3VudCAgICAgIik7CiAgICBTeXN0ZW0ub3V0LnByaW50bG4o Ilx0LS11cmwgVVJMICAgICAgICAgICAgICAgICAgICIpOwogICAgCiAgICBTeXN0ZW0uZXhpdCgx KTsKICB9Cn0K ------=_Part_27007_21513279.1151933672081--