Mailing-List: contact dev-help@river.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@river.apache.org
From: Gregg Wonderly <gregg@wonderly.org>
Content-Type: text/plain;
	charset=us-ascii
Content-Transfer-Encoding: quoted-printable
Mime-Version: 1.0 (1.0)
Subject: Re: Trunk merge and thread pools
Message-Id: <DD3E6057-954F-4C82-87ED-E1839321A2D6@wonderly.org>
Date: Mon, 7 Dec 2015 01:09:36 +0300
References: <0b65910a77586d2c68419f3c32a92980@org.tizen.email>
In-Reply-To: <0b65910a77586d2c68419f3c32a92980@org.tizen.email>
To: dev@river.apache.org
Sender: Gregg Wonderly <gregg@wonderly.org>

Well Peter, there are lots of things one can do about load management.  The o=
bvious solutions are visible in current load balancing on web servers.  That=
 simple mechanism of receiving the request and dispatching it into the real s=
ervers provides the ability to manage load with appropriate logic.

So, put your slowest hardware there, use a small fixed sized dispatch pool a=
nd tune its size to an appropriate percent of available time.  That is, time=
 each service requests time to process.  Bias those times by appropriate var=
iation in processing time differences.

As Amazon does, you can use a PID mechanism to automate throttling.

Gregg

Sent from my iPad

> On Dec 3, 2015, at 3:32 PM, Peter <jini@zeus.net.au> wrote:
>=20
> Care to share more of your insight?
>=20
> Peter.
>=20
> Sent from my Samsung device.
>   Include original message
> ---- Original message ----
> From: Gregg Wonderly <gergg@cox.net>
> Sent: 03/12/2015 06:37:15 pm
> To: dev@river.apache.org
> Subject: Re: Trunk merge and thread pool
>=20
> The original use of thread  pooling was more than likely about getting wor=
k done faster by not undergoing overhead of thread creation, since in distri=
buted systems, deferring work can create deadlock by introducing indefinite w=
ait scenarios if resource limits keep work from being dispatched.=20
>=20
> As a general rule of thumb, I have found that waiting till the point of th=
read creation, to create introduce load control, is never the right design. =
 Instead, load control must happen at the head/beginning of any request into=
 a distributed system.=20
>=20
> Gregg=20
>=20
> Sent from my iPhone=20
>=20
>>  On Dec 3, 2015, at 3:26 AM, Peter <jini@zeusnet.au> wrote:=20
>>  =20
>>  Just tried wrapping an Executors.newCachedThreadPool with a thread facto=
ry that creates threads as per the original org.apache.river.thread.NewThrea=
dAction.=20
>>  =20
>>  Performance is much improved, the hotspot is gone.=20
>>  =20
>>  There are regression tests with sun bug Id's, which cause oome.  I thoug=
ht this might =20
>>  prevent the executor from running,  but to my surprise both tests pass. =
  These tests failed when I didn't pool threads and just let them be gc'd.  T=
hese tests created over 11000 threads with waiting tasks.  In practise I wou=
ldn't expect that to happen as an IOException should be thrown.  However the=
re are sun bug id's 6313626 and 6304782 for these regression tests, if anyon=
e has a record of these bugs or any information they can share, it would be m=
uch appreciated.=20
>>  =20
>>  It's worth noting that the jvm memory options should be tuned properly t=
o avoid oome in any case.=20
>>  =20
>>  Lesson here is, creating threads and gc'ing them is much faster than thr=
ead pooling if your thread pool is not well optimised.=20
>>  =20
>>  It's worth noting that ObjectInputStream is now the hotspot for the test=
, the tested code's hotspots are DatagramSocket and SocketInputStream.=20
>>  =20
>>  ClassLoading is thread confined, there's a lot of class loading going on=
, but because it is uncontended, it only consumes 0.2% cpu, about the same a=
s our security architecture overhead (non encrypted).=20
>>  =20
>>  Regards,=20
>>  =20
>>  Peter.=20
>>  =20
>>  Sent from my Samsung device.=20
>>    Include original message=20
>>  ---- Original message ----=20
>>  From: Bryan Thompson <bryan@systap.com>=20
>>  Sent: 02/12/2015 11:25:03 pm=20
>>  To: <dev@river.apache.org> <dev@river.apache.org>=20
>>  Subject: Re: Trunk merge and thread pools=20
>>  =20
>>  Ah. I did not realize that we were discussing a river specific ThreadPoo=
l =20
>>  vs a Java Concurrency classes ThreadPoolExecutor.  I assume that it woul=
d =20
>>  be difficult to just substitute in one of the standard executors? =20
>>  =20
>>  Bryan =20
>>  =20
>>>  On Wed, Dec 2, 2015 at 8:18 AM, Peter <jini@zeus.net.au> wrote: =20
>>>  =20
>>>   First it's worth considering we have a very suboptimal threadpool.  Th=
ere =20
>>>   are qa and jtreg tests that limit our ability to do much with ThreadPo=
ol. =20
>>>  =20
>>>   There are only two instances of ThreadPool, shared by various jeri =20=

>>>   endpoint implementations, and other components. =20
>>>  =20
>>>   The implementation is allowed to create numerous threads, only limited=
 by =20
>>>   available memory and oome.  At least two tests cause it to create over=
 =20
>>>   11000 threads. =20
>>>  =20
>>>   Also, it previously used a LinkedList queue,  but now uses a =20
>>>   BlockingQueue,  however the queue still uses poll, not take. =20
>>>  =20
>>>   The limitation seems to be the concern by the original developers that=
 =20
>>>   there may be interdependencies between tasks.  Most tasks are method =20=

>>>   invocations from incoming and outgoing remote calls. =20
>>>  =20
>>>   It probably warrants further investigation to see if there's a suitabl=
e =20
>>>   replacement. =20
>>>  =20
>>>   Regards, =20
>>>  =20
>>>   Peter. =20
>>>  =20
>>>   Sent from my Samsung device. =20
>>>     Include original message =20
>>>   ---- Original message ---- =20
>>>   From: Bryan Thompson <bryan@systap.com> =20
>>>   Sent: 02/12/2015 09:46:13 am =20
>>>   To: <dev@river.apache.org> <dev@river.apache.org> =20
>>>   Subject: Re: Trunk merge and thread pools =20
>>>  =20
>>>   Peter, =20
>>>  =20
>>>   It might be worth taking this observation about the thread pool behavi=
or to =20
>>>   the java concurrency list.  See what feedback you get.  I would certai=
nly =20
>>>   be interested in what people there have to say about this. =20
>>>  =20
>>>   Bryan
>=20