From dev-return-70416-archive-asf-public=cust-asf.ponee.io@activemq.apache.org Wed Mar 20 13:31:20 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 4E14218062C for ; Wed, 20 Mar 2019 14:31:20 +0100 (CET) Received: (qmail 64418 invoked by uid 500); 20 Mar 2019 13:31:19 -0000 Mailing-List: contact dev-help@activemq.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@activemq.apache.org Delivered-To: mailing list dev@activemq.apache.org Received: (qmail 64406 invoked by uid 99); 20 Mar 2019 13:31:18 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 20 Mar 2019 13:31:18 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 2FF25C0584 for ; Wed, 20 Mar 2019 13:31:18 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 3.644 X-Spam-Level: *** X-Spam-Status: No, score=3.644 tagged_above=-999 required=6.31 tests=[DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=2, KAM_LINEPADDING=1.2, KAM_SHORT=0.001, KHOP_DYNAMIC=1.344, RCVD_IN_DNSWL_LOW=-0.7, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=me.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id 2VT_zEPTvqZS for ; Wed, 20 Mar 2019 13:31:15 +0000 (UTC) Received: from mr85p00im-ztdg06021701.me.com (mr85p00im-ztdg06021701.me.com [17.58.23.196]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id 8A09C624A0 for ; Wed, 20 Mar 2019 13:31:15 +0000 (UTC) Received: from mail.outlook.com (unknown [52.125.140.63]) by mr85p00im-ztdg06021701.me.com (Postfix) with ESMTPSA id 974D2A002B4 for ; Wed, 20 Mar 2019 13:31:08 +0000 (UTC) Date: Wed, 20 Mar 2019 13:30:36 +0000 (UTC) From: michael.andre.pearce@me.com.INVALID To: dev@activemq.apache.org Message-ID: <23C8A0768C50A6D9.9ec8973a-7ca9-4fc4-ba8b-94b381dc097e@mail.outlook.com> In-Reply-To: References: <23C8A0768C50A6D9.8fb7cecd-a630-4875-97cb-ae009d512680@mail.outlook.com> Subject: Re: Artemis Scalability Thoughts MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_8469_1428525355.1553088636437" X-Mailer: Outlook for iOS and Android X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-03-20_08:,, signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 mlxscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1812120000 definitions=main-1903200104 ------=_Part_8469_1428525355.1553088636437 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Performance was similar. As such agreed if there is an impact for another usecase thats genuine maybe leave it as is. Get Outlook for Android On Wed, Mar 20, 2019 at 1:49 PM +0100, "Francesco Nigro" wrote: @michael > If youre talking about the solution that you regressed just before we released. Then we did test it in our real testing env. I didnt notice and negative impact for our use cases. That's nice to be heard, MIchael. Probably I could have avoided to revert it, but consider that the accademic use case ie 1 P/1 C/1 Q is just a way to emulate a "low-utilization" case. As Christopher has said, there are users that don't want to trade low utilization performance...and I understand it. The thing is, have you seen if using TranferLinkedQueue has improved over the original thread pool with many producers/consumers/queues scenario? > Regards to code, i actually thought what you had it made code cleaner. That's another benefit, but the change was meant to provide only benefits, that's why I've reverted it... > think we engineer for multiple consumers/multiple producers and should test with such setup. I agree with you, but it hurts my heart to deliver a patch that cause perf regressions without providing a huge benefit ie proven in other cases... Thank guys, Franz Il giorno mer 20 mar 2019 alle ore 13:34 ha scritto: > So Franz. > > > > > If youre talking about the solution that you regressed just before we > released. Then we did test it in our real testing env. I didnt notice and > negative impact for our use cases. > > > > > > > Regards to code, i actually thought what you had it made code cleaner. > > > > > Im not sure what use case your concerned about with your change, generally > i expect brokers to have more than just one queue and just one > producer/consumer in real world use cases. I think having just one > producer, one consumer and one queue on a whole broker is very academic and > not a typical real world case. I think we engineer for multiple > consumers/multiple producers and should test with such setup. > > > > > Get Outlook for Android > > > > > > > > On Wed, Mar 20, 2019 at 1:15 PM +0100, "Francesco Nigro" < > nigro.fra@gmail.com> wrote: > > > > > > > > > > > > That being said > is there an actual real world throughput issue here? > > Yes and no: it's a chance to improve things, especially for cloud uses: it > is a fact that now that Specter and Meltdown are there we don't want to > waste CPU time > sitting idle/on contention if isn't needed and as I've said "is a giant > lock on any task submitted". > IMO having a talk on how to improve it is not over-engineering, but just > engineering, given that scaling non-persistent messages (or persistent with > very fast disks) > is something that we expect from a broker: from a commercial point of view > is nice that we could scale by adding brokers, but if you can save 2 > machines to get > the same throughput I think is a nice improvement for (m)any users. > > > , I don't > know that I see much value in over engineering and micro managing this > stuff unless there's a real world measurable benefit to be gained vs just > theoretical benchmarks as it's just going to make things harder to maintain > and mistakes easier to make in the future. > > Cassandra from Datastax has gained about 2X throughput by solving this, but > it can be said that's a "different scenario" too: as an engineer I can say > no, is not. > I've "recently" addressed with the client team a similar "issue" on > qpid-jms, getting near 2X throughput (nudge nudge Robbie Gemmel/Tim Bish). > And this "issue" (actually, a "chance to improve things") has been well > hidden altough in front of anyone from a long time: > https://issues.apache.org/jira/browse/QPIDJMS-396. > > The reason why I've written on the dev list is to understand if anyone has > had the chance to measure in a real load scenario something like this. > > > Il giorno mer 20 mar 2019 alle ore 12:07 Christopher Shannon < > christopher.l.shannon@gmail.com> ha scritto: > > > I don't think sacrificing low utilization is a good idea. That being > said > > is there an actual real world throughput issue here? In general, I don't > > know that I see much value in over engineering and micro managing this > > stuff unless there's a real world measurable benefit to be gained vs just > > theoretical benchmarks as it's just going to make things harder to > maintain > > and mistakes easier to make in the future. > > > > On Wed, Mar 20, 2019 at 6:51 AM Francesco Nigro > > wrote: > > > > > HI folks, > > > > > > I'm writing here to share some thoughts related to the Artemis > threading > > > model and how it affects broker scalability. > > > > > > Currently (on 2.7.0) we relies on a shared thread pool ie > > > ActiveMQThreadPoolExecutor backed by a LinkedBlockingQueue-ish queue to > > > process tasks. > > > Thanks to the the Actor abstraction we use a lock-free queue to > serialize > > > tasks (or items), > > > processing them in batch in the shared thread pool, awaking a consumer > > > thread only if needed (the logic is contained in ProcessorBase). > > > The awaking operation (ie ProcessorBase::onAddedTaskIfNotRunning) will > > > execute on the shared thread pool a specific task to drain and execute > a > > > batch of tasks only if necessary, not on every added task/item. > > > > > > Looking at the contention graphs of the broker (ie the bar width are > the > > > nanoseconds before entering into a lock) is quite clear the limitation > of > > > the current implementation: > > > > > > [image: image.png] > > > > > > In violet are shown the offer and poll operations on the > > > LinkedBlockingQueue of the shared thread pool, happening from any > thread > > of > > > the pool (the thread is the base of each bar, in red). > > > The LinkedBlockingQueue indeed has a ReentrantLock to protect any > > > operation on the linked q and is clear that having a giant lock in > front > > of > > > high contention point won't scale. > > > > > > The above graph has been obtained with a single producer/single > > > consumer/single queue/not-persistent run, but I don't have enough > > resources > > > to check what could happen with more and more > producers/consumers/queues. > > > The critical part is the offering/polling of tasks on the shared thread > > > pool and in theory a maxed-out broker shouldn't have many idle threads > to > > > be awaken, but given that more producers/consumers/queues means many > > > different Actors, in order to guarantee each actor tasks to be > executed, > > > the shared thread pool will need to process many unnecessary "awake" > > tasks, > > > creating lot of contention on the blocking linked q, slowing down the > > > entire broker. > > > > > > In the past I've tried to replace the current shared thread pool > > > implementation with a ForkJoinPool or (the most recent attempt) by > using > > a > > > lock-free q instead of BlockingLinkedQueue, with no success ( > > > https://github.com/apache/activemq-artemis/pull/2582). > > > > > > Below the contention graph using a lock-free q in the shared thread > pool: > > > > > > [image: image.png] > > > > > > In violet now we have QueueImpl::deliver and RefsOperation::afterCommit > > > that are contending QueueImpl lock, but the numbers for each bar are > very > > > different: in the previous graph the contention on the shared thread > pool > > > lock is of 600 ns, while here is 20-80 ns and it can scale with number > of > > > queues, while the previous version not. > > > > > > All green right? So, why I've reverted the lock-free thread pool? > > > > > > Because with a low utilization of the broker (ie 1 producer/1 > consumer/1 > > > queue) the latencies and throughput were actually worse: cpu > utilization > > > graphs were showing that ProcessorBase::onAddedTaskIfNotRunning was > > > spending most of its time by awaking the shared thread pool. The same > was > > > happening with a ForkJoin pool, sadly. > > > It seems (and it is just a guess) that, given that tasks get consumed > > > faster (there is no lock preventing them to get polled and executed), > the > > > thread pool is getting idle sooner (the default thread pool size is of > 30 > > > and I have a machine with just 8 real cores), forcing any new task > > > submission to awake any of the thread pool to process incoming tasks. > > > > > > What are your thoughts on this? > > > I don't want to trade so much the "low utilization" performance for the > > > scaling TBH, that's why I've preferred to revert the change. > > > Note that other applications with scalability needs (eg Cassandra) have > > > changed their shared pool approach based on SEDA to a thread-per-pool > > > architecture for this same reason. > > > > > > Cheers, > > > Franz > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ------=_Part_8469_1428525355.1553088636437--