From dev-return-110987-archive-asf-public=cust-asf.ponee.io@cloudstack.apache.org Tue Mar 13 03:31:40 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 651EC18064D for ; Tue, 13 Mar 2018 03:31:39 +0100 (CET) Received: (qmail 97653 invoked by uid 500); 13 Mar 2018 02:31:37 -0000 Mailing-List: contact dev-help@cloudstack.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@cloudstack.apache.org Delivered-To: mailing list dev@cloudstack.apache.org Received: (qmail 97631 invoked by uid 99); 13 Mar 2018 02:31:37 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Mar 2018 02:31:36 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 911141806E6 for ; Tue, 13 Mar 2018 02:31:36 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.879 X-Spam-Level: * X-Spam-Status: No, score=1.879 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (1024-bit key) header.d=cloudops.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id XBO8_CgVE7f0 for ; Tue, 13 Mar 2018 02:31:34 +0000 (UTC) Received: from mail-ua0-f178.google.com (mail-ua0-f178.google.com [209.85.217.178]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id 3A3135F393 for ; Tue, 13 Mar 2018 02:31:34 +0000 (UTC) Received: by mail-ua0-f178.google.com with SMTP id f6so8251544ual.0 for ; Mon, 12 Mar 2018 19:31:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cloudops.com; s=google; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=oTN4zGuzYJfy67W3wfkrSrUC+7sZVEoTRGxp2Zm11hs=; b=RlhTstzlSPvQjIhRQMjE3oj0jbR1FC6VSv2kg3f8JkOr8bSwe3sNOifbvZYUseoC8s noscWkbn4lk5Y6rrw1C0tBCO8P8O5Um+sdDXggYb4TMDOOeDdZzyPZo4yWKsnFET22kQ 6rIio48RhC+7KHVsgGvIUT95ayoT3MekqXVQo= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=oTN4zGuzYJfy67W3wfkrSrUC+7sZVEoTRGxp2Zm11hs=; b=eCrATqnOQRybgW0yB4qDEs8hT+Am4uT/Zrzafl4qwQ0fLET/kiYSxHqKCwKmfFidwW Ot3P+cg2qduK1UioNDpIiN675SGQsuK/f3ziBYuqPCa15gPz6Z0hj3xFCr3tFWMt8wqX eLLbVItt+WuIMSWpW5Kqb+vFjzHFG/p55lsIuto5Sb2+TXPkvd5EP1sCIUtJAb65gmuT ID+1SJ3hiOXNDs2LA2lnEok7XrUspFz+NIAwCOtZRPlVQoHt7xWCuV5eISLrsK/zzmYR zUOnEcWdYTm1FDgm3hdgNqaT7CCLLGnjbBz1xmnFlH8Yhxbc67logWTJ1QM9TIs/tqsN QfnQ== X-Gm-Message-State: AElRT7GZ4j4i3mvJxMhTxwQQUOsXflhGK/j2h4m8fk/09J81MDIVA1xf uyRor6X/T2OisVJxMfSaBGJUuGzoMUrBtk50/K6UBgfnFA== X-Google-Smtp-Source: AG47ELvd2layS3LmsOnO5Bi239B/E6JXnoYPLocyxX7xOkxW6q8dppIqCJ7iOOdG1OZxXCjge9WguKlnTEMp7jgVV+E= X-Received: by 10.159.53.141 with SMTP id t13mr7139381uad.4.1520908293652; Mon, 12 Mar 2018 19:31:33 -0700 (PDT) MIME-Version: 1.0 Received: by 10.176.0.210 with HTTP; Mon, 12 Mar 2018 19:31:03 -0700 (PDT) In-Reply-To: References: From: Khosrow Moossavi Date: Mon, 12 Mar 2018 22:31:03 -0400 Message-ID: Subject: Re: [DISCUSS] CloudStack Connection Pools To: dev@cloudstack.apache.org Content-Type: multipart/alternative; boundary="001a113dfd309454d40567420e4c" --001a113dfd309454d40567420e4c Content-Type: text/plain; charset="UTF-8" Hi Nicolas From my past experiences, I prefer 1) HikariCP 2) Tomcat Pool 3) C3P0 4) DBCP in that order. Although I don't have any benchmark of my own to provide, and the ones you mentioned are really informative anyway. To me the broader subject is the _one_ who uses the pool, I mean if the transactions are handled in a faster way and released sooner and with shorter locks, generally speaking if it's more efficient, I don't think from ACS point of view there won't be much difference between the above mentioned options. On the same subject, it might be more interesting to use Spring Boot in general and Spring Boot Data in particular rather than only changing the CP functionality, and slowly refactor/retire the DAO layer in favor of Spring Boot equivalent implementation. Khosrow Moossavi CloudOps On Mon, Mar 12, 2018 at 9:32 PM, Nicolas Vazquez < Nicolas.Vazquez@shapeblue.com> wrote: > Hi all, > > > I would like to introduce a topic for discussion, regarding DB connection > pools used in CloudStack, currently Apache Commons DBCP 1.4 ( > http://commons.apache.org/) is used. I've been investigating this topic > as we are having complains of random issues on MySQL connection pool on > large environments. Please let me know if this topic has already been > discussed before. > > > First of all, DBCP 1.4 has been released on 2010 ( > https://commons.apache.org/proper/commons-dbcp/changes-report.html), and > no minor/patch version has been released since then. It seems to work in > high performance with relatively low traffic and low load applications. > However, it is single threaded, and in order to be thread-safe, the entire > pool needs to be locked. It is also reported that an CPU and concurrent > threads increases, the performance gets affected. This is a serious issue > on highly concurrent systems, such as CloudStack. > > > I've been investigating some options to replace it: > - The first option can be upgrading to version 2.x. Issues on performance > and concurrency could be solved using this version. > - Tomcat JDBC Connection Pool. Please check: https://tomcat.apache.org/ > tomcat-7.0-doc/jdbc-pool.html. > > - Other replacement options found: BoneCP, C3P0, HikariCP > > > Given these options, I've been looking for benchmarks to compare them (*). > Looks like HikariCP (http://brettwooldridge.github.io/HikariCP/) could be > the best replacement, improving performance and stability. Another good > replacement option could be Tomcat. > > > I've been also examining the codebase, data source initialization is done > on TransactionLegacy class under the cloud-framework-db project. > Replacement work should be done on this class. Instead of pure replacement, > a global setting can be introduced to make the admins able to select which > connection pool to use. > > What do you think? Any possitive/negative feedback is welcome as well as > new ideas. As mentioned before, I don't know if it has been discussed > before, sorry in advance if it has. > > Kind regards, > Nicolas > > (*) Links to benchmarks and comparissons: > https://www.wix.engineering/single-post/how-many-threads- > does-it-take-to-fill-a-pool > https://www.wix.engineering/single-post/how-does-hikaricp- > compare-to-other-connection-pools > compare-to-other-connection-pools>https://beansroasted. > wordpress.com/2017/07/29/connection-pool-analysis/ > https://beansroasted.wordpress.com/tag/connection-pool-comparison/ > > https://github.com/brettwooldridge/HikariCP/wiki/%22My-benchmark- > doesn't-show-a-difference.%22 > http://www.trustiv.co.uk/2014/06/battle-connection-pools > > Nicolas.Vazquez@shapeblue.com > www.shapeblue.com > , > @shapeblue > > > > --001a113dfd309454d40567420e4c--