From dev-return-32493-archive-asf-public=cust-asf.ponee.io@ignite.apache.org Fri Mar 23 14:38:00 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 1B02218064D for ; Fri, 23 Mar 2018 14:37:59 +0100 (CET) Received: (qmail 69435 invoked by uid 500); 23 Mar 2018 13:37:59 -0000 Mailing-List: contact dev-help@ignite.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@ignite.apache.org Delivered-To: mailing list dev@ignite.apache.org Received: (qmail 69423 invoked by uid 99); 23 Mar 2018 13:37:58 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Mar 2018 13:37:58 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id E09D7182DD7 for ; Fri, 23 Mar 2018 13:37:57 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.979 X-Spam-Level: * X-Spam-Status: No, score=1.979 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gridgain-com.20150623.gappssmtp.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id OxH1Y8Pg2ms5 for ; Fri, 23 Mar 2018 13:37:55 +0000 (UTC) Received: from mail-ua0-f182.google.com (mail-ua0-f182.google.com [209.85.217.182]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id B56705F36A for ; Fri, 23 Mar 2018 13:37:54 +0000 (UTC) Received: by mail-ua0-f182.google.com with SMTP id q12so7774858uae.4 for ; Fri, 23 Mar 2018 06:37:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gridgain-com.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=pL2uMmrZj2SglrsUqCacoGrHZvRTV8z6cOTp4yyPy4s=; b=hEl+lycUlYIHMJLryC2hVRlmeN3g2UwPtLCmA1JhltQdGKP35UpbJh8zl4XjJfAsnA LOTnRIX6oin9aJFb89XQzjjktG3zAKuEH7QCh/i0WVQUM7QHtN+4dnbFNW90L5LlEM8f cIzRqJzVSsMTLWlP4Oyu+br1K8dkBbTjWIw2YvWnuzRzWS0UycK3xPYTYR1SlMHVUnKe Lq6iRcoMPkHYj3nNtu5W0pl0cYhE2OsElFy+SUVE6zPTxA61GA/Dl38/EbSsyEz/z0H2 Xl3dHFbtvMOAh/Ug42WgbPuudpepwN2I3dMMK4LURUeWUC8XI6sIaZTLQmNzVLgHHWgP 45Iw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=pL2uMmrZj2SglrsUqCacoGrHZvRTV8z6cOTp4yyPy4s=; b=BEsEJBENrTxjN7zzHseg6NMS8BOVXcy+p+omNz3HL1W8iLc7CeeVLWLHGk1d8cKRNP TuM1TP6ZxJSgLVkYEf+frHFkUxSeAi6QPio+kjHbVPdEB1Y+8ltz6YSrgPclhvIVQBM2 xgLou5bQ1LF3zu80IcRrwVhxndZtI0PX+RJ0pqUpUWHRcd3uHpfH/cmYxUopbfk+TTnv DCXEm5INWc3WFQxraW8Jjh4/hUwxIEgFv0TVnPX9mfnjVV4ItKrnFOiix0lto5qy0ukU Wk7aBDvmlmSLmzkDyCyoihOT99JsV02cssGXrEGquJmkGAQQFCnryYKbsBpfpjRm9d+6 8v1Q== X-Gm-Message-State: AElRT7Gw3xwjIEhR59d1yASr3N0Ra7htjZQvo0HL8V1s/x5DJphMPnbF QO7LWFPSojZgF9IOong7z4jadSbbqTaSDz9246nfIkHt X-Google-Smtp-Source: AG47ELsNwoYAIEYa9Zg4V7qhPSKMd++cnadwb2zLJ/Ihk5IrnFhf1Al+JIfIb/LwzMkwKkM91VaWiCnl23Ty2Z/UOEI= X-Received: by 10.176.81.116 with SMTP id f49mr19296493uaa.156.1521812267429; Fri, 23 Mar 2018 06:37:47 -0700 (PDT) MIME-Version: 1.0 Received: by 10.103.63.139 with HTTP; Fri, 23 Mar 2018 06:37:46 -0700 (PDT) In-Reply-To: References: From: Ilya Lantukh Date: Fri, 23 Mar 2018 16:37:46 +0300 Message-ID: Subject: Re: Rebalancing - how to make it faster To: dev@ignite.apache.org Content-Type: multipart/alternative; boundary="94eb2c1921fc9d963a0568148716" --94eb2c1921fc9d963a0568148716 Content-Type: text/plain; charset="UTF-8" Denis, > - Don't you want to aggregate the tickets under an IEP? Yes, I think so. > - Does it mean we're going to update our B+Tree implementation? Any ideas how risky it is? One of tickets that I created ( https://issues.apache.org/jira/browse/IGNITE-7935) involves B+Tree modification, but I am not planning to do it in the nearest future. It shouldn't affect existing tree operations, only introduce new ones (putAll, invokeAll, removeAll). > - Any chance you had a prototype that shows performance optimizations the approach you are suggesting to take? I have a prototype for simplest improvements (https://issues.apache.org/ jira/browse/IGNITE-8019 & https://issues.apache.org/jira/browse/IGNITE-8018) - together they increase throughput by 5-20%, depending on configuration and environment. Also, I've tested different WAL modes - switching from LOG_ONLY to NONE gives over 100% boost - this is what I expect from https://issues.apache.org/jira/browse/IGNITE-8017. On Thu, Mar 22, 2018 at 9:48 PM, Denis Magda wrote: > Ilya, > > That's outstanding research and summary. Thanks for spending your time on > this. > > Not sure I have enough expertise to challenge your approach, but it sounds > 100% reasonable to me. As side notes: > > - Don't you want to aggregate the tickets under an IEP? > - Does it mean we're going to update our B+Tree implementation? Any > ideas how risky it is? > - Any chance you had a prototype that shows performance optimizations of > the approach you are suggesting to take? > > -- > Denis > > On Thu, Mar 22, 2018 at 8:38 AM, Ilya Lantukh > wrote: > > > Igniters, > > > > I've spent some time analyzing performance of rebalancing process. The > > initial goal was to understand, what limits it's throughput, because it > is > > significantly slower than network and storage device can theoretically > > handle. > > > > Turns out, our current implementation has a number of issues caused by a > > single fundamental problem. > > > > During rebalance data is sent in batches called > > GridDhtPartitionSupplyMessages. Batch size is configurable, default > value > > is 512KB, which could mean thousands of key-value pairs. However, we > don't > > take any advantage over this fact and process each entry independently: > > - checkpointReadLock is acquired multiple times for every entry, leading > to > > unnecessary contention - this is clearly a bug; > > - for each entry we write (and fsync, if configuration assumes it) a > > separate WAL record - so, if batch contains N entries, we might end up > > doing N fsyncs; > > - adding every entry into CacheDataStore also happens completely > > independently. It means, we will traverse and modify each index tree N > > times, we will allocate space in FreeList N times and we will have to > > additionally store in WAL O(N*log(N)) page delta records. > > > > I've created a few tickets in JIRA with very different levels of scale > and > > complexity. > > > > Ways to reduce impact of independent processing: > > - https://issues.apache.org/jira/browse/IGNITE-8019 - aforementioned > bug, > > causing contention on checkpointReadLock; > > - https://issues.apache.org/jira/browse/IGNITE-8018 - inefficiency in > > GridCacheMapEntry implementation; > > - https://issues.apache.org/jira/browse/IGNITE-8017 - automatically > > disable > > WAL during preloading. > > > > Ways to solve problem on more global level: > > - https://issues.apache.org/jira/browse/IGNITE-7935 - a ticket to > > introduce > > batch modification; > > - https://issues.apache.org/jira/browse/IGNITE-8020 - complete redesign > of > > rebalancing process for persistent caches, based on file transfer. > > > > Everyone is welcome to criticize above ideas, suggest new ones or > > participate in implementation. > > > > -- > > Best regards, > > Ilya > > > -- Best regards, Ilya --94eb2c1921fc9d963a0568148716--