From dev-return-45704-archive-asf-public=cust-asf.ponee.io@ignite.apache.org Mon Apr 15 07:28:46 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id 84A3018064C for ; Mon, 15 Apr 2019 09:28:46 +0200 (CEST) Received: (qmail 23360 invoked by uid 500); 15 Apr 2019 07:28:45 -0000 Mailing-List: contact dev-help@ignite.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@ignite.apache.org Delivered-To: mailing list dev@ignite.apache.org Received: (qmail 23348 invoked by uid 99); 15 Apr 2019 07:28:45 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 15 Apr 2019 07:28:45 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id D689FC2C67 for ; Mon, 15 Apr 2019 07:28:44 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.799 X-Spam-Level: * X-Spam-Status: No, score=1.799 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id g03oWLcTZEyd for ; Mon, 15 Apr 2019 07:28:43 +0000 (UTC) Received: from mail-ua1-f47.google.com (mail-ua1-f47.google.com [209.85.222.47]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id DE4505FD84 for ; Mon, 15 Apr 2019 07:18:40 +0000 (UTC) Received: by mail-ua1-f47.google.com with SMTP id h4so5167769uaj.9 for ; Mon, 15 Apr 2019 00:18:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=/iIerSFc7e6je48if+/nS6seiQIVAUUWnQtafuX95+w=; b=GYuhPxHHD+BHZvmOVJPXlvqbpFZH/HhG+Zrn0ZyyFTDkvfxhxCri5EQQTcfIHForXq UjL8qXMKBPiZakyojMAUNVVteMNN8pvVoPcHlNNlNr5gC3f1Y18CisAArf/b8mxYvB7N FAx/JYPORWaKmdw5fHTuNrbpL4zDa/LWzdp3rlA9F/Q1lDh3N5/h428nKTBuuAfrlT5l 42wA7GYeaT6kFPmb8cxvNdnjSWAsNHYPt2skf54fbc9fOwS5Tgz7PQur8G/1U6nMFDvd VXYaZZzobqBDoYRBu4ZlsxC85lzWv7KgTYwgvOzBOIbgYYxJELA6XnXh9LH8TYOuiXLa mliA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=/iIerSFc7e6je48if+/nS6seiQIVAUUWnQtafuX95+w=; b=bSMVLAdpNsjUjRZasKcpJz7Zj0RDmzKv4KgF3tNzM1Fg5bctyiTK7YGE7k0iIgFOC2 R2udk37OhwUhK3YnOu2PxRskcbaE8PBrF+1AXLDno16S+8yzBmmsEq3wPgEI1Uyg8VmO QBpLRxWW3V7PnrJ1cZH+8jHFBMdci5XPofk9axo6CVqWh5erv7wHuI5ZBim93CJncbxg M/npC85b47RZIiT0soT5PyZ7DcAtvtFU0cpUqx49Glc/RwC7NsN9KqXIuI/C90Wg9mN8 39PQ7IaQNivQ8dtYKrq9FulFvj0/AjKwX1SfbteB8P+VSGXZ1pt7KYp2eSZu6CEvVxWv oc1Q== X-Gm-Message-State: APjAAAWf6zkcvO+0t9oA9Xzq8V7L/htcT2ZUia9+cGeZ2B4oshiyH7V2 wZi5TfrUD1p0E9/eJpSKNEs1RRwKJsWOUBnUNk/hjw== X-Google-Smtp-Source: APXvYqzG0q+JpWbmusg6BeHoy74XsudHGX40ujXFHVowIWijNQ3Z8rKyk6wlJtzh6QIW2KQChb2Nkc8Yf70Jpy1oIwo= X-Received: by 2002:ab0:26c1:: with SMTP id b1mr37413507uap.92.1555312720174; Mon, 15 Apr 2019 00:18:40 -0700 (PDT) MIME-Version: 1.0 References: <7355b4ee6499b57e755b6fa4367ab016bc799785.camel@gmail.com> In-Reply-To: <7355b4ee6499b57e755b6fa4367ab016bc799785.camel@gmail.com> From: Ilya Kasnacheev Date: Mon, 15 Apr 2019 10:18:28 +0300 Message-ID: Subject: Re: Impossible memory region created in TC test To: dev@ignite.apache.org Content-Type: multipart/alternative; boundary="0000000000003363a405868c76ed" --0000000000003363a405868c76ed Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hello! Ignite does not touch every page of data region, and until you touch a page OS will not allocate any specific physical RAM to the virtual RAM address of that page. Moreover, AFAIK Ignite will not even allocate all the memory permitted by data region until it is needed. It will allocate memory in chunks, which means your system will slow to grind trying to find RAM for the next chunk as you try to load data into such node. If you would use Pre-Touch feature which was suggested in this developer list a few months ago, you will see it fail explicitly. Regards, --=20 Ilya Kasnacheev =D0=BF=D0=BD, 15 =D0=B0=D0=BF=D1=80. 2019 =D0=B3. =D0=B2 09:49, Nikolay Izh= ikov : > Hello, Igniters. > > Why test on TC can allocate Data Region bigger than the amount of > available RAM? > How can I disable this feature in a specific test? > > "memoryAllocated=3D1024.0 GiB ... required=3D1093413MB, available=3D96675= MB" > > I have a test [1] that try to create Data Region of 1024 Gib size. > I want to check OOM exception that should occur while this DataRegion > starting. > > Locally, I have it. > > But, On TC that DataRegion successfully created [2] > I see following log messages: > > ``` > [2019-04-13 21:31:30,426][INFO > ][exchange-worker-#91818%test-server%][PageMemoryImpl] Started page memor= y > [memoryAllocated=3D1024.0 GiB, pages=3D260204016, tableSize=3D19.8 GiB, > checkpointBuffer=3D2.0 GiB] > .... > [2019-04-13 21:31:18,004][WARN > ][test-runner-#91737%pagemem.PageMemoryLazyAllocationWithPDSTest%][Ignite= Kernal%test-server] > Nodes started on local machine require more than 80% of physical RAM what > can lead to significant slowdown due to swapping (please decrease JVM hea= p > size, data region size or checkpoint buffer size) [required=3D1093413MB, > available=3D96675MB] > ``` > > [1] > https://github.com/apache/ignite/pull/6388/files#diff-754f5ea2f3579b81c50= 662f3d20c9126R138 > > [2] > https://ci.ignite.apache.org/viewLog.html?buildId=3D3591587&buildTypeId= =3DIgniteTests24Java8_RunAll#testNameId-7083016202406489951 > --0000000000003363a405868c76ed--