From user-return-30890-archive-asf-public=cust-asf.ponee.io@ignite.apache.org Thu Jun 25 10:04:17 2020 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id 7283A180181 for ; Thu, 25 Jun 2020 12:04:17 +0200 (CEST) Received: (qmail 19158 invoked by uid 500); 25 Jun 2020 10:04:16 -0000 Mailing-List: contact user-help@ignite.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@ignite.apache.org Delivered-To: mailing list user@ignite.apache.org Received: (qmail 19148 invoked by uid 99); 25 Jun 2020 10:04:16 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 25 Jun 2020 10:04:16 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id B92961A41A4 for ; Thu, 25 Jun 2020 10:04:15 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 0 X-Spam-Level: X-Spam-Status: No, score=0 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd2-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-ec2-va.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id wxBxialrfLY8 for ; Thu, 25 Jun 2020 10:04:13 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=209.85.221.193; helo=mail-vk1-f193.google.com; envelope-from=ilya.kasnacheev@gmail.com; receiver= Received: from mail-vk1-f193.google.com (mail-vk1-f193.google.com [209.85.221.193]) by mx1-ec2-va.apache.org (ASF Mail Server at mx1-ec2-va.apache.org) with ESMTPS id 5AC29BB87E for ; Thu, 25 Jun 2020 10:04:13 +0000 (UTC) Received: by mail-vk1-f193.google.com with SMTP id s192so1271989vkh.3 for ; Thu, 25 Jun 2020 03:04:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=nlUPQfJJ6ZReaok/fyTcp+7/stEuVROvQ3/UPXQ1Srw=; b=Mfl+GM0hnyVbgVYjDclLlhrELoGurS9yA5VZzJJD3NgIqGnaSMu58HEvUT/MjsNFId vpHo1cWi/gsdCgJAPOjKreqgRxjvVlBH2ZzkA2fK4Oa5yOs7zOAEVmwIKzl+tsuMRNPZ WJCo/8dLQ8ii5S+V5vX5047/ayDeFE9ao6K7ObFYg0pEnGHBGYjerXYOTlxRPd0gSTVn NV26vSlkSZOtLPx/9/n0GGcSX8B2/yVXTTYpG3RCmhs3nixmjTBeUX1J10g9G1jCLypD ZVveYE3fmjj80/2r3e27JjFy6uJuA837+evWicXldyVDeXYU+pAzmdhdnyHHC3INl6n7 kHkg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=nlUPQfJJ6ZReaok/fyTcp+7/stEuVROvQ3/UPXQ1Srw=; b=DFmZLOWa9oLeb766uJSPxhyKHyK5vuw/ikhL0GPK1+1CHca/gNlKvVJ53aS9oqWm0o U/fMu36V677RMZrXCkC45wD3PXnwx9PVQj1M5Ur4sJDJ8LnNmwHWp2JUbaX3HSPJJI5u HfzMGTWi6W5RlIY5r/ODRVgftxtLS/jEVpbY6iFCT2pc6+uUhavikeJaHifevf4aSktQ ECNpPJYxpVWvCSepRYQ6Aqe/Ec9M/f8XH3xHquIiXPA0nfZGojDfhXGz61Swc8DT7u3v b+02r5erscbsg5Kobkds0VXKaiUgE/lUhpyb/J1MQpnDaVpun+t5s8l3y+lW5QeVTCAj THoQ== X-Gm-Message-State: AOAM532nrAfKlWkVefrX3Dv8dP7Kyy3CrwejhGrFuK7uFikhgE/AdcVv ks/D0jKZHi7DY627mcd/IzaPfz2jbTk0HHsla67PAkkF X-Google-Smtp-Source: ABdhPJyg0US9NITOlCmN1Tk+LHjzht0u/c/KdOdYy+w4fZgz503wmJvGXBAzz3vXrYBv3dA/zHa8nX6iboRUOMIzk1Y= X-Received: by 2002:ac5:c76e:: with SMTP id c14mr2152192vkn.60.1593079452557; Thu, 25 Jun 2020 03:04:12 -0700 (PDT) MIME-Version: 1.0 References: <1592925901036-0.post@n6.nabble.com> In-Reply-To: <1592925901036-0.post@n6.nabble.com> From: Ilya Kasnacheev Date: Thu, 25 Jun 2020 13:04:01 +0300 Message-ID: Subject: Re: Ignite Load Performance when memory exhausted To: user@ignite.apache.org Content-Type: multipart/alternative; boundary="000000000000de658605a8e5b6b4" --000000000000de658605a8e5b6b4 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hello! Until you got this message, you did not need to catch up with disk (except for WAL writing, maybe), but once you see it, good days are over and you need to do checkpoints with all pages which were updated recently. There are things to tune here, but don't expect miracles. https://apacheignite.readme.io/docs/durable-memory-tuning Regards, --=20 Ilya Kasnacheev =D0=B2=D1=82, 23 =D0=B8=D1=8E=D0=BD. 2020 =D0=B3. =D0=B2 18:25, njcstreet <= njcstreet@gmail.com>: > Hi, > > I am evaluating Apache Ignite for a use case where I know we may not have > enough RAM to store the entire database. For example we might store 30 da= ys > of data, but only have enough memory for 15, and in fact most of the > queries > will be against the last 5 days of data. I am doing some tests loading th= e > 30 days of data (roughly 2 billion objects) into a test environment. > > > > =C2=B7 The test environment has 6 servers each with 128GB RAM, wh= ich I > am > allocating roughly 70% to Ignite to allow the OS enough memory. > > =C2=B7 This is using Native Persistence in Ignite 2.8.1. > > =C2=B7 The disks I am using are not as fast as production would b= e (they > are not SSD) but they are still reasonably fast. > > =C2=B7 The object I am storing is reasonably simple, it has 25 > properties > which are integers (2 of these have indexes on) and 4 which are floats, b= ut > there are a lot of them (roughly 2 billion). > > =C2=B7 The cache the data is being loaded into is partitioned. > > =C2=B7 Data is loaded using IDataStreamer =E2=80=93 there is a si= ngle loader > process which has 10 threads reading files, then submitting into a single > DataStreamer. > > > > What I am observing is that: > > > > =C2=B7 Whilst the default region has RAM available, the inserts a= re > quite > quick =E2=80=93 even though the data is being written to disk (the WAL is > enabled). > I get on average 150K objects inserted per second. > > =C2=B7 Once the default region runs out of RAM, I see an error me= ssage > > =E2=80=9CWARN|org.apache.ignite.internal.processors.cache.persistence.pag= emem.PageMemoryImpl|Page > replacements started, pages will be rotated with disk, this will affect > storage performance (consider increasing > DataRegionConfiguration#setMaxSize).=E2=80=9D > > =C2=B7 After this, inserts are much slower, going down to about 1= 9K > objects per second. > > > > Now I kind of understand what is happening here is that for every new pag= e > in memory that needs to be allocated, Ignite is having to purge another > page > from memory (this is implied by the error message). But what I don=E2=80= =99t > understand is why the performance impact is so great (150K objects / sec > when memory is available vs 19K per sec when memory is exhausted). Before > and after the region ran out of RAM, the data was being written to disk. > After the region ran out of RAM, I would expect the only difference to be > that it needs to purge some data from RAM to make space for the data that > is > loading =E2=80=93 I wouldn=E2=80=99t have thought there should be any mor= e disk activity > since through Ignite Persistence the data was already being written to > disk. > Maybe I have something mis-configured, or I have misunderstood something? > > > > -- > Sent from: http://apache-ignite-users.70518.x6.nabble.com/ > --000000000000de658605a8e5b6b4 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hello!

Until you got this message, you = did not need to catch=C2=A0up with disk (except for WAL writing, maybe), bu= t once you see it, good days are over and you need to do checkpoints with a= ll pages which were updated recently.

There are th= ings to tune here, but don't expect miracles.


=
Regards,
--
Ilya Kasnacheev


=D0=B2=D1=82, 23 =D0=B8=D1=8E=D0=BD. 2020 =D0=B3. = =D0=B2 18:25, njcstreet <njcstree= t@gmail.com>:
Hi,

I am evaluating Apache Ignite for a use case where I know we may not have enough RAM to store the entire database. For example we might store 30 days=
of data, but only have enough memory for 15, and in fact most of the querie= s
will be against the last 5 days of data. I am doing some tests loading the<= br> 30 days of data (roughly 2 billion objects) into a test environment.



=C2=B7=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0The test environment has 6 servers = each with 128GB RAM, which I am
allocating roughly 70% to Ignite to allow the OS enough memory.

=C2=B7=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0This is using Native Persistence in= Ignite 2.8.1.

=C2=B7=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0The disks I am using are not as fas= t as production would be (they
are not SSD) but they are still reasonably fast.

=C2=B7=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0The object I am storing is reasonab= ly simple, it has 25 properties
which are integers (2 of these have indexes on) and 4 which are floats, but=
there are a lot of them (roughly 2 billion).

=C2=B7=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0The cache the data is being loaded = into is partitioned.

=C2=B7=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Data is loaded using IDataStreamer = =E2=80=93 there is a single loader
process which has 10 threads reading files, then submitting into a single DataStreamer.



What I am observing is that:



=C2=B7=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Whilst the default region has RAM a= vailable, the inserts are quite
quick =E2=80=93 even though the data is being written to disk (the WAL is e= nabled).
I get on average 150K objects inserted per second.

=C2=B7=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Once the default region runs out of= RAM, I see an error message
=E2=80=9CWARN|org.apache.ignite.internal.processors.cache.persistence.pagem= em.PageMemoryImpl|Page
replacements started, pages will be rotated with disk, this will affect
storage performance (consider increasing
DataRegionConfiguration#setMaxSize).=E2=80=9D

=C2=B7=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0After this, inserts are much slower= , going down to about 19K
objects per second.



Now I kind of understand what is happening here is that for every new page<= br> in memory that needs to be allocated, Ignite is having to purge another pag= e
from memory (this is implied by the error message). But what I don=E2=80=99= t
understand is why the performance impact is so great (150K objects / sec when memory is available vs 19K per sec when memory is exhausted). Before and after the region ran out of RAM, the data was being written to disk. After the region ran out of RAM, I would expect the only difference to be that it needs to purge some data from RAM to make space for the data that i= s
loading =E2=80=93 I wouldn=E2=80=99t have thought there should be any more = disk activity
since through Ignite Persistence the data was already being written to disk= .
Maybe I have something mis-configured, or I have misunderstood something?


--
Sent from: http://apache-ignite-users.70518.x6.nabbl= e.com/
--000000000000de658605a8e5b6b4--