From user-return-18768-archive-asf-public=cust-asf.ponee.io@ignite.apache.org Thu Apr 5 08:39:03 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 6DA1018063B for ; Thu, 5 Apr 2018 08:39:02 +0200 (CEST) Received: (qmail 53503 invoked by uid 500); 5 Apr 2018 06:39:01 -0000 Mailing-List: contact user-help@ignite.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@ignite.apache.org Delivered-To: mailing list user@ignite.apache.org Received: (qmail 53493 invoked by uid 99); 5 Apr 2018 06:39:01 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Apr 2018 06:39:01 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id A9DA71A060D for ; Thu, 5 Apr 2018 06:39:00 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.879 X-Spam-Level: ** X-Spam-Status: No, score=2.879 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_REPLY=1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd2-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id pf4sMb4wQAkq for ; Thu, 5 Apr 2018 06:38:59 +0000 (UTC) Received: from mail-it0-f43.google.com (mail-it0-f43.google.com [209.85.214.43]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id 7D9765F3BA for ; Thu, 5 Apr 2018 06:38:59 +0000 (UTC) Received: by mail-it0-f43.google.com with SMTP id 71-v6so2057722ith.2 for ; Wed, 04 Apr 2018 23:38:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=3dz4sbxsZ4dQLdftQ1hQa+Tq8MWgnSOKFU3mz5kirgI=; b=sgHDCVMqDJhgTXIeXdMTVOfSLxxXd88fLI6posam773hcbjcVS1FzFAvaQtvjNo8kY l8mLJTVI7GUXWYGTns82ib94gjUFIQBK/M2jJBDb19CY3QTWOWqJHbsPqtHs3SjgT3V2 M6rzPQyvHorLC1rN9AXs2l27NsYEM1hQDuiIhc/ZHnSSsx4UhgcvC9SLRdKxUl15W/y2 AeyWBYY/D1bjb3Nu+n7Ndwq+uaWwwXmdTr9wNRnqsYImZ6Kjm7zHoM7WvFf9+38X4agF W+FtH+re8buJ1iraqkUlMwHKiWDJSfhiem7hvLt8120k6z3e/LAFP6vEC1Ssu7WxQCbq Z+3Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=3dz4sbxsZ4dQLdftQ1hQa+Tq8MWgnSOKFU3mz5kirgI=; b=Ywkxp6h1iT1tcqV7imETw2G2K3EdRvywlmnHqLNWAAZmqrvIPyCkdItadSU1mc/dyx ftxphqBQ2cGYezdjzWzuWNLsaDstS1f5+PlvZ3i1D5ELdUh66+11LUfDMsqYZRFrGYrb Rolg/HMISmOJcXZyY/6xTJ6fJVZK2GVZAKjPV1D4FDGo9mLPww3og1aLupjtuYryUsPg Qekzhdv4Ebc3PWDGKUkeKL7dPrnZ5n4ehuz/i4A5abtxnSC1/UX4ANwV/cGMAytB/qgM CSYIqePYvTCPrHVtlr0MPe8dlnMJz7LZ1/+Etx7IP3bERWeNGr3gcYzSOBlxbEZjBoI5 3uFA== X-Gm-Message-State: ALQs6tCW/sx49JqapM3i7PJ1HB7D/Ou6f/cGCbXJS/2Xs8rbiWpj6Uep K4M9dRepPnLL0RCp+ERWLhxDW3bgiCN5nDW7zCISNw== X-Google-Smtp-Source: AIpwx49363TcCfI71ZPAymFwqHbqgqM4ugJv2Dloje/ciz3dif/FXBnTFjTit/IqiUOu3Jh/k9eELdfI44i3pjmEVS8= X-Received: by 2002:a24:6c0f:: with SMTP id w15-v6mr11699822itb.29.1522910332969; Wed, 04 Apr 2018 23:38:52 -0700 (PDT) MIME-Version: 1.0 Received: by 10.213.65.81 with HTTP; Wed, 4 Apr 2018 23:38:52 -0700 (PDT) In-Reply-To: <1522892664104-0.post@n6.nabble.com> References: <1522892664104-0.post@n6.nabble.com> From: Pavel Vinokurov Date: Thu, 5 Apr 2018 09:38:52 +0300 Message-ID: Subject: Re: Slow invoke call To: user@ignite.apache.org Content-Type: multipart/alternative; boundary="0000000000006c0ba505691431ec" --0000000000006c0ba505691431ec Content-Type: text/plain; charset="UTF-8" Hi Sam, When you store entries into the off-heap, any update operation requires copying value from the off-heap to the heap. Thus frequent updates of the "heavy" entry lead to significant overheads for copying data. Have you tested only with the heap memory? Thanks, Pavel 2018-04-05 4:44 GMT+03:00 javastuff.sam@gmail.com : > Hi, > > I have a usecase where we are storing byte array into one of the OFFHEAP > ATOMIC cache. Multiple threads keep on appending bytes to it using remote > operation (Entry processor / invoke call). > > Below is the logic for remote operation. > > if (mutableEntry.exists()) { > MyObject m = (MyObject)mutableEntry.getValue(); > byte[] bytes = new byte[m.getContent().length + > newContent.getContent().length]; > System.arraycopy(m.getContent(), 0, bytes, 0, > m.getContent().length); > System.arraycopy(newContent.getContent(), 0, bytes, > m.getContent().length,newContent.getContent().length); > m.setContent(bytes); > mutableEntry.setValue(m); > } else { > mutableEntry.setValue(newContent); > } > > We tested by adding about 25MB of content. > > What we have seen is - > 1. In 8 threads test we have seen average time for Invoke() is 36ms. > 2. Out of total time taken to execute Invoke(), 50% time taken by above > remote logic. Where does other 50% spend? Probably just to invoke the > method. > > What can we tune here? We are considering InvokeAll(), any more ideas? > > If we increase this to 250MB we have seen a very slow performance and after > hours it goes Out-of-memory, setOffHeapMaxMemory(0) does not help on 32GB > box. > > We are using Ignite 1.9, Java 1.8, Parallel GC and app JVM is 4GB. > Unfortunately, we can not upgrade Ignite version as of now. > > Thanks, > -Sam > > > > > -- > Sent from: http://apache-ignite-users.70518.x6.nabble.com/ > -- Regards Pavel Vinokurov --0000000000006c0ba505691431ec Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi Sam,

When you store entries int= o the off-heap, any update operation requires copying value from the off-he= ap to the heap.
Thus frequent updates of=C2=A0 the "heavy&qu= ot; entry=C2=A0 lead to significant overheads for copying data.
H= ave you tested only with the heap memory?

Th= anks,
Pavel

2018-04-05 4:44 GMT+03:00 javastuff.sam@gmail.com <javastuff.sam@gmail.com&= gt;:
Hi,

I have a usecase where we are storing byte array into one of the OFFHEAP ATOMIC cache. Multiple threads keep on appending bytes to it using remote operation (Entry processor / invoke call).

Below is the logic for remote operation.

if (mutableEntry.exists()) {
=C2=A0 =C2=A0 =C2=A0 =C2=A0 MyObject m =3D (MyObject)mutableEntry.getV= alue();
=C2=A0 =C2=A0 =C2=A0 =C2=A0 byte[] bytes =3D new byte[m.getContent().length= +
newContent.getContent().length];
=C2=A0 =C2=A0 =C2=A0 =C2=A0 System.arraycopy(m.getContent(), 0, bytes,= 0, m.getContent().length);
=C2=A0 =C2=A0 =C2=A0 =C2=A0 System.arraycopy(newContent.getContent(), = 0, bytes,
m.getContent().length,newContent.getContent().length);
=C2=A0 =C2=A0 =C2=A0 =C2=A0 m.setContent(bytes);
=C2=A0 =C2=A0 =C2=A0 =C2=A0 mutableEntry.setValue(m);
} else {
=C2=A0 =C2=A0 =C2=A0 =C2=A0 mutableEntry.setValue(newContent);
}

We tested by adding about 25MB of content.

What we have seen is -
1. In 8 threads test we have seen average time for Invoke() is 36ms.
2. Out of total time taken to execute Invoke(), 50% time taken by above
remote logic. Where does other 50% spend? Probably just to invoke the
method.

What can we tune here? We are considering InvokeAll(), any more ideas?

If we increase this to 250MB we have seen a very slow performance and after=
hours it goes Out-of-memory, setOffHeapMaxMemory(0) does not help on 32GB box.

We are using Ignite 1.9, Java 1.8, Parallel GC and app JVM is 4GB.
Unfortunately, we can not upgrade Ignite version as of now.

Thanks,
-Sam




--
Sent from: http://apache-ignite-users.70518.x6.= nabble.com/



--

Regards

Pa= vel Vinokurov

--0000000000006c0ba505691431ec--