From dev-return-36064-archive-asf-public=cust-asf.ponee.io@ignite.apache.org Fri Jun 29 10:38:00 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 2E7A4180663 for ; Fri, 29 Jun 2018 10:37:59 +0200 (CEST) Received: (qmail 88535 invoked by uid 500); 29 Jun 2018 08:37:59 -0000 Mailing-List: contact dev-help@ignite.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@ignite.apache.org Delivered-To: mailing list dev@ignite.apache.org Received: (qmail 88517 invoked by uid 99); 29 Jun 2018 08:37:58 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 29 Jun 2018 08:37:58 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 22CC8C170A for ; Fri, 29 Jun 2018 08:37:58 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.888 X-Spam-Level: * X-Spam-Status: No, score=1.888 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001, T_DKIMWL_WL_MED=-0.01] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id CfWR120pnWL6 for ; Fri, 29 Jun 2018 08:37:54 +0000 (UTC) Received: from mail-vk0-f44.google.com (mail-vk0-f44.google.com [209.85.213.44]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id 5CE9F5F18A for ; Fri, 29 Jun 2018 08:37:54 +0000 (UTC) Received: by mail-vk0-f44.google.com with SMTP id l64-v6so4883672vkl.12 for ; Fri, 29 Jun 2018 01:37:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=6cjIWZ98mIyoepNyJKeR8Q/2rsYc1Lme4PTiE1YLjSo=; b=nYD1gFaZ4RnWXmSGa1esbBxFbn6TizDNHceeZvCgQFwdy7dFHMrpXtyplq9rItSmty jLzM2Jjb35QlnZpn8zIBGWroauCP/W9R6WPcxeIcgwvW4u0OdHDR/EFtnxghpqeobUbz okg1uxrLB8rN2gCpp/Kxj+rxNHYvkOt2oRf5E+0xjr1ASbPYuE5xeXgMsezX9E0BnijB 1L+mJXqQ55IEys2DVDtQJLndKVbMLhpxK16KeG997m47/jZHHS0aI/Hmv0HFbOkOQV9t e33oKWCgWgMzkV1/AGyRlc8TRpB2lSYKFsbqFWb6M53RqJWjGoXiZV4alT87Dbz0HJRn SWpA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=6cjIWZ98mIyoepNyJKeR8Q/2rsYc1Lme4PTiE1YLjSo=; b=UKxJwgMSjL7nZykc/WMv6FMSXXpZxM8uZGeqvdKHB22GYolMeERuCATh6Jaw0PLP9g 4Dih2tZiY2KRCIP2HIod+JAyK8w853RHvDjlUBO6o44wN+LaBwNYq+bJs+JYzp3Zwkf9 rEg2vWG7ufFVQLTTCMWz39CsR94bTi+CGnI5/Dm9sBKS7iOuelUnoepqmHIQ28WHPAVB GsJKMZ4uBIjOqyiG116CrNuZG+/11MDS80+pCuvB/mKahpDWzadEj+xS8pLWJRn167h5 IcybOGMcw1+jvz0elBXq1JQ/tUApzmvtsWBotFbRrm75MpG1OsvhdGrLBITASch0OjFp Ugvg== X-Gm-Message-State: APt69E0jZXVtoTse8gGk3jr47gmuDSn4hw5iGlpjbsszFXS4vps/xcIn 7Slvk1GxtfQlVB6CKRTU4ClLb1DGcK6Z0JFOnWFPvg== X-Google-Smtp-Source: AAOMgpdYraCI+A0lmPc8RECKG4Zh1Mi1WxIMxChQElCYQC3aSL3eeW+/cg8Bct2b5HWejsdOS8OIwMtMxm7YBxgbW7M= X-Received: by 2002:a1f:8cb:: with SMTP id 194-v6mr8703422vki.177.1530261468474; Fri, 29 Jun 2018 01:37:48 -0700 (PDT) MIME-Version: 1.0 From: Andrey Mashenkov Date: Fri, 29 Jun 2018 11:37:37 +0300 Message-ID: Subject: Potential OOM while iterating over query cursor. Review needed. To: dev Content-Type: multipart/alternative; boundary="0000000000003df265056fc3c3c8" --0000000000003df265056fc3c3c8 Content-Type: text/plain; charset="UTF-8" Hi Igniters, There is an issue IGNITE-8892 [1] related to OOM during distributed query execution. This issue is not limited with ScanQuery usage and looks like affected all query types. The use case is quite simple. 1 server and 1 client. Client starts scan query with default flags and iterate over cursor. If whole query result is not fit to memory - JVM will crashed with OOM, but it is not expected as client takes entries one by one and throw out them immediately. Reproducer is attached to the ticket. Same query works fine if query starts on server. Seems, we have no DistributedQueryFuture in that case and all works fine. I've found GridCacheDistributedQueryFuture collects all entries and try to return the collection via onDone(). This behaviour turn on with a flag 'keepAll' which is true by default. Iterating over cache via cache.iterator() has no OOM issues as we set keepAll flag to false. Why we have keepAll=true by default as seems noone expects future.get() will return any data and all queries works through queue in paging mode? Will it be ok to get rid of 'allCol' and keepAll flag at all? I've made a PR and TC looks fine. Could someone review it, please? [1] https://issues.apache.org/jira/browse/IGNITE-8892 -- Best regards, Andrey V. Mashenkov --0000000000003df265056fc3c3c8--