From user-return-30510-archive-asf-public=cust-asf.ponee.io@flink.apache.org Fri Oct 25 15:49:14 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id EC74D180648 for ; Fri, 25 Oct 2019 17:49:13 +0200 (CEST) Received: (qmail 25900 invoked by uid 500); 25 Oct 2019 15:49:11 -0000 Mailing-List: contact user-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@flink.apache.org Received: (qmail 25889 invoked by uid 99); 25 Oct 2019 15:49:11 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Oct 2019 15:49:11 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 7616FC05D2 for ; Fri, 25 Oct 2019 15:49:10 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 0.001 X-Spam-Level: X-Spam-Status: No, score=0.001 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.2, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-he-de.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id VoIzGmFIiECn for ; Fri, 25 Oct 2019 15:49:09 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=2607:f8b0:4864:20::236; helo=mail-oi1-x236.google.com; envelope-from=fhueske@gmail.com; receiver= Received: from mail-oi1-x236.google.com (mail-oi1-x236.google.com [IPv6:2607:f8b0:4864:20::236]) by mx1-he-de.apache.org (ASF Mail Server at mx1-he-de.apache.org) with ESMTPS id 4F6307F73A for ; Fri, 25 Oct 2019 15:49:08 +0000 (UTC) Received: by mail-oi1-x236.google.com with SMTP id a15so1944267oic.0 for ; Fri, 25 Oct 2019 08:49:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=5cobyHRRpXQmRjj/aNcagiTTCYhElCDorWSqgy13glc=; b=Jz5eNVsB+njA0ikRcNm0aF4fYKxych7r6DOXdrqhYzTLweyws2Fjzx5RITaX2M5O6I Tv1uJHTpvWxdmQ3LjJUzTd07YRXI26zVTu0ZCFDFvLWdh1D7hC9/3K4dlAnB4Hp/IDnq Y4ss52nlhMO39APuE+ozw1GyUGDJPU/BNMJULU0Ool1piUsY1plNtzEWRYjA5UgONfx+ hCedZbdxrVeTWFEIv0f4CG9/hfdajclyp9Kylq1bvcGpxQZeMvZ+Byb9jx+Gsl0qR/wq cSrKtPaYIn0hOppv3SpwQI2mwgTt/tYardUO1u4CcGl5RGKJP6eSzCsFrByh7zZ/jOlS 1DeA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=5cobyHRRpXQmRjj/aNcagiTTCYhElCDorWSqgy13glc=; b=hLVzSe5wQHvOBn57b5zhKNF1zJXlYLtg3NpzP+FoSi3uXNJlM1/bjd8oGrYN7AB7Yv DX/0mF/7NaN9LWaU9b1gCkal1iDxZsNaxGIfZcVy3lTLacv2zghZfa4TilZHA7l0OMUm lmRpU8BnzdeP2GOboKgqr9i1GChc/Jo5fe11yxM5GPevyjm0gyUFLcjsXHa4TRMZ6DYa 1Oq9EBnx7/jxJfVnuvfZDIdQNXUFCDggnT0mDeD9P7OYvnexOI0aWPYtx0dFyrkX0Z4h gDgW1YC4PToy8d0yzKoXwWSucCcsAeM3njF6zesDg8N2VFOvtWO/GvytoxJZiNnle6BH 11lQ== X-Gm-Message-State: APjAAAXFokHFRs3YxXAwgqyfacb1IuNzfteTQ9Ja+5Elf7/nr/B0VcTg GR22tmdvQBeDG3EAO8mQuFs3RCHI83XpH4yrnwQ= X-Google-Smtp-Source: APXvYqwFGgQaDvQHVIKWj4v+p4jloM0rQSDPzAhTqUULO78yMvvLa9p5GtxCtPpt/T6Onx7159RsQQ1pFAEP4BTT/yo= X-Received: by 2002:a54:4f8e:: with SMTP id g14mr3705037oiy.5.1572018546910; Fri, 25 Oct 2019 08:49:06 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Fabian Hueske Date: Fri, 25 Oct 2019 17:48:29 +0200 Message-ID: Subject: Re: Flink 1.5+ performance in a Java standalone environment To: Jakub Danilewicz Cc: user Content-Type: multipart/alternative; boundary="00000000000011ab520595be17aa" --00000000000011ab520595be17aa Content-Type: text/plain; charset="UTF-8" Hi Jakub, I had a look at the changes of Flink 1.5 [1] and didn't find anything obvious. Something that might cause a different behavior is the new deployment and process model (FLIP-6). In Flink 1.5, there is a switch to disable it and use the previous deployment mechanism. You could try to disable the new new model [2] and see if this cause the performance issue. Note that the legacy mode was removed in one of the later versions. Best, Fabian [1] https://flink.apache.org/news/2018/05/25/release-1.5.0.html [2] https://ci.apache.org/projects/flink/flink-docs-release-1.5/release-notes/flink-1.5.html#update-configuration-for-reworked-job-deployment Am Do., 24. Okt. 2019 um 19:37 Uhr schrieb Jakub Danilewicz < jdanilewicz@alto-analytics.com>: > Hi, > > I have recently tried to upgrade Flink from 1.2.0 to the newest version > and noticed that starting from the version 1.5 the performance is much > worse when processing fixed graphs in a standalone JVM environment (Java > 8). > > This affects all the use-cases when a Gelly graph (pre-built from a fixed > collection of nodes/edges) gets processed by any of our custom algorithms > (VertexCentric, ScatterGather or GSA), especially when using parallel > processing for a local ExecutionEnvironment. The processing times > (compared to the versions <= 1.4.2) double/triple, while CPU and memory > consumption increase significantly. > > Are there any fine-tuning steps/tricks for the job processing engine > behind Flink 1.5+ that would improve the performance in the scenarios > described above? > > Best, > > Jakub > --00000000000011ab520595be17aa Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi Jakub,

I had a look at th= e changes of Flink 1.5 [1] and didn't find anything obvious.
= Something that might cause a different behavior is the new deployment and p= rocess model (FLIP-6).

In Flink 1.5, there is a sw= itch to disable it and use the previous deployment mechanism.
You could try to disable the new new model [2] and see if this caus= e the performance issue.

Note that the legacy mode= was removed in one of the later versions.

Bes= t, Fabian


Am Do., 24. Okt. 2019 um 19:37=C2= =A0Uhr schrieb Jakub Danilewicz <jdanilewicz@alto-analytics.com>:
Hi,

=
I have recently tried to upgrade Flink from 1.2.0 to the newest = version and noticed that starting from the version 1.5 the performance is m= uch worse when processing fixed graphs in a standalone JVM environment (Jav= a 8).

This affects all the use-cases when a G= elly graph (pre-built from a fixed collection of nodes/edges) gets processe= d by any of our custom algorithms (VertexCentric, ScatterGather or GSA), es= pecially when using parallel processing for a local ExecutionEnvironment. The processing times (compared = to the versions <=3D 1.4.2) double/triple, while CPU and memory consumpt= ion increase significantly.

Are there any fine-tun= ing steps/tricks for the job processing engine behind Flink 1.5+ that would= improve the performance in the scenarios described above?

Best,

Jakub
--00000000000011ab520595be17aa--