Return-Path: X-Original-To: apmail-flink-user-archive@minotaur.apache.org Delivered-To: apmail-flink-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id EBDC818A1E for ; Mon, 10 Aug 2015 13:59:07 +0000 (UTC) Received: (qmail 32306 invoked by uid 500); 10 Aug 2015 13:59:07 -0000 Delivered-To: apmail-flink-user-archive@flink.apache.org Received: (qmail 32228 invoked by uid 500); 10 Aug 2015 13:59:07 -0000 Mailing-List: contact user-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@flink.apache.org Delivered-To: mailing list user@flink.apache.org Received: (qmail 32213 invoked by uid 99); 10 Aug 2015 13:59:07 -0000 Received: from Unknown (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 10 Aug 2015 13:59:07 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 549041A9C5D for ; Mon, 10 Aug 2015 13:59:07 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.041 X-Spam-Level: ** X-Spam-Status: No, score=2.041 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, HTML_MESSAGE=3, RCVD_IN_MSPIKE_H2=-1.108, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd2-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=googlemail.com Received: from mx1-eu-west.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id kDTHMLAGQ94j for ; Mon, 10 Aug 2015 13:59:06 +0000 (UTC) Received: from mail-ig0-f172.google.com (mail-ig0-f172.google.com [209.85.213.172]) by mx1-eu-west.apache.org (ASF Mail Server at mx1-eu-west.apache.org) with ESMTPS id AD96B27737 for ; Mon, 10 Aug 2015 13:59:05 +0000 (UTC) Received: by igbjg10 with SMTP id jg10so1209102igb.0 for ; Mon, 10 Aug 2015 06:59:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlemail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=G1qbk6KDXsdTmiNrBRG20SvxKk9q+yP/BPMCcHZJVnY=; b=giHqLmHZAe4XiHE78zDvy78j+ffBJDHocXcP77nKl2O+fKNt51iyKCikTY1+ezPchB klIO0VKGiLzQHYlBeLa05ZFiL8wcF0AfYZ0ZF7ws8Mi6X4XOD7TUajOQ/CDA3aqB3SaT IZC6B+bX6NJhlG4d84eSsSKDXf3hw6s5udUDRwai0n8OeHrLQb+8HzvSsIvIN/QCq4D/ kb7Pryg7OB4J18udsISD5JXXX8S+R9W8XI7Afs8RxTkXUCfApmfbDK6///bqGex5WdTW t5muTA1JagPVKiGPxnAIFeAYauqrvNMDggV+G8YIYKrsFR7UsvFbGfqYxN5LGOeutYOs Sg6Q== MIME-Version: 1.0 X-Received: by 10.50.122.102 with SMTP id lr6mr3277506igb.56.1439215144722; Mon, 10 Aug 2015 06:59:04 -0700 (PDT) Received: by 10.107.141.133 with HTTP; Mon, 10 Aug 2015 06:59:04 -0700 (PDT) Date: Mon, 10 Aug 2015 15:59:04 +0200 Message-ID: Subject: spark vs flink low memory available From: =?UTF-8?B?UGEgUsO2?= To: user , user@flink.apache.org Content-Type: multipart/alternative; boundary=089e01538a8074e647051cf563f5 --089e01538a8074e647051cf563f5 Content-Type: text/plain; charset=UTF-8 hi community, i have build a spark and flink k-means application. my test case is a clustering on 1 million points on 3node cluster. in memory bottlenecks begins flink to outsource to disk and work slowly but works. however spark lose executers if the memory is full and starts again (infinety loop?). i try to customize the memory setting with the help from the mailing list here , thanks. but spark not work. is it necessary to have any configurations to be set? i mean flink work with low memory, spark must also be able to or not? best regards, paul --089e01538a8074e647051cf563f5 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
hi community,

i have= build a spark and flink k-means application.
my test case is a cl= ustering on 1 million points on 3node cluster.

in memory bottlenecks b= egins flink to outsou= rce to disk and work<= /span> slowly but works.
however spark lose executers if the memory is full and starts again (infinety loop?).

i try to customize the memory setting with the help from the mailing l= ist here, thanks.
but spark not work.
is it necessary to have any <= span class=3D"hps">configurations to be set? i mean flink work with low memory, spark must <= /span>also be able to= or not?

best regards,
paul
--089e01538a8074e647051cf563f5--