Return-Path: X-Original-To: apmail-flink-user-archive@minotaur.apache.org Delivered-To: apmail-flink-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 983E21746F for ; Mon, 10 Nov 2014 09:54:41 +0000 (UTC) Received: (qmail 28037 invoked by uid 500); 10 Nov 2014 09:54:41 -0000 Delivered-To: apmail-flink-user-archive@flink.apache.org Received: (qmail 27967 invoked by uid 500); 10 Nov 2014 09:54:41 -0000 Mailing-List: contact user-help@flink.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@flink.incubator.apache.org Delivered-To: mailing list user@flink.incubator.apache.org Received: (qmail 27956 invoked by uid 99); 10 Nov 2014 09:54:41 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 10 Nov 2014 09:54:41 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,MIME_QP_LONG_LINE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: local policy) Received: from [85.13.129.7] (HELO dd2236.kasserver.com) (85.13.129.7) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 10 Nov 2014 09:54:13 +0000 Received: from [192.168.0.32] (unknown [95.91.208.176]) by dd2236.kasserver.com (Postfix) with ESMTPSA id 934064AA02F7 for ; Mon, 10 Nov 2014 10:54:11 +0100 (CET) User-Agent: Microsoft-MacOutlook/14.3.4.130416 Date: Mon, 10 Nov 2014 10:54:05 +0100 Subject: Re: How to make Flink to write less temporary files? From: Malte Schwarzer To: Message-ID: Thread-Topic: How to make Flink to write less temporary files? In-Reply-To: Mime-version: 1.0 Content-type: multipart/alternative; boundary="B_3498461651_10693528" X-Virus-Checked: Checked by ClamAV on apache.org > This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. --B_3498461651_10693528 Content-type: text/plain; charset="ISO-8859-1" Content-transfer-encoding: quoted-printable My blobStore fileds are small, but each *.channel file is around 170MB. Before I start by Flink job I=B9ve 25GB free space available in my tmp-dir an= d my taskmanager heap size is currently at 24GB. I=B9m using a cluster with 10 nodes. Is this enough space to process a 1TB file? Von: Stephan Ewen Antworten an: Datum: Montag, 10. November 2014 10:35 An: Betreff: Re: How to make Flink to write less temporary files? I would assume that the blobStore fields are rather small (they are only ja= r files so far). I would look for *.channel files, which are spilled intermediate results. They can get pretty large for large jobs. --B_3498461651_10693528 Content-type: text/html; charset="ISO-8859-1" Content-transfer-encoding: quoted-printable
My blobStore fileds are small= , but each *.channel file is around 170MB. Before I start by Flink job IR= 17;ve 25GB free space available in my tmp-dir and my taskmanager heap size i= s currently at 24GB. I’m using a cluster with 10 nodes.

=
Is this enough space to process a 1TB file?

<= span id=3D"OLK_SRC_BODY_SECTION">
Von: Stephan Ewen <sewen@apache.org>
An= tworten an: <use= r@flink.incubator.apache.org>
Datum= : Montag, 10. November 2014 10:35
= An: <user@flink.= incubator.apache.org>
Betreff: Re: How to make Flink to write less temporary files?

<= /div>

I would assume that the blobStore fields are rather small = (they are only jar files so far).

I would look for *.channel= files, which are spilled intermediate results. They can get pretty large fo= r large jobs.

--B_3498461651_10693528--