flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Malte Schwarzer ...@mieo.de>
Subject Re: How to make Flink to write less temporary files?
Date Mon, 10 Nov 2014 09:54:05 GMT
My blobStore fileds are small, but each *.channel file is around 170MB.
Before I start by Flink job I¹ve 25GB free space available in my tmp-dir and
my taskmanager heap size is currently at 24GB. I¹m using a cluster with 10
nodes.

Is this enough space to process a 1TB file?

Von:  Stephan Ewen <sewen@apache.org>
Antworten an:  <user@flink.incubator.apache.org>
Datum:  Montag, 10. November 2014 10:35
An:  <user@flink.incubator.apache.org>
Betreff:  Re: How to make Flink to write less temporary files?


I would assume that the blobStore fields are rather small (they are only jar
files so far).

I would look for *.channel files, which are spilled intermediate results.
They can get pretty large for large jobs.



Mime
View raw message