flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aljoscha Krettek <aljos...@apache.org>
Subject Re: What is the equivalent of Spark RDD is Flink
Date Fri, 25 Dec 2015 09:17:46 GMT
Hi Sourav,
you are right, in Flink the equivalent to an RDD would be a DataSet (or a
DataStream if you are working with the streaming API).

Contrary to Spark, a Flink job is executed lazily when
ExecutionEnvironment.execute() is called. Only then does Flink build an
executable program from the graph of transformations that was built by
calling the transformation methods on DataSet. That’s why I called it lazy.
The operations will also be automatically parallelized. The parallelism of
operations can either be configured in the cluster configuration
(conf/flink-conf.yaml), on a per job basis
(ExecutionEnvironment.setParallelism(int)) or per operation, by calling
setParallelism(int) on a DataSet.

(Above you can always replace DataSet by DataStream, the same explanations
hold.)

So, to get back to your question, yes, the operation of reading the file
(or files in a directory) will be parallelized to several worker nodes
based on the previously mentioned settings.

Let us now if you need more information.

Cheers,
Aljoscha

On Thu, 24 Dec 2015 at 16:49 Sourav Mazumder <sourav.mazumder00@gmail.com>
wrote:

> Hi,
>
> I am new to Flink. Trying to understand some of the basics of Flink.
>
> What is the equivalent of Spark's RDD in Flink ? In my understanding the
> closes think is DataSet API. But wanted to reconfirm.
>
> Also using DataSet API if I ingest a large volume of data (val lines :
> DataSet[String] = env.readTextFile(<some file path and name>)), which may
> not fit in single slave node, will that data get automatically distributed
> in the memory of other slave nodes ?
>
> Regards,
> Sourav
>

Mime
View raw message