flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Márton Balassi <balassi.mar...@gmail.com>
Subject Re: Best way to write data to HDFS by Flink
Date Wed, 10 Jun 2015 08:30:26 GMT
Dear Hawin,

You can pass a hdfs path to DataStream's and DataSet's writeAsText and
writeAsCsv methods.
I assume that you are running a Streaming topology, because your source is
Kafka, so it would look like the following:

StreamExecutionEnvironment env =
StreamExecutionEnvironment.getExecutionEnvironment();

env.addSource(PerisitentKafkaSource(..))
      .map(/* do you operations*/)

.wirteAsText("hdfs://<namenode_name>:<namenode_port>/path/to/your/file");

Check out the relevant section of the streaming docs for more info. [1]

[1]
http://ci.apache.org/projects/flink/flink-docs-master/apis/streaming_guide.html#connecting-to-the-outside-world

Best,

Marton

On Wed, Jun 10, 2015 at 10:22 AM, Hawin Jiang <hawin.jiang@gmail.com> wrote:

> Hi All
>
>
>
> Can someone tell me what is the best way to write data to HDFS when Flink
> received data from Kafka?
>
> Big thanks for your example.
>
>
>
>
>
>
>
>
>
> Best regards
>
> Hawin
>
>
>

Mime
View raw message