Mailing-List: contact user-help@flink.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@flink.apache.org
MIME-Version: 1.0
From: Vinay Patil <vinay18.patil@gmail.com>
Date: Tue, 23 Aug 2016 21:20:46 -0500
Message-ID: <CAMpYU5T+Zkgaq+k3vajX+EJqjVz=g3fvKHZjP05ys6uKk0bj9g@mail.gmail.com>
Subject: Dealing with Multiple sinks in Flink
To: user@flink.apache.org
Content-Type: multipart/alternative; boundary=001a113ee1bcd3ffe8053ac7ed05
archived-at: Wed, 24 Aug 2016 02:20:59 -0000

--001a113ee1bcd3ffe8053ac7ed05
Content-Type: text/plain; charset=UTF-8

Hi,

In our flink pipeline we are currently writing the data to multiple S3
objects/folders based on some conditions, so the issue I am facing is as
follows :

Consider these S3 folders :
temp_bucket/processedData/20160823/
temp_bucket/rawData/20160822/
temp_bucket/errorData/20160821/

Now when the parallelism is set to 1, the data gets written to all S3
folders above, but when I set it to larger value the data is written only
to the first folder and not the others.

I am testing the flink job on EMR with 4 task managers having 16 slots,
even if I keep parallelism as 4 , I am facing the same issue.
(running from IDE is resulting in same output, Tested this with Flink 1.0.3
and 1.1.1)

I am not understanding why this is happening.


Regards,
Vinay Patil

--001a113ee1bcd3ffe8053ac7ed05
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Hi,<div><br></div><div>In our flink pipeline we are curren=
tly writing the data to multiple S3 objects/folders based on some condition=
s, so the issue I am facing is as follows :</div><div><br></div><div>Consid=
er these S3 folders :<br>temp_bucket/processedData/20160823/</div><div>temp=
_bucket/rawData/20160822/<br></div><div>temp_bucket/errorData/20160821/<br>=
</div><div><br></div><div>Now when the parallelism is set to 1, the data ge=
ts written to all S3 folders above, but when I set it to larger value the d=
ata is written only to the first folder and not the others.</div><div><br><=
/div><div>I am testing the flink job on EMR with 4 task managers having 16 =
slots, even if I keep parallelism as 4 , I am facing the same issue.</div><=
div>(running from IDE is resulting in same output, Tested this with Flink 1=
.0.3 and 1.1.1)</div><div><br></div><div>I am not understanding why this is=
 happening.</div><div><br></div><div><br clear=3D"all"><div><div class=3D"g=
mail_signature" data-smartmail=3D"gmail_signature"><div dir=3D"ltr"><div><d=
iv dir=3D"ltr"><font color=3D"#000000">Regards,</font><div><font color=3D"#=
000000">Vinay Patil</font></div></div></div></div></div></div>
</div></div>

--001a113ee1bcd3ffe8053ac7ed05--