Mailing-List: contact user-help@flink.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@flink.apache.org
MIME-Version: 1.0
In-Reply-To: <CA+3U0ZkgP-Cp9+LtDD5b1EcO=jDseUYzKRTdE-=c59+TMczmfw@mail.gmail.com>
References: <CALTW16F9HvhPV4cT2oOUdcx1dt8pN_Qy69WnRy21wCP6CwqiQA@mail.gmail.com>
 <CA+3U0ZkgP-Cp9+LtDD5b1EcO=jDseUYzKRTdE-=c59+TMczmfw@mail.gmail.com>
From: Jonathan Share <jon.share@gmail.com>
Date: Wed, 23 Nov 2016 18:40:53 +0100
Message-ID: <CALTW16H+q1Vdw2uTVDG2igV71KbqACjtY6yrzkY3EMXaO73LSQ@mail.gmail.com>
Subject: Re: S3 checkpointing in AWS in Frankfurt
To: user@flink.apache.org
Content-Type: multipart/alternative; boundary=001a1146ff48ff39540541fb63c2
archived-at: Wed, 23 Nov 2016 17:41:00 -0000

--001a1146ff48ff39540541fb63c2
Content-Type: text/plain; charset=UTF-8

Hi Greg,

Standard storage class, everything is on defaults, we've not done anything
special with the bucket.

Cloud Watch only appears to give me total billing for S3 in general, I
don't see a breakdown unless that's something I can configure somewhere.

Regards,
Jonathan


On 23 November 2016 at 16:29, Greg Hogan <code@greghogan.com> wrote:

> Hi Jonathan,
>
> Which S3 storage class are you using? Do you have a breakdown of the S3
> costs as storage / API calls / early deletes / data transfer?
>
> Greg
>
> On Wed, Nov 23, 2016 at 2:52 AM, Jonathan Share <jon.share@gmail.com>
> wrote:
>
>> Hi,
>>
>> I'm interested in hearing if anyone else has experience with using Amazon
>> S3 as a state backend in the Frankfurt region. For political reasons we've
>> been asked to keep all European data in Amazon's Frankfurt region. This
>> causes a problem as the S3 endpoint in Frankfurt requires the use of AWS
>> Signature Version 4 "This new Region supports only Signature Version 4"
>> [1] and this doesn't appear to work with the Hadoop version that Flink is
>> built against [2].
>>
>> After some hacking we have managed to create a docker image with a build
>> of Flink 1.2 master, copying over jar files from the hadoop
>> 3.0.0-alpha1 package and this appears to work, for the most part but we
>> still suffer from some classpath problems (conflicts between AWS API used
>> in hadoop and those we want to use in out streams for interacting with
>> Kinesis) and the whole thing feels a little fragile. Has anyone else tried
>> this? Is there a simpler solution?
>>
>> As a follow-up question, we saw that with checkpointing on three
>> relatively simple streams set to 1 second, our S3 costs were higher than
>> the EC2 costs for our entire infrastructure. This seems slightly
>> disproportionate. For now we have reduced checkpointing interval to 10
>> seconds and that has greatly improved the cost projections graphed via
>> Amazon Cloud Watch, but I'm interested in hearing other peoples experience
>> with this. Is that the kind of billing level we can expect or is this a
>> symptom of a mis-configuration? Is this a setup others are using? As we are
>> using Kinesis as the source for all streams I don't see a huge risk with
>> larger checkpoint intervals and our Sinks are designed to mostly tolerate
>> duplicates (some improvements can be made).
>>
>> Thanks in advance
>> Jonathan
>>
>>
>> [1] https://aws.amazon.com/blogs/aws/aws-region-germany/
>> [2] https://issues.apache.org/jira/browse/HADOOP-13324
>>
>
>

--001a1146ff48ff39540541fb63c2
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Hi Greg,<div><br></div><div>Standard storage class, everyt=
hing is on defaults, we&#39;ve not done anything special with the bucket.</=
div><div><br></div><div>Cloud Watch only appears to give me total billing f=
or S3 in general, I don&#39;t see a breakdown unless that&#39;s something I=
 can configure somewhere.</div><div><br></div><div>Regards,</div><div>Jonat=
han</div><div><br></div></div><div class=3D"gmail_extra"><br><div class=3D"=
gmail_quote">On 23 November 2016 at 16:29, Greg Hogan <span dir=3D"ltr">&lt=
;<a href=3D"mailto:code@greghogan.com" target=3D"_blank">code@greghogan.com=
</a>&gt;</span> wrote:<br><blockquote class=3D"gmail_quote" style=3D"margin=
:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir=3D"ltr"><=
div>Hi Jonathan,<br><br></div><div>Which S3 storage class are you using? Do=
 you have a breakdown of the S3 costs as storage / API calls / early delete=
s / data transfer?<span class=3D"HOEnZb"><font color=3D"#888888"><br><br></=
font></span></div><span class=3D"HOEnZb"><font color=3D"#888888"><div></div=
>Greg<br></font></span></div><div class=3D"HOEnZb"><div class=3D"h5"><div c=
lass=3D"gmail_extra"><br><div class=3D"gmail_quote">On Wed, Nov 23, 2016 at=
 2:52 AM, Jonathan Share <span dir=3D"ltr">&lt;<a href=3D"mailto:jon.share@=
gmail.com" target=3D"_blank">jon.share@gmail.com</a>&gt;</span> wrote:<br><=
blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px=
 #ccc solid;padding-left:1ex"><div dir=3D"ltr">Hi,<div><br></div><div>I&#39=
;m interested in hearing if anyone else has experience with using Amazon S3=
 as a state backend in the Frankfurt region. For political reasons we&#39;v=
e been asked to keep all European data in Amazon&#39;s Frankfurt region. Th=
is causes a problem as the S3 endpoint in Frankfurt requires the use of AWS=
 Signature Version 4 &quot;<span style=3D"color:rgb(51,51,51);font-family:h=
elveticaneue,helvetica,helvetica,arial,sans-serif;font-size:14px">This new =
Region supports only Signature Version 4</span>&quot; [1] and this doesn=
9;t appear to work with the Hadoop version that Flink is built against [2].=
</div><div><br></div><div>After some hacking we have managed to create a do=
cker image with a build of Flink 1.2 master, copying over jar files from th=
e hadoop 3.0.0-alpha1=C2=A0package and this appears to work, for the most p=
art but we still suffer from some classpath problems (conflicts between AWS=
 API used in hadoop and those we want to use in out streams for interacting=
 with Kinesis) and the whole thing feels a little fragile. Has anyone else =
tried this? Is there a simpler solution?</div><div><br></div><div>As a foll=
ow-up question, we saw that with checkpointing on three relatively simple s=
treams set to 1 second, our S3 costs were higher than the EC2 costs for our=
 entire infrastructure. This seems slightly disproportionate. For now we ha=
ve reduced checkpointing interval to 10 seconds and that has greatly improv=
ed the cost projections graphed via Amazon Cloud Watch, but I&#39;m interes=
ted in hearing other peoples experience with this. Is that the kind of bill=
ing level we can expect or is this a symptom of a mis-configuration? Is thi=
s a setup others are using? As we are using Kinesis as the source for all s=
treams I don&#39;t see a huge risk with larger checkpoint intervals and our=
 Sinks are designed to mostly tolerate duplicates (some improvements can be=
 made).=C2=A0</div><div><br></div><div>Thanks in advance</div><div>Jonathan=
</div><div><br></div><div><br></div><div>[1] <a href=3D"https://aws.amazon.=
com/blogs/aws/aws-region-germany/" target=3D"_blank">https://aws.amazon.com=
/blogs/a<wbr>ws/aws-region-germany/</a></div><div>[2]=C2=A0<a href=3D"https=
://issues.apache.org/jira/browse/HADOOP-13324" target=3D"_blank">https://is=
sues.apache.org/<wbr>jira/browse/HADOOP-13324</a></div></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div>

--001a1146ff48ff39540541fb63c2--