flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Павел Поцелуев <potsel...@yandex-team.ru>
Subject Re: Bandwidth throttling of checkpoints uploading to s3
Date Fri, 12 Jul 2019 16:11:44 GMT
<ol><li>We use FsStateBackend and state snapshot size is about 700 Mbyte.</li><li>We
are thinking about migration to RocksDBStateBackend and turning on incremental checkpoints.</li><li>I
think incremental size would be small in our current use case so incremental checkpoints can
solve the problem.</li><li>I think it is about 50 <span style="background:#f8f9fa;color:#222222;float:none;font:400
14px sans-serif;text-decoration-style:initial;text-transform:none;white-space:normal;word-spacing:0px">Mbit</span>/s.</li></ol><div>Thanks.</div><div> </div><div>12.07.2019,
17:27, "Yu Li" &lt;carp84@gmail.com&gt;:</div><blockquote><div>Hi
Pavel,<div> </div><div>Currently there's no such throttling functionality
in Flink and I think it's a valid requirement. But before opening a JIRA for this, please
allow me to ask for more details to better understand your scenario:</div><div>1.
What kind of state backend are you using? Since you observe high load to network, I guess
the state is large and you are using RocksDB backend?</div><div>2. If you're using
RocksDB backend, have you configured to use incremental checkpoint? To be more specified,
have you set the "state.backend.incremental" property to true? (by default it's false)</div><div>3.
If you're using RocksDB backend with full checkpoint, what the incremental size of checkpoint
would be (within a checkpoint interval)?</div><div>4. What's the max bandwidth
you'd like to throttle to for S3?</div><div> </div><div>Asking because
if you're using RocksDB with full checkpoint and the incremental checkpoint size is as small
as not exceeding your expected throttle for S3, you could directly try incremental checkpoint
to resolve the current problem.</div><div> </div><div>Thanks.</div><div> <div><div><div><div><div>Best
Regards,</div><div>Yu</div></div></div></div></div></div></div> <div><div>On
Fri, 12 Jul 2019 at 20:39, Pavel Potseluev &lt;<a href="mailto:potseluev@yandex-team.ru">potseluev@yandex-team.ru</a>&gt;
wrote:</div><blockquote style="border-left-color:rgb( 204 , 204 , 204 );border-left-style:solid;border-left-width:1px;margin:0px
0px 0px 0.8ex;padding-left:1ex"><div>Hello!</div><div> </div><div>We
use flink with periodically checkpointing to s3 file system. And when flink uploads checkpoint
to s3 it makes high load to the network. We have found in AWS CLI S3 configuration option
called <span style="background:#ffffff;color:#333333;float:none;font:400 14px 'consolas'
, 'monaco' , 'menlo' , 'courier new' , monospace;text-align:left;text-decoration-style:initial;text-transform:none;white-space:normal;word-spacing:0px"><a
href="https://docs.aws.amazon.com/cli/latest/topic/s3-config.html#max-bandwidth">max_bandwidth</a>
</span><span style="background:#ffffff;float:none;font-size:14px;font-style:normal;font-weight:400;text-align:left;text-decoration-style:initial;text-transform:none;word-spacing:0px">which
allows to limit rate in bytes per second</span><span style="background:#ffffff;color:#333333;float:none;font:400
14px 'consolas' , 'monaco' , 'menlo' , 'courier new' , monospace;text-align:left;text-decoration-style:initial;text-transform:none;white-space:normal;word-spacing:0px">.</span>
Is there a way to have the same functionality with flink?</div><div> </div><div>-- </div><div
style="background:rgb( 255 , 255 , 255 );color:rgb( 0 , 0 , 0 );font:400 15px 'arial' , sans-serif;text-decoration-style:initial;text-transform:none;white-space:normal;word-spacing:0px">Best
regards,</div><div style="background:rgb( 255 , 255 , 255 );color:rgb( 0 , 0 , 0
);font:400 15px 'arial' , sans-serif;text-decoration-style:initial;text-transform:none;white-space:normal;word-spacing:0px">Pavel
Potseluev</div><div style="background:rgb( 255 , 255 , 255 );color:rgb( 0 , 0 , 0
);font:400 15px 'arial' , sans-serif;text-decoration-style:initial;text-transform:none;white-space:normal;word-spacing:0px">Software
developer, Yandex.Classifieds LLC</div><div> </div></blockquote></div></blockquote><div> </div><div> </div><div>-- </div><div
style="background:rgb( 255 , 255 , 255 );color:rgb( 0 , 0 , 0 );font:400 15px 'arial' , sans-serif;text-decoration-style:initial;text-transform:none;white-space:normal;word-spacing:0px">Best
regards,</div><div style="background:rgb( 255 , 255 , 255 );color:rgb( 0 , 0 , 0
);font:400 15px 'arial' , sans-serif;text-decoration-style:initial;text-transform:none;white-space:normal;word-spacing:0px">Pavel
Potseluev</div><div style="background:rgb( 255 , 255 , 255 );color:rgb( 0 , 0 , 0
);font:400 15px 'arial' , sans-serif;text-decoration-style:initial;text-transform:none;white-space:normal;word-spacing:0px">Software
developer, Yandex.Classifieds LLC</div><div> </div>
Mime
View raw message