flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-9560) RateLimiting for FileSystem
Date Thu, 14 Jun 2018 08:56:00 GMT

    [ https://issues.apache.org/jira/browse/FLINK-9560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16512184#comment-16512184

ASF GitHub Bot commented on FLINK-9560:

Github user pnowojski commented on a diff in the pull request:

    --- Diff: docs/ops/filesystems.md ---
    @@ -102,8 +102,8 @@ fs.<scheme>.limit.input: (number, 0/-1 mean no limit)
     fs.<scheme>.limit.output: (number, 0/-1 mean no limit)
     fs.<scheme>.limit.timeout: (milliseconds, 0 means infinite)
     fs.<scheme>.limit.stream-timeout: (milliseconds, 0 means infinite)
    -fs.<scheme>.limit.rateLimitingInput: (bytes/s, 0 means infinite)
    -fs.<scheme>.limit.rateLimitingOutput: (bytes/s, 0 means infinite)
    +fs.<scheme>.limit.input-rate-limit: (bytes/s, 0 means infinite . By default, there
is no limits)
    --- End diff --
    `there is no limit` (limit instead of limits)
    And sorry for confusion, but I just noticed that word `limit` occurs twice in property
name. It would be enough to name it:
    Sorry for previous comment in wrong direction!

> RateLimiting for FileSystem
> ---------------------------
>                 Key: FLINK-9560
>                 URL: https://issues.apache.org/jira/browse/FLINK-9560
>             Project: Flink
>          Issue Type: Improvement
>          Components: FileSystem
>    Affects Versions: 1.5.0
>            Reporter: Etienne CARRIERE
>            Priority: Major
> *Pain*: On our system, we see that during checkpoint , all the bandwidth is take to send
the checkpoint to object storage (s3 in our case)
> *Proposal* : After the creation of some limitation on Filesystem (mostly number of connections
with the  tickets FLINK-8125/FLINK-8198/FLINK-9468), I propose to add ratelimiting "per Filesystem"
> *Proposal of implementation* : Modify LimitedConnectionsFileSystem to add a ratelimiter
on both Input and OutputStream.

This message was sent by Atlassian JIRA

View raw message