flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Till Rohrmann (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-4399) Add support for oversized messages
Date Tue, 16 Aug 2016 12:31:20 GMT

    [ https://issues.apache.org/jira/browse/FLINK-4399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15422667#comment-15422667
] 

Till Rohrmann commented on FLINK-4399:
--------------------------------------

Hi Felix,

both issues have the same idea. However, this issue tries to solve the problem at a lower
level, namely at the level of the remote procedure calls. This means that we support "large"
messages/RPCs in general instead of only supporting it for the accumulators.

> Add support for oversized messages
> ----------------------------------
>
>                 Key: FLINK-4399
>                 URL: https://issues.apache.org/jira/browse/FLINK-4399
>             Project: Flink
>          Issue Type: Sub-task
>          Components: Distributed Coordination
>         Environment: FLIP-6 feature branch
>            Reporter: Stephan Ewen
>             Fix For: 1.2.0
>
>
> Currently, messages larger than the maximum Akka Framesize cause an error when being
transported. We should add a way to pass messages that are larger than the Framesize, as may
happen for:
>   - {{collect()}} calls that collect large data sets (via accumulators)
>   - Job submissions and operator deployments where the functions closures are large (for
example because it contains large pre-loaded data)
>   - Function restore in cases where restored state is larger than checkpointed state
(union state)
> I suggest to use the {{BlobManager}} to transfer large payload.
>   - On the sender side, oversized messages are stored under a transient blob (which is
deleted after first retrieval, or after a certain number of minutes)
>   - The sender sends a "pointer to blob message" instead.
>   - The receiver grabs the message from the blob upon receiving the pointer message
> The RPC Service should be optionally initializable with a "large message handler" which
is internally the {{BlobManager}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message