flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Stephan Ewen (JIRA)" <j...@apache.org>
Subject [jira] [Created] (FLINK-1675) Rework Accumulators
Date Tue, 10 Mar 2015 14:56:38 GMT
Stephan Ewen created FLINK-1675:

             Summary: Rework Accumulators
                 Key: FLINK-1675
                 URL: https://issues.apache.org/jira/browse/FLINK-1675
             Project: Flink
          Issue Type: Bug
          Components: JobManager, TaskManager
    Affects Versions: 0.9
            Reporter: Stephan Ewen
             Fix For: 0.9

The accumulators need an overhaul to address various issues:

1.  User defined Accumulator classes crash the client, because it is not using the user code
classloader to decode the received message.

2.  They should be attached to the ExecutionGraph, not the dedicated AccumulatorManager. That
makes them accessible also for archived execution graphs.

3.  Accumulators should be sent periodically, as part of the heart beat that sends metrics.
This allows them to be updated in real time

4. Accumulators should be stored fine grained (per executionvertex, or per execution) and
the final value should be on computed by merging all involved ones. This allows users to access
the per-subtask accumulators, which is often interesting.

5. Accumulators should subsume the aggregators by allowing to be "versioned" with a superstep.
The versioned ones should be redistributed to the cluster after each superstep.

This message was sent by Atlassian JIRA

View raw message