flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tzu-Li (Gordon) Tai (JIRA)" <j...@apache.org>
Subject [jira] [Created] (FLINK-8421) HeapInternalTimerService should reconfigure compatible key / namespace serializers on restore
Date Fri, 12 Jan 2018 16:30:00 GMT
Tzu-Li (Gordon) Tai created FLINK-8421:
------------------------------------------

             Summary: HeapInternalTimerService should reconfigure compatible key / namespace
serializers on restore
                 Key: FLINK-8421
                 URL: https://issues.apache.org/jira/browse/FLINK-8421
             Project: Flink
          Issue Type: Bug
    Affects Versions: 1.4.0, 1.5.0
            Reporter: Tzu-Li (Gordon) Tai
            Priority: Blocker
             Fix For: 1.3.3, 1.5.0, 1.4.1


The {{HeapInternalTimerService}} still uses simple {{equals}} checks on restored / newly provided
serializers for compatibility checks. This should be replaced with the {{TypeSerializer::ensureCompatibility}}
checks instead, so that new serializers can be reconfigured.

For Flink 1.4.0 release and current master, this is a critical bug since the {{KryoSerializer}}
has different default base registrations than before due to FLINK-7420. i.e if the key of
a window is serialized using the {{KryoSerializer}} in 1.3.x, the restore would never succeed
in 1.4.0.

For 1.3.x, this fix would be an improvement, such that the {{HeapInternalTimerService}} restore
will make use of serializer reconfiguration.

Other remarks:
* We need to double check all operators that checkpoint / restore from **raw** state. Apparently,
the serializer compatibility checks were only implemented for managed state.
* Migration ITCases apparently do not have enough coverage. A migration test job that uses
a key type which required the {{KryoSerializer}}, and uses windows, would have caught this
issue.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message