flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Steven Wu <stevenz...@gmail.com>
Subject how to emit record to downstream operator in snapshotState and/or onProcessingTime
Date Sun, 10 Jun 2018 23:07:27 GMT
I have a process function defined with these interfaces

public class MyProcessFunction<IN> extends ProcessFunction<IN, OUT>
    implements CheckpointedFunction, ProcessingTimeCallback {...}

In snapshotState() method, I want to close files and emit the metadata
about the closed files to downstream operator. it doesn't seem possible
with *snapshotState(FunctionSnapshotContext context*) interface.

I can keep metadata in snapshot and restore them during recovery. but if
there is no input record coming for a long time, * processElement(T value,
Context ctx, Collector<DataFile> out)* won't be called. Then I can't
forward the restored data to downstream operator with guaranteed latency.

I can add a timer. but it doesn't seem that *onProcessingTime(long
timestamp)* allows me to forward output to downstream operator either.

Thanks,
Steven

Mime
View raw message