flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From StephanEwen <...@git.apache.org>
Subject [GitHub] flink issue #6075: [FLINK-9407] [hdfs connector] Support orc rolling sink wr...
Date Wed, 25 Jul 2018 09:19:46 GMT
Github user StephanEwen commented on the issue:

    @zhangminglei  For your interest - there is a new Bucketing Sink in the Flink master (called
`StreamingFileSink`), with a different design: Managing all state in Flink state (so it is
consistent), with a new File System writer abstraction to generalize across HDFS, POSIX, and
S3 (S3 still WIP) and with a more pluggable way to add encoders, like parquet and orc.
    As an example, we added a Parquet writer, which is quite straightforward and flexible
with the new interface.
    Would be great to get your opinion on that and see if your ORC writer code also works
with that.
    If it works out, the new StreamingFileSink could replace the current BucketingSink.


View raw message