flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: Periodic flush sink?
Date Sat, 29 Apr 2017 14:57:02 GMT
I expect Flink expert to answer your question.

bq. I get a flush of the buffers atleast every few seconds

>From hbase point of view, during low traffic period, the above may result
in many small hfiles, leading to more work for the compaction.


On Sat, Apr 29, 2017 at 7:32 AM, Niels Basjes <Niels@basjes.nl> wrote:

> Hi,
> I have a sink that writes my records into HBase.
> The data stream is attached to measurements from an internal testing
> instance of the website.
> As a consequence there are periods of really high load (someone is doing a
> load test) and really low load (only a hand full of people are testing
> stuff).
> I read the records from Kafka and I want to write the records into HBase.
> Because under high load it is more efficient to buffer the writes between
> the client and the server and as indicated by HBase I use a BufferedMutator.
> This BufferedMutator works with a 'fixed size' buffer and under high load
> setting it to a few MiB improves the performance writing to HBase greatly.
> However under low load you have to wait until the buffer is full and that
> can be a LONG time (hours) when the load is really low.
> I want to fire a periodic event into my sink to ensure I get a flush of
> the buffers atleast every few seconds.
> Simply implement a standard Java  TimerTask and fire that using a Timer?
> Or is there a better way of doing that in Flink?
> --
> Best regards / Met vriendelijke groeten,
> Niels Basjes

View raw message