apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Atri Sharma <atri.j...@gmail.com>
Subject Re: Window Commits
Date Wed, 26 Aug 2015 06:37:19 GMT
On Wed, Aug 26, 2015 at 3:52 AM, Chetan Narsude <chetan@datatorrent.com>

> I have a hunch that there may be a problem in terms of adding the latency.
> But ultimately we will use benchmark to rule out the hunches if you
> strongly believe in it.

I am definitely not dead in support for this. I feel this is also a trial
and error based patch. I may be completely wrong here.

> Here is what happens today: bufferserver tries to hold the data in memory
> for as long as possible but not longer than needed. If you do not persist
> the data to memory, you do not have to load it as well as it's already in
> memory. This greatly reduces the disk related latency. Even when we have to
> persist the data, we pick the block (it's pending correct implementation),
> which we will not need back in memory immediately.

Yes, but then you may be open for problems under load and limited memory
(since you cannot swap in more data into bufferserver because there are
write pages pending.

Can you describe a bit on how the block is picked, please? This sounds
really interesting! Is there somewhere I can read about it/ read the code?

> The converse of it is presumably true as well. If you start persisting the
> data in anticipation of buffer being full, you will also need to load this
> data back when needed. This will result in frequent round-trips to disk
> adding to the latency.
Agreed, but those data loads might be smaller and most respectful towards
locality of reference.



  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message