heron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Saikat Kanjilal <sxk1...@hotmail.com>
Subject Re: Notes on bp(back-pressure) discussion on Nov.17
Date Sat, 18 Nov 2017 01:11:49 GMT
Hi Fu,
Thanks for the proposals, is it possible to outline the pros and cons that you guys discussed?
 That would help the community in weighing in on this.
Thanks in advance

Sent from my iPhone

On Nov 17, 2017, at 4:15 PM, Fu Maosong <maosongfu@gmail.com<mailto:maosongfu@gmail.com>>
wrote:

Today, Ning, Huijun, Sanjeev and Maosong discussed about the
bp(back-pressure) today, and here are the notes:

*Two major issues of current bp algorithm*:
1. one single point of bp will stop the whole topology
2. when a topology in bp state, the overall throughput for the whole
topology can reduce

*Some proposals*:
For 1.
- load shedding -- for example, drop tuples in stmgr.
- better bp algo. -- no need to stop the whole topology; can just handle
the slowness to particular instances, for example, spill the buffer to the
disk

For 2.
- Rate control on the source(spout) side. So even in bp state, the overall
throughput can be higher than normal
- Run-time scale-up

Let me know if I missing anything.

--
With my best Regards
------------------
Fu Maosong
Twitter Inc.
Mobile: +001-415-244-7520
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message