pulsar-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] yingfeng edited a comment on issue #3214: [Feature Request]Tiered storage like Pravega
Date Wed, 19 Dec 2018 07:18:23 GMT
yingfeng edited a comment on issue #3214: [Feature Request]Tiered storage like Pravega
URL: https://github.com/apache/pulsar/issues/3214#issuecomment-448491885
   Thank you. There are some other design features for auto tiering:
   1. Retention policy, describes a policy about how much data to retain within a stream.
This is similar with what Pulsar has done to let customer configure how data was migrated
to low tier storage.
   2. Scaling policy, describes if and under what circumstances a stream should `automatically`
scale its number of segments. This is a bit more complicated: the controller monitors traffic
of stream segment writing requests, if some thresholds have been reached, it will decide how
to scale. There are two types of traffic reports that can be received for segments. First
type identifies if a segment should be scaled up (split) and second type identifies if a segment
should be scaled down. For segments eligible for scale up, controller immediately posts request
for segment scale up in the request stream for request event processor to process. However,
for scale down, Controller needs to wait for at least two neighboring segments to become eligible
for scale down. This auto scaling could be seen [here](https://github.com/pravega/pravega/blob/master/documentation/src/docs/pravega-concepts.md#elastic-streams-auto-scaling)
with more detail.
   ![Uploading 图片.png…](https://raw.githubusercontent.com/pravega/pravega/master/documentation/src/docs/img/segment.split.merge.overtime.new.png)
   @sijie I notice that Pravega will also be presented at Flink Forward in the same room,
just after your presentation, perhaps you could get something more face by face :) 

This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:

With regards,
Apache Git Services

View raw message