flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Igal Shilman <i...@ververica.com>
Subject Re: Stateful functions presentation code (UI part)
Date Fri, 01 Nov 2019 11:20:55 GMT
Hi Flavio, let me try to clarify:

The intention of this example is to demonstrate how
different entities (drivers, passengers, etc') participates in a protocol
(ride matching). For that we have the stateful functions application, and a
standalone java application that just generates the events to trigger the
simulation (i.e. a passenger requests a ride would be an event that the
simulator emits).
The visualization aspect of this example is not a part of what we are
trying to demonstrate but rather, an out-of-band peek into the live system
(websocket that duplicates the events directly to the UI) which we thought
is a nice addition to the example that visualizes how cars are moving on
the grid :-)

If your goal is to create a scalable visualization of car locations, then I
guess your suggestion to break the grid into distinct geo-regions can work.
For that you would need:
1. An additional egress backed by a Kafka topic.
2. Modify the driver function to send location changes to that egress,
where you'd use the region id (geohash with a resolution to your liking) as
key [1].
3. At the consuming side subscribe to the partition that holds the region
of interest.

But, realistically speaking for this kind of a visualization you would be
better off consuming from the locations update topic (from-driver topic)
and populating an optimized geo-indexed store,
and querying it directly.

I hope this clarifies things,
Igal

[1]
https://github.com/ververica/stateful-functions/blob/master/stateful-functions-examples/stateful-functions-ridesharing-example/stateful-functions-ridesharing-example-functions/src/main/java/com/ververica/statefun/examples/ridesharing/KafkaSpecs.java#L121

On Thu, Oct 31, 2019 at 6:21 PM Flavio Pompermaier <pompermaier@okkam.it>
wrote:

> Thanks Igal, this is more or less what I was expecting..this implies that
> ALL events are received on the UI side.
> I was concerned about the tradeoffs of this choice: when I zoom on the map
> I could simply ignore messages outside the boundaries (but I still spend
> many cpu resource in the reading of useless messages).
>
> In the case of a worldwide company (eg uber or simolar) it's probably better
> to create a topic per geographical arean..but also in this case then the UI
> should know when to attach or detach from queue topics when it reach a
> section of the map served by different topics..in another case I could just
> have too many events also in the single map section (let's think about some
> big city with many user).
>
> Is there any talk at FF ir someone else that faced those issues too?
>
> Il Gio 31 Ott 2019, 17:44 Igal Shilman <igal@ververica.com> ha scritto:
>
>> For that particular example, the simulator [1] is responsible for
>> simulating physical drivers and passengers that interact with their
>> corresponding
>> stateful functions [2].
>> The interaction between the simulator and the stateful functions is
>> happening via four Kafka topics:
>> * to-driver - messages that are sent from the physical drivers (the
>> simulator [1]) to the stateful functions. The messages always carry a
>> driver-id which acts as a routing key (I think that this is what you mean
>> by correlation id) to a specific driver stateful function (FnDriver)
>> * from-driver - messages that are sent from a stateful function with a
>> specific driver id to the simulator
>> * to-passenger - symmetric to to-driver
>> * from-passenger - symmetric to from-driver.
>> The ingress and egress definition are specified here [3], and you may
>> want to checkout how to router is defined as well [4][5].
>>
>> In addition the simulator is also feeding the UI directly by duplicating
>> the messages to a web socket (see [6])
>>
>> I hope this clarifies the examples.
>>
>> Igal.
>>
>> [1]
>> https://github.com/ververica/stateful-functions/tree/master/stateful-functions-examples/stateful-functions-ridesharing-example/stateful-functions-ridesharing-example-simulator/src/main/java/com/ververica/statefun/examples/ridesharing/simulator
>> [2]
>> https://github.com/ververica/stateful-functions/tree/master/stateful-functions-examples/stateful-functions-ridesharing-example/stateful-functions-ridesharing-example-functions
>> [3]
>> https://github.com/ververica/stateful-functions/blob/master/stateful-functions-examples/stateful-functions-ridesharing-example/stateful-functions-ridesharing-example-functions/src/main/java/com/ververica/statefun/examples/ridesharing/KafkaSpecs.java#L43
>> [4]
>> https://github.com/ververica/stateful-functions/blob/master/stateful-functions-examples/stateful-functions-ridesharing-example/stateful-functions-ridesharing-example-functions/src/main/java/com/ververica/statefun/examples/ridesharing/Module.java#L33
>> [5]
>> https://github.com/ververica/stateful-functions/blob/master/stateful-functions-examples/stateful-functions-ridesharing-example/stateful-functions-ridesharing-example-functions/src/main/java/com/ververica/statefun/examples/ridesharing/InboundDriverRouter.java#L26
>> [6]
>> https://github.com/ververica/stateful-functions/blob/master/stateful-functions-examples/stateful-functions-ridesharing-example/stateful-functions-ridesharing-example-simulator/src/main/java/com/ververica/statefun/examples/ridesharing/simulator/simulation/Driver.java#L70
>>
>> On Thu, Oct 31, 2019 at 4:15 PM Flavio Pompermaier <pompermaier@okkam.it>
>> wrote:
>>
>>> Yes, I'm interested in how to read data from a UI..which egress should I
>>> use? If we use a kafka queue, how to filter data received in the topic?
>>> Should I use a correlation id or use a new topic per user?
>>>
>>> Il Gio 31 Ott 2019, 16:08 Igal Shilman <igal@ververica.com> ha scritto:
>>>
>>>> Hi Flavio,
>>>>
>>>> We haven't included the UI source code just yet, we've only used it for
>>>> demos and talks.
>>>>
>>>> The reason is that (1) we didn't put a lot of effort and time there (2)
>>>> didn't check the time to go through the individual dependencies and
>>>> licences.
>>>> But we will add that very soon.
>>>>
>>>> Would having the UI code there would improve your understanding? or is
>>>> there another reason?
>>>>
>>>> Thanks,
>>>> Igal
>>>>
>>>> On Thu, Oct 31, 2019 at 5:44 AM Flavio Pompermaier <
>>>> pompermaier@okkam.it> wrote:
>>>>
>>>>> Hi Vino,
>>>>> I already checked that code but I can't find the UI part :(
>>>>>
>>>>> On Thu, Oct 31, 2019 at 12:32 PM vino yang <yanghua1127@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi Flavio,
>>>>>>
>>>>>> Please see this link.[1]
>>>>>>
>>>>>> Best,
>>>>>> Vino
>>>>>>
>>>>>> [1]:
>>>>>> https://github.com/ververica/stateful-functions/tree/master/stateful-functions-examples/stateful-functions-ridesharing-example
>>>>>>
>>>>>> Flavio Pompermaier <pompermaier@okkam.it> 于2019年10月31日周四
下午4:53写道:
>>>>>>
>>>>>>> Hi to all,
>>>>>>> yould it be possible to provide also the source code of the UI
part
>>>>>>> of the ride sharing example? It would be interesting to me how
the UI is
>>>>>>> reading the data from the Kafka egress.
>>>>>>>
>>>>>>> Best,
>>>>>>> Flavio
>>>>>>>
>>>>>>
>>>>>

Mime
View raw message