predictionio-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Thomas POCREAU <thomas.pocr...@iadvize.com>
Subject Re: Kafka support as Event Store
Date Thu, 06 Jul 2017 08:59:04 GMT
After some talks in intern, I misunderstood our needs.
Indeed, we will use Kafka HDFS connector
<http://docs.confluent.io/current/connect/connect-hdfs/docs/index.html> to
dump expired data.
So we will basically have Kafka for the fresh events and HDFS for the past
events.


2017-07-06 7:06 GMT+02:00 Thomas POCREAU <thomas.pocreau@iadvize.com>:

> Hi,
>
> Thanks for your responses.
>
> Our goal is to use kafka as our main event store for event sourcing. I'm
> pretty sure that kafka can be used with an infinite retention time.
>
> We could use KStream and the Java sdk but I would like to give a try to an
> implementation of PStore on top of spark-streaming-kafka (
> https://spark.apache.org/docs/latest/streaming-kafka-0-10-integration.html
> ).
>
> My main concern is related to the http interface to push events. We will
> probably use KStream or websockets to load events in our Kafka topic used
> by an app channel.
>
> Are you planning on supporting websockets as an alternative to batch
> import?
>
> Regards,
> Thomas Pocreau
>
> Le 5 juil. 2017 21:36, "Pat Ferrel" <pat@occamsmachete.com> a écrit :
>
>> No, we try not to fork :-) But it would be nice as you say. It can be
>> done with a small intermediary app that just reads from a Kafka topic and
>> send events to a localhost EventServer, which would allow events to be
>> custom extracted from say log files (typical contents of Kafka). We’ve done
>> this in non-PIO projects.
>>
>> The intermediary app should use Spark streaming. I may have a snippet of
>> code around if you need it but it just saves to micro-batch files. You’d
>> have to use the PIO Java-SDK to send them to the EventServer. A relatively
>> simple thing.
>>
>> Donald, what did you have in mind for deeper integration? I guess we
>> could cut out the intermediate app and integrate into a new Kafka aware
>> EventServer endpoint where the raw topic input is stored in the EventStore.
>> This would force any log filtering onto the Kafka source.
>>
>>
>> On Jul 5, 2017, at 10:20 AM, Donald Szeto <donald@apache.org> wrote:
>>
>> Hi Thomas,
>>
>> Supporting Kafka is definitely interesting and desirable. Are you looking
>> to sinking your Kafka messages to event store for batch processing, or
>> stream processing directly from Kafka? The latter would require more work
>> because Apache PIO does not yet support streaming properly.
>>
>> Folks from ActionML might have a flavor of PIO that works with Kafka.
>>
>> Regards,
>> Donald
>>
>> On Tue, Jul 4, 2017 at 8:34 AM, Thomas POCREAU <
>> thomas.pocreau@iadvize.com> wrote:
>>
>>> Hi,
>>>
>>> Thanks a lot for this awesome project.
>>>
>>> I have a question regarding Kafka and it's possible integration as an
>>> Event Store.
>>> Do you have any plan on this matter ?
>>> Are you aware of someone working on a similar sujet ?
>>>
>>> Regards,
>>> Thomas.
>>>
>>>
>>
>>

Mime
View raw message