oodt-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tom Barber (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (OODT-946) Implement Kafka Messaging System Within OODT
Date Sun, 19 Mar 2017 23:35:41 GMT

    [ https://issues.apache.org/jira/browse/OODT-946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15932029#comment-15932029

Tom Barber commented on OODT-946:

Hi folks,

Sorry for the delay in getting back to you all.

[~ErangaHeshan] feel free to ping me here or on the OODT developers mailing list http://oodt.apache.org/community.html

[~tugbadogan] I don't have any specific advice but if you have any questions that you want
answering, please ask away.

> Implement Kafka Messaging System Within OODT
> --------------------------------------------
>                 Key: OODT-946
>                 URL: https://issues.apache.org/jira/browse/OODT-946
>             Project: OODT
>          Issue Type: Bug
>            Reporter: Tom Barber
>            Assignee: Tom Barber
>              Labels: gsoc2017, mentor
> Under the hood of Apache OODT we make use of XMLRPC and/or Apache Avro to help with messaging
and communications. 
> Our proposal for Apache OODT 2.0 is to replace some of this underlying communication
trafficking with Apache Kafka. In doing so this will allow multiple components to subscribe
to streams and register their interest. It also allows us to track messages flowing throughout
the system allowing for much improved auditability.
> The initial work would be done in the file manager, replacing some of the internal communications
with Kafka based messaging and also then providing an external interface that other OODT components
can subscribe to allowing for event registration when files are ingested.
> Secondly we would look to replace the messaging within the Workflow Manager with Kafka
there by allowing easier tracabilty of the data flowing through it and again allowing other
components to register themselves against it. This will also make integration of 3rd party
applications easier.
> Part 3 would introduce Kafka based messaging into the resource manager, allowing components
to be aware of the resources available to them in real time.

This message was sent by Atlassian JIRA

View raw message