oodt-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eranga Heshan <eranga....@gmail.com>
Subject Re: [GSoC] [946] Implement Kafka Messaging System Within OODT
Date Mon, 03 Apr 2017 12:49:52 GMT
Hi Tom,

I uploaded my final proposal to the GSoC Dashboard a few minutes ago.

I have followed these two architecture files until now to understand how
internal communications of OODT actually works.

   - File Manager:
   http://oodt.apache.org/site_docs/cas-filemgr/development/developer.html
   - Workflow Manager:
   https://oodt.apache.org/site_docs/cas-workflow/development/developer.html

However, I could not find an architecture diagram for Resource Manager yet.
I would be grateful if you could share the architecture document of
Resource Manager if any.

Finally, I would be much grateful if you could read my proposal completely
and understand my capability of completing this project successfully. :-)

Thanks and best regards,




Eranga Heshan
*Undergraduate*
Computer Science & Engineering
University of Moratuwa
Mobile:  +94 71 138 2686 <%2B94%2071%20552%202087>
Email: eranga.h.n@gmail.com
<https://www.facebook.com/erangaheshan>   <https://twitter.com/erangaheshan>
   <https://www.linkedin.com/in/erangaheshan>

On Sat, Apr 1, 2017 at 1:14 PM, Eranga Heshan <eranga.h.n@gmail.com> wrote:

> Hi Tom,
>
> I shared my draft proposal with you in Apache Software Foundation through
> the GSoC dashboard. You can view it from there also. I really appreciate
> your comments and feedback on it.
>
> I will submit my final proposal after resolving all your suggestions.
>
> Thanks & Regards,
>
>
>
> Eranga Heshan
> *Undergraduate*
> Computer Science & Engineering
> University of Moratuwa
> Mobile:  +94 71 138 2686 <%2B94%2071%20552%202087>
> Email: eranga.h.n@gmail.com
> <https://www.facebook.com/erangaheshan>
> <https://twitter.com/erangaheshan>
> <https://www.linkedin.com/in/erangaheshan>
>
> On Sat, Apr 1, 2017 at 3:41 AM, Eranga Heshan <eranga.h.n@gmail.com>
> wrote:
>
>> Hi Tom,
>>
>> Thank you for your quick reply.
>>
>> I re-created the design in my draft proposal since I had misunderstood
>> some concepts about OODT. I really appreciate if you could have a look into my
>> draft proposal
>> <https://docs.google.com/document/d/1BP55CNevap-88Vd7gPzOd2trqvAOHb2e3_jfoazA9Q8>
>> again and advice me further.
>>
>> Please let me bother you again for some questions. :-)
>>
>>    1. What is meant by "ingest hook" in your previous mail?
>>    2. I am not much familiar with Workflow Engine and Resource Manager
>>    at this point. Hence, I have added only the proposed Kafka messaging system
>>    for the File Manager in my design diagram. Until I catch up with those,
>>    would that be enough?
>>
>> Thanks & Regards,
>>
>>
>>
>>
>> Eranga Heshan
>> *Undergraduate*
>> Computer Science & Engineering
>> University of Moratuwa
>> Mobile:  +94 71 138 2686 <%2B94%2071%20552%202087>
>> Email: eranga.h.n@gmail.com
>> <https://www.facebook.com/erangaheshan>
>> <https://twitter.com/erangaheshan>
>> <https://www.linkedin.com/in/erangaheshan>
>>
>> On Fri, Mar 31, 2017 at 8:01 AM, Tom Barber <tom.barber@meteorite.bi>
>> wrote:
>>
>>> Hi Eranga
>>>
>>> Sorry for the delay on getting back to you.
>>>
>>> OODT 2.0 is basically whats on the development branch of the repository:
>>> https://github.com/apache/oodt
>>>
>>> As for communications basically you have a File Manager Client and File
>>> Manager Server, the server currently speaks XMLRPC or Avro. In terms of
>>> communication the most obvious path would be: https://github.com/apache/
>>> oodt/blob/master/filemgr/src/main/java/org/apache/oodt/cas/f
>>> ilemgr/system/XmlRpcFileManager.java XML encoded products come into the
>>> file manager, then depending on what your output source would be the
>>> metadata is written to either Lucene, Solr or something custom: eg:
>>> https://github.com/apache/oodt/blob/master/filemgr/src/m
>>> ain/java/org/apache/oodt/cas/filemgr/catalog/solr/SolrCatalo
>>> gFactory.java
>>> Personally I'd be interested in seeing those calls get managed by Kafka,
>>> so that for example multiple backends could register with an ingest hook,
>>> or you could have an audit listener trigger when kafka messages are passed
>>> for an audit log etc.
>>>
>>> I think that diagram is pretty good, basically, comms are mostly passed
>>> between components via RPC calls, but it would certainly be more effective
>>> for both inter component calls and internal bus calls, for example on the
>>> workflow engine where objects are passed around on a trigger type
>>> mechanism, for these to be replaced by Kafka messaging.
>>>
>>> Tom
>>>
>>>
>>> On Thu, Mar 30, 2017 at 6:21 PM, Eranga Heshan <eranga.h.n@gmail.com>
>>> wrote:
>>>
>>>> Hi Tom,
>>>>
>>>> I am currently looking into http://oodt.apache.org/site_do
>>>> cs/cas-filemgr/development/developer.html to get an abstract idea of
>>>> how OODT File Manager handles its internal communications. I hope that
>>>> would help me to identify which components needed to be implemented in
>>>> Kafka. If my approach is wrong please help me to make it correct.
>>>>
>>>> Further, I really appreciate if you could have a look into my proposal
>>>> and give your opinions on how to make it strong.
>>>>
>>>> Thanks & Regards,
>>>>
>>>>
>>>
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message