manifoldcf-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gustavo Beneitez <gustavo.benei...@gmail.com>
Subject Re: Kafka and logstash integration
Date Mon, 10 Sep 2018 10:02:19 GMT
Hi Piergiorgio,
thanks for your response. I also agree Kafka is best for this solution.

In fact, we are facing Elastic cluster problems and developers told me it
might be possible to persist / distribute messages through kafka instead of
directly dealing with Elastic. Also, someone pointed to a logstash solution
as ELK does, but you are right, Kafka should be the way.
As I understand, logstash should "replace" or "complement" web crawler by
just putting "events" into the system that might be transformed into
documents in parallel with other repositories. I like a lot a passive
connector, some kind of listener that could get individual document
requests and incorporate them to the system, not necessary to be tied with
Logstash.

Best regards!


El vie., 7 sept. 2018 a las 14:25, Piergiorgio Lucidi (<
piergiorgio@apache.org>) escribió:

> Hi Gustavo,
>
> nice to meet you!
>
> We actually have a Kafka Repository Connector as mentioned in our latest
> compatibility matrix:
>
> https://manifoldcf.apache.org/release/release-2.10/en_US/included-connectors.html
>
> Considering Logstash we don't have nothing at the moment but I can imagine
> the scenario behind it.
> The architecture for this case is quite different because ManifoldCF is
> fundamentally a repository crawler and the interaction with logstash should
> be inline from logstash to ManifoldCF that is not actually supported for
> each single transaction.
>
> We could think about a passive connector that remains listening to receive
> new contents, anyway we should start talking about which could the best
> approach for implementing this new behavior.
>
> Am I missing something?
> Hope to receive more comments from other folks in the community :)
>
> Cheers,
> PJ
>
>
>
>
> Il giorno mar 4 set 2018 alle ore 09:50 Gustavo Beneitez <
> gustavo.beneitez@gmail.com> ha scritto:
>
> > Hello Karl,
> >
> > is there any kind of integration in the roadmap between manifoldCF and
> > Kafka or LogStash technologies?
> >
> > We are trying to expand replications with several technologies in mind.
> >
> > Thanks!
> >
>
>
> --
> Piergiorgio Lucidi
> Open Source Evangelist and Digital Transformation Specialist
> Member / Mentor / PMC Member / Committer @ The Apache Software Foundation
> Community Star / Wiki Gardener / Global Forum Moderator @ Alfresco
> Author and Technical Reviewer @ Packt Publishing
> Technical Advisory Group Member @ Microsoft
> Top Community Contributor @ Crafter
> Project Leader / Committer @ JBoss
> https://www.open4dev.com
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message