airavata-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shameera Rathnayaka <shameerai...@gmail.com>
Subject Re: KafkaAppender for Airavata
Date Tue, 04 Oct 2016 18:07:48 GMT
Hi Lahiru,

I didn't see any architectural mail regarding this hence asking, Can you
explain your design and deployment strategy here? few follow-up questions,
how many kafka producers, partitions and consumers in the system?, why you
choose kafka over other messaging systems?

Regards,
Shameera.

On Tue, Oct 4, 2016 at 1:58 PM Lahiru Ginnaliya Gamathige <glahiru@gmail.com>
wrote:

> Hi Suresh,
>
> If logstash see back-pressure then it stops processing the messages so its
> always good to have a persistant storage as the input side of the logstash,
> so logstash can crash anytime and kafka offset works very nicely with the
> cases where logstash crashes and you just have to restart the logstash and
> missed logs will be loaded to ES cluster.
>
> There's nothing like sending the logs *to* logstash because logstash is
> simply a data pipeline and not a data store. The other reason is that if we
> push to kafka we can just handle moving the data to ES with minimum number
> of logstash (theoretically one logstash) where we have the full
> configuration with what topics to listen and where to send those logs.
>
> Regards
> Lahiru
>
>
>
> On Tue, Oct 4, 2016 at 10:13 AM, Suresh Marru <smarru@apache.org> wrote:
>
> Hi Lahiru,
>
> Can you please elaborate on the role of the messaging system (Kafka in
> your example) vs directly using log stash plugins [1] as an example?
>
> Suresh
> [1] -
> https://www.elastic.co/guide/en/logstash/current/plugins-inputs-log4j.html
>
> On Oct 4, 2016, at 12:29 PM, Lahiru Ginnaliya Gamathige <glahiru@gmail.com>
> wrote:
>
> Hi Devs,
>
> I've been working on a feature in Airavata to send its logs to a Kafka
> topic and finally move them to elastic search using logstash. Please find
> all the work[1] and the code[2] and I've added a documentation in to wiki
> [3](since we do not have a development location for wiki).
>
> After finishing the issue, Me and Ajinkya did a setup with Airavata and
> elastic search and Kafka cluster and we were able to visualize the logs in
> kibana.
>
> Please provide your valuable feedback on this feature.
>
> [1]https://issues.apache.org/jira/browse/AIRAVATA-2065
> [2]
> https://github.com/apache/airavata/commit/8eea17fb9c8c073784795dfc56618ec556a12ad1
> [3]
> https://cwiki.apache.org/confluence/display/AIRAVATA/Tutorial+09+-+How+to+setup+centralized+logging+for+Airavata
>
>
> Regards
> Lahiru
>
>
>
> --
Shameera Rathnayaka

Mime
View raw message