airavata-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shameera Rathnayaka <shameerai...@gmail.com>
Subject Re: KafkaAppender for Airavata
Date Fri, 07 Oct 2016 16:05:45 GMT
On Fri, Oct 7, 2016 at 10:15 AM Lahiru Ginnaliya Gamathige <
glahiru@gmail.com> wrote:

> Created a Jira for RabbitMQ Appender[1], please implement it if you like.
>

It is always good to plug differnt implementations, If I am going to
implement a new logging appender(RabbitMQAppender or any other). How I can
plug it to Airavata? Can we easily plug differnt implementations? It seem
you have code agains the kafka logging implementation instead of an
Interface, correct me if I am wrong.  I think if you can provide better
solution to plug different appenders this task would be more easier.




>
>
> [1]https://issues.apache.org/jira/browse/AIRAVATA-2145
>
> On Fri, Oct 7, 2016 at 7:05 AM, Lahiru Ginnaliya Gamathige <
> glahiru@gmail.com> wrote:
>
>
>
> On Thu, Oct 6, 2016 at 6:33 AM, Shameera Rathnayaka <
> shameerainfo@gmail.com> wrote:
>
>
>
> On Tue, Oct 4, 2016 at 2:28 PM Lahiru Ginnaliya Gamathige <
> glahiru@gmail.com> wrote:
>
> On Tue, Oct 4, 2016 at 11:07 AM, Shameera Rathnayaka <
> shameerainfo@gmail.com> wrote:
>
> Hi Lahiru,
>
> I didn't see any architectural mail regarding this hence asking, Can you
> explain your design and deployment strategy here? few follow-up questions,
> how many kafka producers, partitions and consumers in the system?, why you
> choose kafka over other messaging systems?
>
> One producer in-process with Airavata JVM(if we deploy gfac, orchestrator,
> airavata-api separate then one produce per each component), consumer is in
> logstash which reads the logs from preconfigured topics. Once we push to
> Kafka its totally up the the developers who deploy airavata to use it
> however they want. Using logstash and ES is just one approach which me and
> Ajinkya tried and a very common solution people use to visualize their logs
> coming from multiple components.
>
>
> How many instances need to deploy your logging solution in productions?
> How Simple/Complex compare to Airavata deployment?
>
> These are completely deployment level details which someone needs to
> experiment with based on the load of Airavata deployment, for the setup I
> tested I used 3 node kafka cluster with 25GB in each and I think its more
> than enough but completely depends on the deployment.
>
>
>
>
> Regarding why Kafka, I've used similar setup and it never gave me a
> problem and worked with thousands of logs without giving performance
> problems or issue with the scale of the logs.  I did a demo/presentation
> about this to the IU folks few weeks back and attached the presentation to
> Jira[1].
>
>
> Just because Kafka works doesn't mean kafka is the best solution for
> Airavata.
>
> Yes obvious statement.
>
> If you explain little bit the setup you have worked on, that would help us
> to get some ideas. What is the other advantage make Kafka is the best one
> here instead of RabbitMQ (Which we already use in Airavata for messaging) ?
>
> Yes, having RabbitMQ appender is also good, so if some user wants to use
> RabbitMQ they can use if someone wants Kafka they can use Kafka. I think we
> should support Kafka for internal communication too. Having KafkaAppender
> will not stop you from implementing more Appenders, there's more
> improvements can be done in this area to provide better infrastructure with
> Airavata.
>
> Regards,
> Shameera.
>
>
>
> [1]https://issues.apache.org/jira/browse/AIRAVATA-2065
> Regards
> Lahiru
>
>
>
>
> Regards,
> Shameera.
>
> On Tue, Oct 4, 2016 at 1:58 PM Lahiru Ginnaliya Gamathige <
> glahiru@gmail.com> wrote:
>
> Hi Suresh,
>
> If logstash see back-pressure then it stops processing the messages so its
> always good to have a persistant storage as the input side of the logstash,
> so logstash can crash anytime and kafka offset works very nicely with the
> cases where logstash crashes and you just have to restart the logstash and
> missed logs will be loaded to ES cluster.
>
> There's nothing like sending the logs *to* logstash because logstash is
> simply a data pipeline and not a data store. The other reason is that if we
> push to kafka we can just handle moving the data to ES with minimum number
> of logstash (theoretically one logstash) where we have the full
> configuration with what topics to listen and where to send those logs.
>
> Regards
> Lahiru
>
>
>
> On Tue, Oct 4, 2016 at 10:13 AM, Suresh Marru <smarru@apache.org> wrote:
>
> Hi Lahiru,
>
> Can you please elaborate on the role of the messaging system (Kafka in
> your example) vs directly using log stash plugins [1] as an example?
>
> Suresh
> [1] -
> https://www.elastic.co/guide/en/logstash/current/plugins-inputs-log4j.html
>
> On Oct 4, 2016, at 12:29 PM, Lahiru Ginnaliya Gamathige <glahiru@gmail.com>
> wrote:
>
> Hi Devs,
>
> I've been working on a feature in Airavata to send its logs to a Kafka
> topic and finally move them to elastic search using logstash. Please find
> all the work[1] and the code[2] and I've added a documentation in to wiki
> [3](since we do not have a development location for wiki).
>
> After finishing the issue, Me and Ajinkya did a setup with Airavata and
> elastic search and Kafka cluster and we were able to visualize the logs in
> kibana.
>
> Please provide your valuable feedback on this feature.
>
> [1]https://issues.apache.org/jira/browse/AIRAVATA-2065
> [2]
> https://github.com/apache/airavata/commit/8eea17fb9c8c073784795dfc56618ec556a12ad1
> [3]
> https://cwiki.apache.org/confluence/display/AIRAVATA/Tutorial+09+-+How+to+setup+centralized+logging+for+Airavata
>
>
> Regards
> Lahiru
>
>
>
> --
> Shameera Rathnayaka
>
> --
> Shameera Rathnayaka
>
>
>
> --
Shameera Rathnayaka

Mime
View raw message