airavata-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Lahiru Ginnaliya Gamathige <glah...@gmail.com>
Subject Re: KafkaAppender for Airavata
Date Fri, 07 Oct 2016 14:15:13 GMT
Created a Jira for RabbitMQ Appender[1], please implement it if you like.


[1]https://issues.apache.org/jira/browse/AIRAVATA-2145

On Fri, Oct 7, 2016 at 7:05 AM, Lahiru Ginnaliya Gamathige <
glahiru@gmail.com> wrote:

>
>
> On Thu, Oct 6, 2016 at 6:33 AM, Shameera Rathnayaka <
> shameerainfo@gmail.com> wrote:
>
>>
>>
>> On Tue, Oct 4, 2016 at 2:28 PM Lahiru Ginnaliya Gamathige <
>> glahiru@gmail.com> wrote:
>>
>>> On Tue, Oct 4, 2016 at 11:07 AM, Shameera Rathnayaka <
>>> shameerainfo@gmail.com> wrote:
>>>
>>> Hi Lahiru,
>>>
>>> I didn't see any architectural mail regarding this hence asking, Can you
>>> explain your design and deployment strategy here? few follow-up questions,
>>> how many kafka producers, partitions and consumers in the system?, why you
>>> choose kafka over other messaging systems?
>>>
>>> One producer in-process with Airavata JVM(if we deploy gfac,
>>> orchestrator, airavata-api separate then one produce per each component),
>>> consumer is in logstash which reads the logs from preconfigured topics.
>>> Once we push to Kafka its totally up the the developers who deploy airavata
>>> to use it however they want. Using logstash and ES is just one approach
>>> which me and Ajinkya tried and a very common solution people use to
>>> visualize their logs coming from multiple components.
>>>
>>
>> How many instances need to deploy your logging solution in productions?
>> How Simple/Complex compare to Airavata deployment?
>>
> These are completely deployment level details which someone needs to
> experiment with based on the load of Airavata deployment, for the setup I
> tested I used 3 node kafka cluster with 25GB in each and I think its more
> than enough but completely depends on the deployment.
>
>>
>>
>
>>> Regarding why Kafka, I've used similar setup and it never gave me a
>>> problem and worked with thousands of logs without giving performance
>>> problems or issue with the scale of the logs.  I did a demo/presentation
>>> about this to the IU folks few weeks back and attached the presentation to
>>> Jira[1].
>>>
>>
>> Just because Kafka works doesn't mean kafka is the best solution for
>> Airavata.
>>
> Yes obvious statement.
>
>> If you explain little bit the setup you have worked on, that would help
>> us to get some ideas. What is the other advantage make Kafka is the best
>> one here instead of RabbitMQ (Which we already use in Airavata for
>> messaging) ?
>>
>> Yes, having RabbitMQ appender is also good, so if some user wants to use
> RabbitMQ they can use if someone wants Kafka they can use Kafka. I think we
> should support Kafka for internal communication too. Having KafkaAppender
> will not stop you from implementing more Appenders, there's more
> improvements can be done in this area to provide better infrastructure with
> Airavata.
>
>> Regards,
>> Shameera.
>>
>>
>>>
>>> [1]https://issues.apache.org/jira/browse/AIRAVATA-2065
>>> Regards
>>> Lahiru
>>>
>>>
>>>
>>>
>>> Regards,
>>> Shameera.
>>>
>>> On Tue, Oct 4, 2016 at 1:58 PM Lahiru Ginnaliya Gamathige <
>>> glahiru@gmail.com> wrote:
>>>
>>> Hi Suresh,
>>>
>>> If logstash see back-pressure then it stops processing the messages so
>>> its always good to have a persistant storage as the input side of the
>>> logstash, so logstash can crash anytime and kafka offset works very nicely
>>> with the cases where logstash crashes and you just have to restart the
>>> logstash and missed logs will be loaded to ES cluster.
>>>
>>> There's nothing like sending the logs *to* logstash because logstash is
>>> simply a data pipeline and not a data store. The other reason is that if we
>>> push to kafka we can just handle moving the data to ES with minimum number
>>> of logstash (theoretically one logstash) where we have the full
>>> configuration with what topics to listen and where to send those logs.
>>>
>>> Regards
>>> Lahiru
>>>
>>>
>>>
>>> On Tue, Oct 4, 2016 at 10:13 AM, Suresh Marru <smarru@apache.org> wrote:
>>>
>>> Hi Lahiru,
>>>
>>> Can you please elaborate on the role of the messaging system (Kafka in
>>> your example) vs directly using log stash plugins [1] as an example?
>>>
>>> Suresh
>>> [1] - https://www.elastic.co/guide/en/logstash/current/plugins-
>>> inputs-log4j.html
>>>
>>> On Oct 4, 2016, at 12:29 PM, Lahiru Ginnaliya Gamathige <
>>> glahiru@gmail.com> wrote:
>>>
>>> Hi Devs,
>>>
>>> I've been working on a feature in Airavata to send its logs to a Kafka
>>> topic and finally move them to elastic search using logstash. Please find
>>> all the work[1] and the code[2] and I've added a documentation in to wiki
>>> [3](since we do not have a development location for wiki).
>>>
>>> After finishing the issue, Me and Ajinkya did a setup with Airavata and
>>> elastic search and Kafka cluster and we were able to visualize the logs in
>>> kibana.
>>>
>>> Please provide your valuable feedback on this feature.
>>>
>>> [1]https://issues.apache.org/jira/browse/AIRAVATA-2065
>>> [2]https://github.com/apache/airavata/commit/8eea17fb9c8c073
>>> 784795dfc56618ec556a12ad1
>>> [3]https://cwiki.apache.org/confluence/display/AIRAVATA/Tuto
>>> rial+09+-+How+to+setup+centralized+logging+for+Airavata
>>>
>>>
>>> Regards
>>> Lahiru
>>>
>>>
>>>
>>> --
>>> Shameera Rathnayaka
>>>
>>> --
>> Shameera Rathnayaka
>>
>
>

Mime
View raw message