Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 0F0EC200B8B for ; Tue, 4 Oct 2016 20:28:48 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 0D853160ACC; Tue, 4 Oct 2016 18:28:48 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id CEFE8160AC7 for ; Tue, 4 Oct 2016 20:28:46 +0200 (CEST) Received: (qmail 57037 invoked by uid 500); 4 Oct 2016 18:28:46 -0000 Mailing-List: contact dev-help@airavata.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@airavata.apache.org Delivered-To: mailing list dev@airavata.apache.org Received: (qmail 57027 invoked by uid 99); 4 Oct 2016 18:28:45 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 04 Oct 2016 18:28:45 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 54B1AC0E0C for ; Tue, 4 Oct 2016 18:28:45 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.179 X-Spam-Level: ** X-Spam-Status: No, score=2.179 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_REPLY=1, HTML_MESSAGE=2, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id 1Av5P7d2s1kC for ; Tue, 4 Oct 2016 18:28:43 +0000 (UTC) Received: from mail-oi0-f41.google.com (mail-oi0-f41.google.com [209.85.218.41]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 63D4B5F246 for ; Tue, 4 Oct 2016 18:28:42 +0000 (UTC) Received: by mail-oi0-f41.google.com with SMTP id r126so244618377oib.0 for ; Tue, 04 Oct 2016 11:28:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=nimFhlMJqCHCZIvDKC5RkvqM3GLTH8wPJu4eaon7rK8=; b=LLvSo+i09lSQgE5KGkUVVOAUOMXXSB0MAETlKlzuFXMeSWTRIG9zJjV71NX6LozgVF eCF5V8QlFG6CbK3NxZObsF//GYN7QB795D2RPsEqtDdZbTthJgNWaW8bwbWeWEqKUGDP 4XfURzTfibwSJmC6cavvare7gmASaEZPfiZD2CMvsG6Rrtw1+54k72aboJQgT1i1TzQw UJ2vZLtlWhTVgGkBCyKfV8m7ZeN5OajeNizgiOrVry1d0YDC1Rya1UTkwqOm9mAzkoZc fEBSbPBgj+36S66qrZr0qsCYy0ncWwgwd6D3S6O+ek5AI+dr429W28ILwcm6QBqgNv3u c68g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=nimFhlMJqCHCZIvDKC5RkvqM3GLTH8wPJu4eaon7rK8=; b=mI0QWywX+egabaxOUw9Vhm+JOirOuQcdWRcQM8y6l+QXzPwkgXCyiDTAKwKm9ww9dE 1ZXDy9R0G6hDZupH34zS+NuGwsiq4vPO4OAw2pAVCBE3aO9vwFGzFoCgCh/EVDniffJS LMZUcZvCnJanjFIBH2Iz+5nBNyITDh2d65Qq/MZt4/T8kP6YE5N1SYfZe5N9zg4f2sdt ygrOJIvrSl3pBdWgSgk+PcFGBKHttF6grHvOmtOyxAtTXdL/OCIkytpq41GQJfHUVcrf 1D3z5YRkLY46Wr8BbEzP4LSqBe5eBOsbiOGc/nAcHLdvK1R2pxdp2acUWcA8taATImiu 5orQ== X-Gm-Message-State: AA6/9Rk5D9/SwWSgYCDqAMLPBC7AtLaCJOWgFcTkgoJEfjOLhQhOttHwSku2oA3P9AiLrnKUBn+FHXxiPI8ncQ== X-Received: by 10.157.18.178 with SMTP id g47mr3081440otg.230.1475605721051; Tue, 04 Oct 2016 11:28:41 -0700 (PDT) MIME-Version: 1.0 Received: by 10.182.47.68 with HTTP; Tue, 4 Oct 2016 11:28:40 -0700 (PDT) In-Reply-To: References: From: Lahiru Ginnaliya Gamathige Date: Tue, 4 Oct 2016 11:28:40 -0700 Message-ID: Subject: Re: KafkaAppender for Airavata To: dev Content-Type: multipart/alternative; boundary=94eb2c03ba74d4fbe0053e0e3a29 archived-at: Tue, 04 Oct 2016 18:28:48 -0000 --94eb2c03ba74d4fbe0053e0e3a29 Content-Type: text/plain; charset=UTF-8 On Tue, Oct 4, 2016 at 11:07 AM, Shameera Rathnayaka wrote: > Hi Lahiru, > > I didn't see any architectural mail regarding this hence asking, Can you > explain your design and deployment strategy here? few follow-up questions, > how many kafka producers, partitions and consumers in the system?, why you > choose kafka over other messaging systems? > One producer in-process with Airavata JVM(if we deploy gfac, orchestrator, airavata-api separate then one produce per each component), consumer is in logstash which reads the logs from preconfigured topics. Once we push to Kafka its totally up the the developers who deploy airavata to use it however they want. Using logstash and ES is just one approach which me and Ajinkya tried and a very common solution people use to visualize their logs coming from multiple components. Regarding why Kafka, I've used similar setup and it never gave me a problem and worked with thousands of logs without giving performance problems or issue with the scale of the logs. I did a demo/presentation about this to the IU folks few weeks back and attached the presentation to Jira[1]. [1]https://issues.apache.org/jira/browse/AIRAVATA-2065 Regards Lahiru > > Regards, > Shameera. > > On Tue, Oct 4, 2016 at 1:58 PM Lahiru Ginnaliya Gamathige < > glahiru@gmail.com> wrote: > >> Hi Suresh, >> >> If logstash see back-pressure then it stops processing the messages so >> its always good to have a persistant storage as the input side of the >> logstash, so logstash can crash anytime and kafka offset works very nicely >> with the cases where logstash crashes and you just have to restart the >> logstash and missed logs will be loaded to ES cluster. >> >> There's nothing like sending the logs *to* logstash because logstash is >> simply a data pipeline and not a data store. The other reason is that if we >> push to kafka we can just handle moving the data to ES with minimum number >> of logstash (theoretically one logstash) where we have the full >> configuration with what topics to listen and where to send those logs. >> >> Regards >> Lahiru >> >> >> >> On Tue, Oct 4, 2016 at 10:13 AM, Suresh Marru wrote: >> >> Hi Lahiru, >> >> Can you please elaborate on the role of the messaging system (Kafka in >> your example) vs directly using log stash plugins [1] as an example? >> >> Suresh >> [1] - https://www.elastic.co/guide/en/logstash/current/plugins- >> inputs-log4j.html >> >> On Oct 4, 2016, at 12:29 PM, Lahiru Ginnaliya Gamathige < >> glahiru@gmail.com> wrote: >> >> Hi Devs, >> >> I've been working on a feature in Airavata to send its logs to a Kafka >> topic and finally move them to elastic search using logstash. Please find >> all the work[1] and the code[2] and I've added a documentation in to wiki >> [3](since we do not have a development location for wiki). >> >> After finishing the issue, Me and Ajinkya did a setup with Airavata and >> elastic search and Kafka cluster and we were able to visualize the logs in >> kibana. >> >> Please provide your valuable feedback on this feature. >> >> [1]https://issues.apache.org/jira/browse/AIRAVATA-2065 >> [2]https://github.com/apache/airavata/commit/8eea17fb9c8c073 >> 784795dfc56618ec556a12ad1 >> [3]https://cwiki.apache.org/confluence/display/AIRAVATA/Tuto >> rial+09+-+How+to+setup+centralized+logging+for+Airavata >> >> >> Regards >> Lahiru >> >> >> >> -- > Shameera Rathnayaka > --94eb2c03ba74d4fbe0053e0e3a29 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable


On Tue, Oct 4, 2016 at 11:07 AM, Shameera Rathnayaka = <shameerainf= o@gmail.com> wrote:
Hi Lahiru,=C2=A0

I didn'= ;t see any architectural mail regarding this hence asking, Can you explain = your design and deployment strategy here? few follow-up questions, how many= kafka producers, partitions and consumers in the system?,=C2=A0why you cho= ose=C2=A0kafka=C2=A0over other messaging systems?
One producer in-process with Airavata JVM(if we deploy gfac, o= rchestrator, airavata-api separate then one produce per each component), co= nsumer is in logstash which reads the logs from preconfigured topics. Once = we push to Kafka its totally up the the developers who deploy airavata to u= se it however they want. Using logstash and ES is just one approach which m= e and Ajinkya tried and a very common solution people use to visualize thei= r logs coming from multiple components.

Regarding = why Kafka, I've used similar setup and it never gave me a problem and w= orked with thousands of logs without giving performance problems or issue w= ith the scale of the logs.=C2=A0 I did a demo/presentation about this to th= e IU folks few weeks back and attached the presentation to Jira[1].

Regards
Lahiru

=C2=A0
<= blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-l= eft:1px solid rgb(204,204,204);padding-left:1ex">

=
Regards,
Shameera.=C2=A0

On Tue, Oct 4, 2016 at 1= :58 PM Lahiru Ginnaliya Gamathige <glahiru@gmail.com> wrote:
Hi Suresh,

If logstash s= ee back-pressure then it stops processing the messages so its always good t= o have a persistant storage as the input side of the logstash, so logstash = can crash anytime and kafka offset works very nicely with the cases where l= ogstash crashes and you just have to restart the logstash and missed logs w= ill be loaded to ES cluster.

There's nothing like sending the= logs to logstash because logstash is simply a data pipeline and not a da= ta store. The other reason is that if we push to kafka we can just handle m= oving the data to ES with minimum number of logstash (theoretically one log= stash) where we have the full configuration with what topics to listen and = where to send those logs.=C2=A0

Regards
La= hiru

=C2=A0

On Tue, Oct 4, 2016= at 10:13 AM, Suresh Marru <smarru@apache.org> wrote:
Hi Lahiru,

Can you pl= ease elaborate on the role of the messaging system (Kafka in your example) = vs directly using log stash plugins [1] as an example?

Suresh
[1] -=C2=A0https://www.elastic.co/guide/en/= logstash/current/plugins-inputs-log4j.html

On Oct 4, 2016, at 12:29 PM, Lahiru Ginnaliya Gamathige <glahiru@gmail.com> wrote:
=
Hi Devs,

I've = been working on a feature in Airavata to send its logs to a Kafka topic and= finally move them to elastic search using logstash. Please find all the wo= rk[1] and the code[2] and I've added a documentation in to wiki [3](sin= ce we do not have a development location for wiki).=C2=A0

Afte= r finishing the issue, Me and Ajinkya did a setup with Airavata and elastic= search and Kafka cluster and we were able to visualize the logs in kibana.=

Please provide your valuable feedback on this feature.=C2=A0

[1]https://issues.apache.org/jira/browse/AIRAVATA-2065
[2]https://github.com/apache/airav= ata/commit/8eea17fb9c8c073784795dfc56618ec556a12ad1


Regards
Lahiru


--
Shameera Rathnayaka

--94eb2c03ba74d4fbe0053e0e3a29--