Mailing-List: contact infrastructure-dev-help@apache.org; run by ezmlm
Precedence: bulk
Reply-To: infrastructure-dev@apache.org
From: Christofer Dutz <christofer.dutz@c-ware.de>
To: "infrastructure-dev@apache.org" <infrastructure-dev@apache.org>
Subject: AW: Tool proposal for helping run and monitor the ASF Infra Services
Thread-Topic: Tool proposal for helping run and monitor the ASF Infra Services
Thread-Index: AQHSACsXT+Ufefpl3EKCXbECIzzPx6BdC8ejgAAenoCAAAFHmA==
Date: Sat, 27 Aug 2016 19:07:25 +0000
Message-ID: <HE1PR0501MB24281114C37AA4E88B2AD00BA2EF0@HE1PR0501MB2428.eurprd05.prod.outlook.com>
References: <347EE69B-877E-40D9-A407-5F77C373952B@codecentric.de>
 <HE1PR0501MB242837434E73B4A7D5F3923CA2EA0@HE1PR0501MB2428.eurprd05.prod.outlook.com>
 <8E4C8591-491C-47FB-99EB-BF3F5C88CA35@apache.org>
 <HE1PR0501MB2428C56109403F2E29C33111A2EF0@HE1PR0501MB2428.eurprd05.prod.outlook.com>,<BAF56BB3-8224-403F-8AAC-9CE81603CA17@apache.org>
In-Reply-To: <BAF56BB3-8224-403F-8AAC-9CE81603CA17@apache.org>
Accept-Language: de-DE, en-US
Content-Language: de-DE
received-spf: None (protection.outlook.com: c-ware.de does not designate
 permitted sender hosts)
spamdiagnosticoutput: 1:99
spamdiagnosticmetadata: NSPM
Content-Type: multipart/alternative;
	boundary="_000_HE1PR0501MB24281114C37AA4E88B2AD00BA2EF0HE1PR0501MB2428_"
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-originalarrivaltime: 27 Aug 2016 19:07:25.7948
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 9d387546-1437-4b89-846c-691d64a7e74d
X-MS-Exchange-Transport-CrossTenantHeadersStamped: HE1PR0501MB2427
archived-at: Sat, 27 Aug 2016 19:07:43 -0000

--_000_HE1PR0501MB24281114C37AA4E88B2AD00BA2EF0HE1PR0501MB2428_
Content-Type: text/plain; charset="Windows-1252"
Content-Transfer-Encoding: quoted-printable

Hi Chris,


that sounds like a good option. A test Jira should provide several services=
 that Instana can monitor.


I'll be consumed by some paid work on Monday and Tuesday and I'll try to ge=
t started on this on Wednesday. So I guess you can be expecting some questi=
ons from me then ;-)


Chris

________________________________
Von: Chris Lambertus <cml@apache.org>
Gesendet: Samstag, 27. August 2016 21:01:07
An: infrastructure-dev@apache.org
Betreff: Re: Tool proposal for helping run and monitor the ASF Infra Servic=
es

I have a few test VMs - if you want to develop a puppet module for this in =
the same style as we=92ve set up datadog, that would be excellent. I think =
our Jira test instance would be a good candidate for this on the linux side=
. Feel free to hit me up on hipchat or direct email to discuss particulars =
about the puppet setup.

Thanks,
-Chris


> On Aug 27, 2016, at 10:29 AM, Christofer Dutz <christofer.dutz@c-ware.de>=
 wrote:
>
> Hi Chris,
>
>
> Ok so I started the Agent ... the Windows build agent is rather unexcitin=
g as it only has a hand full of java processes and a MsSQL Server with all =
network shut down. When encountering software, that is explicitly instrumen=
ted such as an Apache Webserver, Tomcat, the introspection will provide mor=
e insight.
>
>
> Eventually I could whip up a puppet script which you could use to setup t=
he Instana Agent on a linux machine. Then you should be able to setup an ag=
ent on one of the more interesting machines. Are there any test VMs I could=
 test such a script on?
>
>
> I just sent you (Chris L.) an invitation to the Instana Web Console, so f=
eel free to sign up. If anyone else is interested, I'll be glad to invite h=
im/her too :-)
>
>
> The "Datacenter A / Rack 42" is just a dummy as I don't have any clue abo=
ut what datacenter the machine/vm is running on.
>
>
> Chris
>
> ________________________________
> Von: Chris Lambertus <cml@apache.org>
> Gesendet: Samstag, 27. August 2016 08:20:15
> An: infrastructure-dev@apache.org
> Cc: Christofer Dutz
> Betreff: Re: Tool proposal for helping run and monitor the ASF Infra Serv=
ices
>
> Chris,
>
> We=92re interested, but we don=92t really have the bandwidth to set up ev=
aluations for tools like this right now. If you truly believe it=92s worthw=
hile, then go ahead and fire up the agent on hudson-win and let=92s see wha=
t we get out of it. If it doesn=92t require any elevated permissions, I=92m=
 happy to see a live demo.
>
>
>
>
> On Aug 23, 2016, at 11:46 PM, Christofer Dutz <christofer.dutz@c-ware.de<=
mailto:christofer.dutz@c-ware.de>> wrote:
>
> Hi all,
>
>
> How about trying it out for one chain of services? I bet that would expla=
in a lot more than anything else? As I mentioned, the SAAS account is alrea=
dy setup and I can create some user accounts for you guys.
>
>
> On the windows1 machine (hudson-win.apache.org<http://hudson-win.apache.o=
rg/> in directory "f:\instana-agent") I have the agent installed (but inact=
ive) so you could have a look at the agent itself.
>
>
> We could fire up the agent and start a Jenkins build and it will definite=
ly show us communication with the jenkins machine, repository.apache.org<ht=
tp://repository.apache.org/> and the proxy (Well probably with the proxy). =
So if we setup the agent on windows, the proxy and repository.apache.org<ht=
tp://repository.apache.org/> I guess this should give a first impression. A=
nd if you don't like or want it in the end, just stop and delete the agent =
and no harm was done. What do you think?
>
>
> I would be glad to assist you in this.
>
>
> Chris
>
> ________________________________
> Von: Mirko Novakovic <mirko.novakovic@codecentric.de<mailto:mirko.novakov=
ic@codecentric.de>>
> Gesendet: Dienstag, 23. August 2016 16:04:52
> An: infrastructure-dev@apache.org<mailto:infrastructure-dev@apache.org>
> Betreff: Re: AW: Tool proposal for helping run and monitor the ASF Infra =
Services
>
> Hi Daniel,
>
> I am the Instana CEO and happy to answer your questions. First of all I w=
ould like to point out that we are an APM company/product, so more in the s=
pace of NewRelic than DataDog. But as we are monitoring the application as =
a whole (and model a dynamic dependency graph for it) we also monitor the i=
nfrastructure. And have out of the box dashboards etc but the core is our d=
istributed tracing and service quality management - all automatic and witho=
ut configuration.
>
> Regarding the questions:
>
> - I can guarantee that there is no time limit on this offering for Apache=
 - the only thing we should discuss is the number of components you would m=
onitor, as it is SAAS there should be some maximum defined so that the cost=
s for us do not explode. If you want to have Instana On Premise we wouldn=
=92t need that limit.
>
> - Our whole system is streaming based, so we stream the data form the age=
nt to the server and from the server the data is pushed to the UI. On the b=
ackend we build a graph and apply a knowledge base on each component to fin=
d changes and issues in realtime. Based on service KPIs (Four Golden Signal=
s by Google) we identify if the changes/issues are having impact on the ser=
vice quality - if yes we report. The algorithms are based on machine learni=
ng, so we see things like sudden drops, slow responses, high error rates, e=
tc etc
>
> - Don=92t know Circonus in details, so I cannot really compare it. Instan=
a for sure is not time consuming as it was designed for auto discovery and =
zero configuration.
>
> - We have different types of integration using API and SDK. For alarms we=
 have out of the box integrations for PagerDuty, OpsGenie and Slack, but al=
so provide a generic Webhook to integrated with any other system that has a=
n API.
>
> - The agent is Apache Karaf based, can update itself using a private or p=
ublic sensor (plugin) repository that is based on Apache Maven technology. =
We provide a set of sensors for approx 40 technologies right now but enable=
 the users to extend this with own sensors.
>
> We can give you more insights or a demo at any time.
>
> Best regards
> Mirko
>
>
> On 2016-08-19 09:16 (+0200), Daniel Gruno <h...@apache.org<http://apache.=
org>> wrote:
> On 08/19/2016 08:20 AM, Christofer Dutz wrote:>
> Hi Chris,>
>
> I knew that someone asked exactly the "how does it compare to datadog" qu=
estion somewhere. Here's the link to that mail thread https://news.ycombina=
tor.com/item?id=3D12147219>
>
> And I can confirm the shortcomings of the time series approach, cause in =
jenkins, I'd say about 70% of recent failures of flex builds were due to ti=
meouts when uploading Maven artifacts to nexus. The current solution doesn'=
t seem to detect that. Not only that I couldn't see any hipchat notificatio=
ns. The infra guys always had to start looking for the real reason of the t=
imeouts as nexus wasn't having any problems at all.>
>
> I'd say the real reason we weren't being notified is because httpd>
> wasn't telling us it was jammed, so it would have made little difference>
> whether we used product X or Y to monitor it, when it wasn't sending out>
> data that could lead us to what the problem was. It wasn't a question of>
> granularity, the data just wasn't enabled for any agent to see.>
>
> I would imagine this would be something _instead_ of datadog, which>
> leads me to some questions:>
>
> - For how long into the future could we have a guarantee that this isn't>
> gonna cost us $$$/year or whatever the price would end up being with a>
> non-free version?>
>
> - What is understood by real time checks here? and what exactly is checke=
d?>
>
> - How does this relate to more advanced monitoring systems like>
> Circonus? As you may know, we had to drop that as it proved to be rather>
> time consuming.>
>
> - What sort of integrations does this system have? How are alerts>
> dispatched?>
>
> I'd also be interested in learning about plugins and how to customize>
> the agents.>
>
> With regards,>
> Daniel.>
>
>
> And I really live the feature of tracking down the response time for one =
service back to other servers to find it the real reason for a system being=
 slow (have a look in the presentation for this. There's a great slide on t=
his)>
>
> Chris>
>
>
> Von meinem Samsung Galaxy Smartphone gesendet.>
>
>
> -------- Ursprngliche Nachricht -------->
> Von: Chris Lambertus <cm...@apache.org<http://apache.org>>>
> Datum: 19.08.16 07:03 (GMT+01:00)>
> An: infrastructure-dev@apache.org<mailto:infrastructure-dev@apache.org>, =
Christofer Dutz <ch...@c-ware.de<http://c-ware.de>>>
> Cc: mirko.novakovic@codecentric.de<mailto:mirko.novakovic@codecentric.de>=
>
> Betreff: Re: Tool proposal for helping run and monitor the ASF Infra Serv=
ices>
>
>
>
> Hiya Chris,>
>
> Thanks for the info and the legwork on this. We currently use DataDog, wh=
ich is very similar to what Instana appears to provide an agent-based monit=
oring solution that gives us that kind of look into our infra. We also have=
 a number of internal tools that report on various goings-on as well. You m=
ight see some of this in #asfinfra on hipchat from SNMP2HipChat. DataDog al=
so reports various problems there, as does our monitoring via PingMyBox.>
>
> Since youre not root@, you may not see some of the stuff that we see, but=
 I think by and large, the majority of the monitors do direct to #asfinfra.=
 Have you noticed gaps in the monitoring? Since we moved to DataDog, weve b=
een quite happy with the resolution and metrics weve been able to get. Its =
been on my back burner for awhile to expose some of our DD dashboards as pu=
blic, but for right now its somewhat limited access. In the interests of tr=
ansparency (but not at the expense of security,) Id be happy to work with y=
ou to expose more of this, and Im happy to address any questions or concern=
s about shortcomings in our monitoring.>
>
> Many thanks to Instana for offering the ASF free services! Id definitely =
like to hear more about what they might be able to offer on top of what we =
already get from DataDog. Ill take a look at the info you sent out. Please =
feel free to follow up with me directly, either via email or hipchat.>
>
> Cheers,>
> -Chris>
>
>
>
>
> On Aug 18, 2016, at 2:13 AM, Christofer Dutz <ch...@c-ware.de<http://c-wa=
re.de>> wrote:>
>
> Hi,>
>
>
>
> I have been on the Infra Hipchat for a few weeks now while trying to migr=
ate the Flex project to Maven and back to the ASF Infra build system. Thank=
s for your support in this and even more thanks for the trust in granting m=
e access and Admin rights on the windows1 build agent.>
>
>
>
> In the chat I observed the daily work of you guys, having to maintain qui=
te a zoo of all sorts of different systems on different platforms. Some pro=
blems you were having seem quite easy to track down ... if the hard disk is=
 full, you clean up. But not all problems are that easy to track down. Thin=
king of the problems with repository.apache.org<http://repository.apache.or=
g> ... here the cause was the proxy being flooded with connections (I think=
 this was the case) ... regular restarts of this helped temporarily, but I =
don't think that helps on the long term as no one had an idea why those con=
nections were hanging there in the first place.>
>
>
>
> A few years ago the company I work for - codecentric - have founded a com=
pany called Instana. They are developing an agent based system for monitori=
ng IT infrastructure. In contrast to most established solutions, they use m=
achine learning strategies to analyze the root cause for problems. While yo=
u can probably achieve similar results with normal tools, the problem is th=
at you need a very detailed domain knowledge to do so and in a regularly ch=
anging environment you need to continuously keep adjusting your metrics. In=
stana does this automatically. I think you can imagine how tricky it is to =
follow the root cause for bad response times through a network of interconn=
ected services.>
>
>
>
> Investing almost all of my free time (and a lot of my paid time) for Apac=
he, noticing a lot of the problems you have to deal with every day, I asked=
 Instana if they would be willing to provide their service to the ASF for f=
ree and they agreed and immediately setup a dedicated instance.>
>
>
>
> I wanted to try the thing out as I would prefer to grab a few beers with =
you at ApcheCon in Cevillia and not get punched in the face for recommendin=
g something bad ;-) ... so I tried this on my private Server playground. I =
unpacked and started the agent and the host appeared on the web console and=
 reported the problems it was having (ones I didn't even know about) as wel=
l as other systems it communicates with ... as soon as I added agents on th=
ese machines the analytics started doing their work across system and I bui=
lt up a map view of my services and their correlation. So it's really a sys=
tem that needs almost no configuration at all :-)>
>
>
> I uploaded the internal product presentation here: https://public.centerd=
evice.de/1a9dc4ed-515e-482e-9fd6-6d60a5562598 (please don't share this outs=
ide of the ASF)>
>
> Please use the password: 4p4cheR0cks (I'll remove that document in about =
two weeks)>
>
>
> By the way ... the screenshots in the presentation are real ... I was ama=
zed of seeing a 3D web UI in production for the first time ;-)>
>
>
>
> So if there is any interest in this offer, I would be more than happy to =
provide credentials to you and assist you in getting started, so you could =
easily try it out. The guys at Instana would also be delighted to give you =
guys an online demo and answer any questions you might be having. Feel free=
 to conatact Mirco directly for this: mirko.novakovic@codecentric.de<mailto=
:mirko.novakovic@codecentric.de>>
>
>
>
> Chris>
>
>
>
>
>
> Mirko Novakovic | Vorstand
>
> codecentric AG | Merscheider Stra=DFe 1 | 42699 Solingen | Deutschland
> tel: +49 (0) 212.23362811 | fax: +49 (0) 212.23362879 | mobil: +49 (0) 16=
3.6681500
> www.codecentric.de<http://www.codecentric.de/><http://www.codecentric.de<=
http://www.codecentric.de/>> | blog.codecentric.de<http://blog.codecentric.=
de/> | www.meettheexperts.de<http://www.meettheexperts.de/><http://www.meet=
theexperts.de<http://www.meettheexperts.de/<http://www.meettheexperts.de<ht=
tp://www.meettheexperts.de/><http://www.meettheexperts.de<http://www.meetth=
eexperts.de/>>> | www.more4fi.de<http://www.more4fi.de/><http://www.more4fi=
.de<http://www.more4fi.de/>>
>
> Sitz der Gesellschaft: Solingen | HRB 25917 | Amtsgericht Wuppertal
> Vorstand: Michael Hochg=FCrtel . Mirko Novakovic . Rainer Vehns
> Aufsichtsrat: Patric Fedlmeier (Vorsitzender) . Klaus J=E4ger . J=FCrgen =
Sch=FCtz
>
> Diese E-Mail einschlie=DFlich evtl. beigef=FCgter Dateien enth=E4lt vertr=
auliche und/oder rechtlich gesch=FCtzte Informationen. Wenn Sie nicht der r=
ichtige Adressat sind oder diese E-Mail irrt=FCmlich erhalten haben, inform=
ieren Sie bitte sofort den Absender und l=F6schen Sie diese E-Mail und evtl=
. beigef=FCgter Dateien umgehend. Das unerlaubte Kopieren, Nutzen oder =D6f=
fnen evtl. beigef=FCgter Dateien sowie die unbefugte Weitergabe dieser E-Ma=
il ist nicht gestattet.
>


--_000_HE1PR0501MB24281114C37AA4E88B2AD00BA2EF0HE1PR0501MB2428_--