Mailing-List: contact infrastructure-dev-help@apache.org; run by ezmlm
Precedence: bulk
Reply-To: infrastructure-dev@apache.org
From: Christofer Dutz <christofer.dutz@c-ware.de>
To: "infrastructure-dev@apache.org" <infrastructure-dev@apache.org>
CC: "mirko.novakovic@codecentric.de" <mirko.novakovic@codecentric.de>
Subject: AW: Tool proposal for helping run and monitor the ASF Infra Services
Thread-Topic: Tool proposal for helping run and monitor the ASF Infra Services
Thread-Index: AQHR+TCG0YI6El1LpEOMekMbD6PDxaBPu5iAgAAVsWo=
Date: Fri, 19 Aug 2016 06:20:51 +0000
Message-ID: <v18ets5rb2230ba1rueinn97.1471587650621@email.android.com>
References: <DB5PR05MB1285EBC00216DDFADB9940D5A2180@DB5PR05MB1285.eurprd05.prod.outlook.com>,<81BD2FEA-413E-489B-BCA6-6F5C79EC8178@apache.org>
In-Reply-To: <81BD2FEA-413E-489B-BCA6-6F5C79EC8178@apache.org>
Accept-Language: de-DE, en-US
Content-Language: de-DE
received-spf: None (protection.outlook.com: c-ware.de does not designate
 permitted sender hosts)
spamdiagnosticoutput: 1:99
spamdiagnosticmetadata: NSPM
Content-Type: multipart/alternative;
	boundary="_000_v18ets5rb2230ba1rueinn971471587650621emailandroidcom_"
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-originalarrivaltime: 19 Aug 2016 06:20:51.1200
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 9d387546-1437-4b89-846c-691d64a7e74d
X-MS-Exchange-Transport-CrossTenantHeadersStamped: HE1PR0501MB2428
archived-at: Fri, 19 Aug 2016 06:21:07 -0000

--_000_v18ets5rb2230ba1rueinn971471587650621emailandroidcom_
Content-Type: text/plain; charset="Windows-1252"
Content-Transfer-Encoding: quoted-printable

Hi Chris,

I knew that someone asked exactly the "how does it compare to datadog" ques=
tion somewhere. Here's the link to that mail thread https://news.ycombinato=
r.com/item?id=3D12147219

And I can confirm the shortcomings of the time series approach, cause in je=
nkins, I'd say about 70% of recent failures of flex builds were due to time=
outs when uploading Maven artifacts to nexus. The current solution doesn't =
seem to detect that. Not only that I couldn't see any hipchat notifications=
. The infra guys always had to start looking for the real reason of the tim=
eouts as nexus wasn't having any problems at all.

And I really live the feature of tracking down the response time for one se=
rvice back to other servers to find it the real reason for a system being s=
low (have a look in the presentation for this. There's a great slide on thi=
s)

Chris


Von meinem Samsung Galaxy Smartphone gesendet.


-------- Urspr=FCngliche Nachricht --------
Von: Chris Lambertus <cml@apache.org>
Datum: 19.08.16 07:03 (GMT+01:00)
An: infrastructure-dev@apache.org, Christofer Dutz <christofer.dutz@c-ware.=
de>
Cc: mirko.novakovic@codecentric.de
Betreff: Re: Tool proposal for helping run and monitor the ASF Infra Servic=
es


Hiya Chris,

Thanks for the info and the legwork on this. We currently use DataDog, whic=
h is very similar to what Instana appears to provide =97 an agent-based mon=
itoring solution that gives us that kind of look into our infra. We also ha=
ve a number of internal tools that report on various goings-on as well. You=
 might see some of this in #asfinfra on hipchat from SNMP2HipChat. DataDog =
also reports various problems there, as does our monitoring via PingMyBox.

Since you=92re not root@, you may not see some of the stuff that we see, bu=
t I think by and large, the majority of the monitors do direct to #asfinfra=
. Have you noticed gaps in the monitoring? Since we moved to DataDog, we=92=
ve been quite happy with the resolution and metrics we=92ve been able to ge=
t. It=92s been on my back burner for awhile to expose some of our DD dashbo=
ards as public, but for right now it=92s somewhat limited access. In the in=
terests of transparency (but not at the expense of security,) I=92d be happ=
y to work with you to expose more of this, and I=92m happy to address any q=
uestions or concerns about shortcomings in our monitoring.

Many thanks to Instana for offering the ASF free services! I=92d definitely=
 like to hear more about what they might be able to offer on top of what we=
 already get from DataDog. I=92ll take a look at the info you sent out. Ple=
ase feel free to follow up with me directly, either via email or hipchat.

Cheers,
-Chris


> On Aug 18, 2016, at 2:13 AM, Christofer Dutz <christofer.dutz@c-ware.de> =
wrote:
>
> Hi,
>
>
>
> I have been on the Infra Hipchat for a few weeks now while trying to migr=
ate the Flex project to Maven and back to the ASF Infra build system. Thank=
s for your support in this and even more thanks for the trust in granting m=
e access and Admin rights on the windows1 build agent.
>
>
>
> In the chat I observed the daily work of you guys, having to maintain qui=
te a zoo of all sorts of different systems on different platforms. Some pro=
blems you were having seem quite easy to track down ... if the hard disk is=
 full, you clean up. But not all problems are that easy to track down. Thin=
king of the problems with repository.apache.org ... here the cause was the =
proxy being flooded with connections (I think this was the case) ... regula=
r restarts of this helped temporarily, but I don't think that helps on the =
long term as no one had an idea why those connections were hanging there in=
 the first place.
>
>
>
> A few years ago the company I work for - codecentric - have founded a com=
pany called Instana. They are developing an agent based system for monitori=
ng IT infrastructure. In contrast to most established solutions, they use m=
achine learning strategies to analyze the root cause for problems. While yo=
u can probably achieve similar results with normal tools, the problem is th=
at you need a very detailed domain knowledge to do so and in a regularly ch=
anging environment you need to continuously keep adjusting your metrics. In=
stana does this automatically. I think you can imagine how tricky it is to =
follow the root cause for bad response times through a network of interconn=
ected services.
>
>
>
> Investing almost all of my free time (and a lot of my paid time) for Apac=
he, noticing a lot of the problems you have to deal with every day, I asked=
 Instana if they would be willing to provide their service to the ASF for f=
ree and they agreed and immediately setup a dedicated instance.
>
>
>
> I wanted to try the thing out as I would prefer to grab a few beers with =
you at ApcheCon in Cevillia and not get punched in the face for recommendin=
g something bad ;-) ... so I tried this on my private Server playground. I =
unpacked and started the agent and the host appeared on the web console and=
 reported the problems it was having (ones I didn't even know about) as wel=
l as other systems it communicates with ... as soon as I added agents on th=
ese machines the analytics started doing their work across system and I bui=
lt up a map view of my services and their correlation. So it's really a sys=
tem that needs almost no configuration at all :-)
>
>
> I uploaded the internal product presentation here: https://public.centerd=
evice.de/1a9dc4ed-515e-482e-9fd6-6d60a5562598 (please don't share this outs=
ide of the ASF)
>
> Please use the password: 4p4cheR0cks (I'll remove that document in about =
two weeks)
>
>
> By the way ... the screenshots in the presentation are real ... I was ama=
zed of seeing a 3D web UI in production for the first time ;-)
>
>
>
> So if there is any interest in this offer, I would be more than happy to =
provide credentials to you and assist you in getting started, so you could =
easily try it out. The guys at Instana would also be delighted to give you =
guys an online demo and answer any questions you might be having. Feel free=
 to conatact Mirco directly for this: mirko.novakovic@codecentric.de
>
>
>
> Chris


--_000_v18ets5rb2230ba1rueinn971471587650621emailandroidcom_--