Mailing-List: contact dev-help@cloudstack.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@cloudstack.apache.org
From: Simon Weller <sweller@ena.com>
To: "dev@cloudstack.apache.org" <dev@cloudstack.apache.org>
Subject: Re: [discuss] CloudStack logging
Thread-Topic: [discuss] CloudStack logging
Thread-Index: AdGWS/pfeT/B+GNsRrqz8RH+SRpeSwACo1oAAAASENAAAF+Fmg==
Date: Thu, 14 Apr 2016 14:18:44 +0000
Message-ID: 
 <BY2PR02MB2007CE3D052036C2AA30EEACA9970@BY2PR02MB2007.namprd02.prod.outlook.com>
References: 
 <HE1PR07MB140174A2FBD9D93BFF08679E8C970@HE1PR07MB1401.eurprd07.prod.outlook.com>
 <156561992.3091.160e133c-ee3f-4c6e-9b8b-e4d468651d78.open-xchange@ox.pcextreme.nl>,<HE1PR07MB1401BBD515C231D1BBEC63AB8C970@HE1PR07MB1401.eurprd07.prod.outlook.com>
In-Reply-To: 
 <HE1PR07MB1401BBD515C231D1BBEC63AB8C970@HE1PR07MB1401.eurprd07.prod.outlook.com>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-messagesentrepresentingtype: 1
x-originating-ip: [132.245.54.101]
x-ms-office365-filtering-correlation-id: 7fc6d1d6-20a0-4100-d32e-08d3646fafbb
x-microsoft-exchange-diagnostics: 
 1;BY2PR02MB2008;5:9WVhbOxHF7dEbtU27WahFeIucay8SBe3Z++RK/es1C7y18svs0lTOVH5xUUPOMk8Uh+kssRrlm9v0Z80F0LUYNdXhq9VgtNnbr6Aq1NOT3UiNV5gk5KSui3Hq80mhJzjuMyt69Xs8FJsCvOcjXbkoQ==;24:bz1gzyhV1FIsuGisZA4C00lHaZdh00pkbhC35TlvAMzQga2pEn+1i7Km0+cvvQUcna/BAhaWeCF4s0RoFZ+ynjws+oD4Ym9y7SGiVodJodI=
x-microsoft-antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:BY2PR02MB2008;
x-microsoft-antispam-prvs: 
 <BY2PR02MB2008F029E68099E6260335FCA9970@BY2PR02MB2008.namprd02.prod.outlook.com>
x-exchange-antispam-report-test: UriScan:;
x-exchange-antispam-report-cfa-test: 
 BCL:0;PCL:0;RULEID:(601004)(2401047)(5005006)(8121501046)(10201501046)(3002001);SRVR:BY2PR02MB2008;BCL:0;PCL:0;RULEID:;SRVR:BY2PR02MB2008;
x-forefront-prvs: 0912297777
x-forefront-antispam-report: 
 SFV:NSPM;SFS:(10009020)(6009001)(13464003)(71364002)(53754006)(377454003)(5004730100002)(19580395003)(19580405001)(107886002)(110136002)(189998001)(74316001)(92566002)(54356999)(87936001)(50986999)(76176999)(10400500002)(5002640100001)(15974865002)(2950100001)(86362001)(586003)(122556002)(2900100001)(15975445007)(1096002)(3900700001)(77096005)(1220700001)(66066001)(81166005)(3660700001)(2906002)(5003600100002)(2501003)(99286002)(9686002)(3280700002)(6116002)(33656002)(102836003)(3846002)(2351001)(5008740100001)(76576001);DIR:OUT;SFP:1101;SCL:1;SRVR:BY2PR02MB2008;H:BY2PR02MB2007.namprd02.prod.outlook.com;FPR:;SPF:None;MLV:sfv;LANG:en;
spamdiagnosticoutput: 1:23
spamdiagnosticmetadata: NSPM
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginatorOrg: ena.com
X-MS-Exchange-CrossTenant-originalarrivaltime: 14 Apr 2016 14:18:44.0591
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 6dc38cd4-4d4f-4826-9649-17854289d170
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY2PR02MB2008
X-ENA-MailScanner-Information: Please contact support@ena.com for more
 information
X-ENA-MailScanner-ID: 699FD1480A28.ABEE2
X-ENA-MailScanner: No viruses found
X-ENA-MailScanner-SpamCheck: not spam, SpamAssassin (not cached, score=-2.7,
	required 4, BAYES_00 -2.20, DKIM_SIGNED 0.10, DKIM_VALID -0.10,
	DKIM_VALID_AU -0.20, OS_UNKNOWN -0.10, SPF_HELO_PASS -0.20)
X-ENA-MailScanner-From: sweller@ena.com
X-ENA-MailScanner-Watermark: 1461248326.76105@KbgNywvSMZok51vs9fyLJA

I'm all for this as well.

Training our systems folks on how to work through the logs is a pain. Focus=
ing in on the most important items is more useful and also makes it easier =
to automate log fetching and categorization.=20

- Si

________________________________________
From: Paul Angus <paul.angus@shapeblue.com>
Sent: Thursday, April 14, 2016 9:06 AM
To: Wido den Hollander; dev@cloudstack.apache.org
Subject: RE: [discuss] CloudStack logging

Grabbing a couple of examples - these should be debug - I can't 'do' anythi=
ng with this information.

INFO  [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-4119d5=
bc) Begin cleanup expired async-jobs
INFO  [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-4119d5=
bc) End cleanup expired async-jobs
INFO  [c.c.h.v.r.VmwareResource] (DirectAgentCronJob-2:ctx-3209f65c) Scan h=
ung worker VM to recycle
INFO  [c.c.h.v.r.VmwareResource] (DirectAgentCronJob-113:ctx-56d9f9f2) Scan=
 hung worker VM to recycle

Whereas this should be WARN. I wouldn't want to lose this by switching off =
DEBUG

DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-91226be7) Detecte=
d management node left, id:2, nodeIP:10.2.0.6
DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-0bccd078) Detecte=
d management node left, id:2, nodeIP:10.2.0.6


Kind regards,

Paul Angus

Regards,

Paul Angus

paul.angus@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue

-----Original Message-----
From: Wido den Hollander [mailto:wido@widodh.nl]
Sent: 14 April 2016 15:04
To: Paul Angus <paul.angus@shapeblue.com>; dev@cloudstack.apache.org
Subject: Re: [discuss] CloudStack logging


> Op 14 april 2016 om 14:49 schreef Paul Angus <paul.angus@shapeblue.com>:
>
>
> Hi All
>
> I think that we can all agree that CloudStack logs are very difficult
> to read, especially for operational staff.
>
> I believe that the primary reason for this is that a large amount of
> what should be INFO level events are categorised as DEBUG.  The
> logging therefore must be set at DEBUG for the logs to capture these
> events.  By defaulting the logging to DEBUG we include a lot of detail
> which is counter-productive when using the logs to diagnose operational i=
ssues.
>

Yes!

> I think the situation can be greatly improved by reviewing the
> categorisation of logged events and where necessary re-categorise then
> such that INFO, WARN and ERROR logging would be sufficient for an 'operat=
ional' admin.
>
> I think a phased approach where the default logging level remains at
> DEBUG while re-categorisation occurs would mean that nothing
> materially changes until we are happy with the categorisations. Then
> we can default the logging level to INFO.
>

I created a issue for this a while ago:
https://issues.apache.org/jira/browse/CLOUDSTACK-8645

Short: Yes. Just change this where needed.

Wido

>
>
> Thoughts everyone?
>
>
>
>
> Kind regards,
>
> Paul Angus
>
>
> Regards,
>
> Paul Angus
>
> paul.angus@shapeblue.com
> www.shapeblue.com
> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK @shapeblue