Return-Path: X-Original-To: apmail-stratos-dev-archive@minotaur.apache.org Delivered-To: apmail-stratos-dev-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 7E9F5112FD for ; Wed, 2 Jul 2014 19:01:04 +0000 (UTC) Received: (qmail 46382 invoked by uid 500); 2 Jul 2014 19:01:04 -0000 Delivered-To: apmail-stratos-dev-archive@stratos.apache.org Received: (qmail 46330 invoked by uid 500); 2 Jul 2014 19:01:04 -0000 Mailing-List: contact dev-help@stratos.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@stratos.apache.org Delivered-To: mailing list dev@stratos.apache.org Received: (qmail 46314 invoked by uid 99); 2 Jul 2014 19:01:03 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jul 2014 19:01:03 +0000 X-ASF-Spam-Status: No, hits=-1.2 required=5.0 tests=HK_RANDOM_ENVFROM,HK_RANDOM_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_HI,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of jeffrngu@cisco.com designates 173.37.142.89 as permitted sender) Received: from [173.37.142.89] (HELO alln-iport-2.cisco.com) (173.37.142.89) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Jul 2014 19:00:58 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=cisco.com; i=@cisco.com; l=7781; q=dns/txt; s=iport; t=1404327658; x=1405537258; h=from:to:subject:date:message-id:references:in-reply-to: mime-version; bh=lz9KSldICmt75VZn48MrjIJg4KAo5KBlcSsW2yG9zzM=; b=LuQPOV2infuKX0d1szUNfQpSJEqeMR1eIvXDGG9LLOnNWjCZUUY3Ed5y 38hjCRFap0KyWmqRuqNgNvn/9Xy6Z/FSh3xiPfM+PfoPQcSvg4BXPsyx5 rWccQ6nIqMu97+Ldl3AX2KuBQV7J9wugeskA6A6XO4LKXE7pypHqtGHXt A=; X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AlsJAJRVtFOtJA2H/2dsb2JhbABagkZHgSyrOgIBAQEBAQEFAW4BmWUBgQcWdYQDAQEBBIEJAgEIEQMBAigHMhQJCAIEE4hCx1kXhXCJIRiEQwWabZQKg0OCMA X-IronPort-AV: E=Sophos;i="5.01,590,1400025600"; d="scan'208,217";a="57830213" Received: from alln-core-2.cisco.com ([173.36.13.135]) by alln-iport-2.cisco.com with ESMTP; 02 Jul 2014 19:00:37 +0000 Received: from xhc-rcd-x06.cisco.com (xhc-rcd-x06.cisco.com [173.37.183.80]) by alln-core-2.cisco.com (8.14.5/8.14.5) with ESMTP id s62J0bqV014184 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=FAIL) for ; Wed, 2 Jul 2014 19:00:37 GMT Received: from xmb-aln-x10.cisco.com ([169.254.5.33]) by xhc-rcd-x06.cisco.com ([173.37.183.80]) with mapi id 14.03.0123.003; Wed, 2 Jul 2014 14:00:36 -0500 From: "Jeffrey Nguyen (jeffrngu)" To: "dev@stratos.apache.org" Subject: Re: Carbon log flooded causing disk full and reboot Thread-Topic: Carbon log flooded causing disk full and reboot Thread-Index: AQHPkmLXFaMNK1VgCUS2kvFUZNzjZZuI5T0AgAQkQ4A= Date: Wed, 2 Jul 2014 19:00:36 +0000 Message-ID: References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: user-agent: Microsoft-MacOutlook/14.3.9.131030 x-originating-ip: [10.21.148.213] Content-Type: multipart/alternative; boundary="_000_CFD9A08F29266jeffrnguciscocom_" MIME-Version: 1.0 X-Virus-Checked: Checked by ClamAV on apache.org --_000_CFD9A08F29266jeffrnguciscocom_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Thanks Imesh, Isuru, Amila and Martin. As Martin mentioned, we had log enabled at DEBUG level. This change will a= dditional level of protection besides having good logging instrumentation i= n the code. With the DailyRollingFileAppender, we can only hope that log = file is rotated every 24 hr before the disk is filled up. I've committed this change to both master and 4.0.0. Please feel free to r= evert or make adjustment to the size/number of files if you think necessary= . Regards, -Jeffrey From: Imesh Gunaratne > Reply-To: "dev@stratos.apache.org" > Date: Sunday, June 29, 2014 1:45 PM To: dev > Subject: Re: Carbon log flooded causing disk full and reboot Hi Jeffrey, A great finding! Did this occur while running Stratos in single JVM mode wi= thout having any DEBUG level logs? Please send these changes to both master and 4.0.0 branches. Thanks On Fri, Jun 27, 2014 at 7:52 PM, Jeffrey Nguyen (jeffrngu) > wrote: Hi, In our Stratos deployment, we experienced issue where wso2carbon.log file g= rew as much as 13 GB causing disk full and instance reboot and Stratos inac= cessible. We're deploying Stratos using default profile. To fix this is= sue, we modified log4j.properties with the changes listed below. Please let me know if it's okay to push this changes upstream. If so, whi= ch branch(es) should the changes be checked in? Thanks, -Jeffrey diff --git a/products/stratos/modules/distribution/src/main/conf/log4j.prop= erties b/products/stratos/modules/distribution/src index a85d346..bb37679 100644 --- a/products/stratos/modules/distribution/src/main/conf/log4j.properties +++ b/products/stratos/modules/distribution/src/main/conf/log4j.properties @@ -113,7 +113,9 @@ log4j.appender.CARBON_MEMORY.threshold=3DDEBUG # CARBON_LOGFILE is set to be a DailyRollingFileAppender using a PatternLa= yout. -log4j.appender.CARBON_LOGFILE=3Dorg.apache.log4j.DailyRollingFileAppender +log4j.appender.CARBON_LOGFILE=3Dorg.apache.log4j.RollingFileAppender +log4j.appender.CARBON_LOGFILE.MaxFileSize=3D10000KB +log4j.appender.CARBON_LOGFILE.MaxBackupIndex=3D10 # Log file will be overridden by the configuration setting in the DB # This path should be relative to WSO2 Carbon Home -- Imesh Gunaratne Technical Lead, WSO2 Committer & PPMC Member, Apache Stratos --_000_CFD9A08F29266jeffrnguciscocom_ Content-Type: text/html; charset="us-ascii" Content-ID: Content-Transfer-Encoding: quoted-printable

Thanks Imesh, Isuru, Amila and Martin.

As Martin mentioned, we had log enabled at DEBUG level.  This cha= nge will additional level of protection besides having good logging instrum= entation in the code.   With the DailyRollingFileAppender, we can= only hope that log file is rotated every 24 hr before the disk is filled up.

I've committed this change to both master and 4.0.0.  Please feel= free to revert or make adjustment to the size/number of files if you think= necessary.

Regards,
-Jeffrey

From: Imesh Gunaratne <imesh@apache.org>
Reply-To: "dev@stratos.apache.org" <dev@stratos.apache.org>
Date: Sunday, June 29, 2014 1:45 PM=
To: dev <dev@stratos.apache.org>
Subject: Re: Carbon log flooded cau= sing disk full and reboot

Hi Jeffrey,

A great finding! Did this occur while running Stratos in single JVM mo= de without having any DEBUG level logs?
Please send these changes to both master and 4.0.0 branches. 

Thanks


On Fri, Jun 27, 2014 at 7:52 PM, Jeffrey Nguyen = (jeffrngu) <jeffrngu@cisco.com> wrote:
Hi,

In our Stratos deployment, we experienced issue where wso2carbon.log f= ile grew as much as 13 GB causing disk full and instance reboot and Stratos= inaccessible.   We're deploying Stratos using default profile.  = To fix this issue, we modified log4j.properties with the changes listed below.

Please let me know if it's okay to push this changes upstream.   = If so, which branch(es) should the changes be checked in?

Thanks,
-Jeffrey


diff --git a/products/stratos/modules/distribution/src/main/conf/log4j= .properties b/products/stratos/modules/distribution/src
index a85d346..bb37679 100644
--- a/products/stratos/modules/distribution/src/main/conf/log4j.proper= ties
+++ b/products/stratos/modules/distribution/src/main/conf/= log4j.properties
@@ -113,7 +113,9 @@ log4j.appender.CARBON_MEMORY.threshold=3DDEBUG=
 
 
 # CARBON_LOGFILE is set to be a DailyRollingFileAppender using a= PatternLayout.
-log4j.appender.CARBON_LOGFILE=3Dorg.apache.log4j.DailyRollingFileAppe= nder
+log4j.appender.CARBON_LOGFILE=3Dorg.apache.log4j.RollingFileAppen= der
+log4j.appender.CARBON_LOGFILE.MaxFileSize=3D10000KB
+log4j.appender.CARBON_LOGFILE.MaxBackupIndex=3D10
 # Log file will be overridden by the configuration setting in th= e DB
 # This path should be relative to WSO2 Carbon Home



--
Imesh Gunaratne

Technical Lead, WSO2
Committer & PPMC Member, Apache Stratos
--_000_CFD9A08F29266jeffrnguciscocom_--