Return-Path: X-Original-To: apmail-stratos-dev-archive@minotaur.apache.org Delivered-To: apmail-stratos-dev-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id AA3EC17613 for ; Sun, 26 Oct 2014 02:59:57 +0000 (UTC) Received: (qmail 46478 invoked by uid 500); 26 Oct 2014 02:59:56 -0000 Delivered-To: apmail-stratos-dev-archive@stratos.apache.org Received: (qmail 46424 invoked by uid 500); 26 Oct 2014 02:59:56 -0000 Mailing-List: contact dev-help@stratos.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@stratos.apache.org Delivered-To: mailing list dev@stratos.apache.org Received: (qmail 45935 invoked by uid 99); 26 Oct 2014 02:59:55 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 26 Oct 2014 02:59:55 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of rajkumarr@wso2.com designates 209.85.214.177 as permitted sender) Received: from [209.85.214.177] (HELO mail-ob0-f177.google.com) (209.85.214.177) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 26 Oct 2014 02:59:28 +0000 Received: by mail-ob0-f177.google.com with SMTP id m8so1124114obr.36 for ; Sat, 25 Oct 2014 19:59:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=wso2.com; s=google; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=PiAPnrQUG8RFa4m1Zdf+RoiXCabFYr6F+UrJ4YqXC1c=; b=XRosHFVXseM7+CG2tXYR2eQZQBPzeTERZJ7wt8ZdMfO0+7+M6lT4UTbWuI9ybghOjJ kpBU6mRJ1Hm1jbpsvs5qbZnthvei17w3eHZfE/6MeCXi8CFIm4uksgsdBTPKo3VL4z+q qKXLqxu86Csb2Aejp7JIqi6+MwCQCPyzc1Zi8= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:content-type; bh=PiAPnrQUG8RFa4m1Zdf+RoiXCabFYr6F+UrJ4YqXC1c=; b=I//nHsXBHDcfmLTmYDEgsFUANHU8y1EiNQhrpCLF8Sxer3pGojwzW95HNx26bfjhvF waJattNXpFLkcKwgY1vR/Bjx8tKP41ZjGSUgb1drchlN93VArAMcS0ItLQvL0C1BSojO nfHQ6jAYMMYHeVIOauhNeArY7M3Az3YdBY9bENpECOdeqxLi8cKpk5zNEfRASThWPwNz rf8fYYrxsUBHK/CzS0vh0ZtuROA2TgOcoD1LN3WmrOTs3ZXGO1qdeqKLmA9sh4JY8i6e ttIwBVbWbulElUkl9rL/CCTQt9wsTOv5P8PO1DXGWScSQfI5QkxEfiMiMykXh3ntvL13 xTRQ== X-Gm-Message-State: ALoCoQmFN3ZnucDTZlD8BncDGUuqWZc3AZBWkVJj8Q/qa2Owd0Fh/svZHXfUY/T9gEkIHt7PMn7D MIME-Version: 1.0 X-Received: by 10.60.47.100 with SMTP id c4mr12642492oen.9.1414292366381; Sat, 25 Oct 2014 19:59:26 -0700 (PDT) Received: by 10.182.188.103 with HTTP; Sat, 25 Oct 2014 19:59:26 -0700 (PDT) In-Reply-To: References: Date: Sun, 26 Oct 2014 08:29:26 +0530 Message-ID: Subject: Re: CEP sends very large values for gradient and second derivative of load average From: Rajkumar Rajaratnam To: "dev@stratos.apache.org" Content-Type: multipart/alternative; boundary=001a11c20b2c1b70fd05064a9a77 X-Virus-Checked: Checked by ClamAV on apache.org --001a11c20b2c1b70fd05064a9a77 Content-Type: text/plain; charset=UTF-8 Hi Chamila, I couldn't find any large values sent by agent. There are traces for CEP sending large values. Find the cep trace log here [1] 1. https://drive.google.com/file/d/0B1haIleqJMHIS3FFVFpfa0JlbVE/view?usp=sharing Thanks. On Sun, Oct 26, 2014 at 8:23 AM, Manula Chathurika Thantriwatte < manulac@wso2.com> wrote: > Hi, > > Shall we enable the cartridge agent debug logs and see what are the load > average values sent from the agent. If those values are normal then we can > narrow down to the CEP. > > Thanks ! > > On Sat, Oct 25, 2014 at 10:43 PM, Chamila De Alwis > wrote: > >> Hi Raj, >> >> Can you inspect the CEP trace logs for the same time period? It will have >> the events published from the agent. >> >> >> Regards, >> Chamila de Alwis >> Software Engineer | WSO2 | +94772207163 >> Blog: code.chamiladealwis.com >> >> >> >> On Sat, Oct 25, 2014 at 10:39 PM, Rajkumar Rajaratnam > > wrote: >> >>> Hi, >>> >>> I am doing a test round with M3 with python agent. And faced this very >>> strange scenario, $Subject. >>> >>> >>> TID: [0] [STRATOS] [2014-10-25 22:10:38,693] DEBUG >>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average >>> stats are reset, ready to do scale check [kub cluster] KubGrp1 >>> TID: [0] [STRATOS] [2014-10-25 22:10:38,693] DEBUG >>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of >>> load avg event: [cluster] php.php.domain [value] -2000.0 >>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG >>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average >>> stats are reset, ready to do scale check [kub cluster] KubGrp1 >>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG >>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of >>> load avg event: [cluster] php.php.domain [value] 0.0 >>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG >>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average >>> stats are reset, ready to do scale check [kub cluster] KubGrp1 >>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG >>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg >>> load avg event: [cluster] php.php.domain [value] 1.9090909 >>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG >>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average >>> stats are reset, ready to do scale check [kub cluster] KubGrp1 >>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG >>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg >>> Memory Consumption event: [cluster] php.php.domain [value] 28.718182 >>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG >>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second >>> Derivation of Memory Consumption event: [cluster] php.php.domain [value] >>> 200000.0 >>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG >>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of >>> load avg event: [cluster] php.php.domain [value] 0.0 >>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG >>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average >>> stats are reset, ready to do scale check [kub cluster] KubGrp1 >>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG >>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second >>> Derivation of load avg event: [cluster] php.php.domain [value] 0.0 >>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG >>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average >>> stats are reset, ready to do scale check [kub cluster] KubGrp1 >>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG >>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second >>> Derivation of load avg event: [cluster] php.php.domain [value] 0.0 >>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG >>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average >>> stats are reset, ready to do scale check [kub cluster] KubGrp1 >>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG >>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg >>> load avg event: [cluster] php.php.domain [value] 3.3636363 >>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG >>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average >>> stats are reset, ready to do scale check [kub cluster] KubGrp1 >>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG >>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of >>> Memory Consumption event: [cluster] php.php.domain [value] 0.11670045 >>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG >>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory >>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1 >>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG >>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg >>> Memory Consumption event: [cluster] php.php.domain [value] 28.7 >>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG >>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory >>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1 >>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG >>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second >>> Derivation of Memory Consumption event: [cluster] php.php.domain [value] 0.0 >>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG >>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory >>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1 >>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG >>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg >>> Memory Consumption event: [cluster] php.php.domain [value] 28.7 >>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG >>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory >>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1 >>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG >>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg >>> load avg event: [cluster] php.php.domain [value] 5.0 >>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG >>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average >>> stats are reset, ready to do scale check [kub cluster] KubGrp1 >>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG >>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of >>> load avg event: [cluster] php.php.domain [value] 2000.0 >>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG >>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average >>> stats are reset, ready to do scale check [kub cluster] KubGrp1 >>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG >>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg >>> Memory Consumption event: [cluster] php.php.domain [value] 28.7 >>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG >>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory >>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1 >>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG >>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of >>> Memory Consumption event: [cluster] php.php.domain [value] 0.0 >>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG >>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory >>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1 >>> TID: [0] [STRATOS] [2014-10-25 22:10:38,699] DEBUG >>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second >>> Derivation of Memory Consumption event: [cluster] php.php.domain [value] 0.0 >>> TID: [0] [STRATOS] [2014-10-25 22:10:38,699] DEBUG >>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory >>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1 >>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG >>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second >>> Derivation of load avg event: [cluster] php.php.domain [value] 2000000.0 >>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG >>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average >>> stats are reset, ready to do scale check [kub cluster] KubGrp1 >>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG >>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of >>> load avg event: [cluster] php.php.domain [value] 13500.0 >>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG >>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average >>> stats are reset, ready to do scale check [kub cluster] KubGrp1 >>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG >>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of >>> Memory Consumption event: [cluster] php.php.domain [value] 50.0 >>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG >>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory >>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1 >>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG >>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg >>> Memory Consumption event: [cluster] php.php.domain [value] 28.738462 >>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG >>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory >>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1 >>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG >>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg >>> load avg event: [cluster] php.php.domain [value] 2.3333333 >>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG >>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average >>> stats are reset, ready to do scale check [kub cluster] KubGrp1 >>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG >>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of >>> load avg event: [cluster] php.php.domain [value] 0.0 >>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG >>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average >>> stats are reset, ready to do scale check [kub cluster] KubGrp1 >>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG >>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of >>> Memory Consumption event: [cluster] php.php.domain [value] 0.0 >>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG >>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory >>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1 >>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG >>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg >>> load avg event: [cluster] php.php.domain [value] 31.0 >>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG >>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average >>> stats are reset, ready to do scale check [kub cluster] KubGrp1 >>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG >>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of >>> Memory Consumption event: [cluster] php.php.domain [value] 0.0 >>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG >>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory >>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1 >>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG >>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg >>> load avg event: [cluster] php.php.domain [value] 39.0 >>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG >>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average >>> stats are reset, ready to do scale check [kub cluster] KubGrp1 >>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG >>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg >>> load avg event: [cluster] php.php.domain [value] 39.0 >>> TID: [0] [STRATOS] [2014-10-25 22:10:38,705] DEBUG >>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average >>> stats are reset, ready to do scale check [kub cluster] KubGrp1 >>> TID: [0] [STRATOS] [2014-10-25 22:10:38,706] DEBUG >>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of >>> load avg event: [cluster] php.php.domain [value] -11000.0 >>> TID: [0] [STRATOS] [2014-10-25 22:10:38,706] DEBUG >>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average >>> stats are reset, ready to do scale check [kub cluster] KubGrp1 >>> TID: [0] [STRATOS] [2014-10-25 22:10:38,708] DEBUG >>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of >>> Memory Consumption event: [cluster] php.php.domain [value] 0.0 >>> TID: [0] [STRATOS] [2014-10-25 22:10:38,708] DEBUG >>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory >>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1 >>> TID: [0] [STRATOS] [2014-10-25 22:10:38,709] DEBUG >>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg >>> Memory Consumption event: [cluster] php.php.domain [value] 28.8 >>> TID: [0] [STRATOS] [2014-10-25 22:10:38,709] DEBUG >>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory >>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1 >>> TID: [0] [STRATOS] [2014-10-25 22:10:38,811] DEBUG >>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} - >>> PodActivationWatcher running : Running >>> TID: [0] [STRATOS] [2014-10-25 22:10:38,942] DEBUG >>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} - >>> PodActivationWatcher running : Running >>> TID: [0] [STRATOS] [2014-10-25 22:10:38,945] DEBUG >>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} - >>> PodActivationWatcher running : Running >>> TID: [0] [STRATOS] [2014-10-25 22:10:43,712] DEBUG >>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} - >>> KubernetesServiceClusterMonitor is running.. >>> KubernetesServiceClusterMonitor [ kubernetesHostClusterId=KubGrp1, >>> clusterId=php.php.domain, serviceId=php] >>> TID: [0] [STRATOS] [2014-10-25 22:10:43,713] DEBUG >>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} - >>> Running min check for [kub-cluster] : KubGrp1 [cluster] : php.php.domain >>> TID: [0] [STRATOS] [2014-10-25 22:10:43,713] INFO >>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running minimum rule: >>> [kub-cluster] KubGrp1 [cluster] php.php.domain >>> TID: [0] [STRATOS] [2014-10-25 22:10:43,714] INFO >>> {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] [cluster] : >>> php.php.domain [Replicas] nonTerminated : 3 >>> TID: [0] [STRATOS] [2014-10-25 22:10:43,714] INFO >>> {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] [cluster] : >>> php.php.domain [Replicas] minReplicas : 3 >>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] INFO >>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running obsolete containers >>> rule [kub-cluster] : KubGrp1 [cluster] : php.php.domain >>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] INFO >>> {org.apache.stratos.autoscaler.rule.RuleLog} - [obsolete-check] [cluster] >>> : php.php.domain [Replicas] obsoleteReplicas : 0 >>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG >>> {org.apache.stratos.autoscaler.rule.AutoscalerRuleEvaluator} - Minimum >>> check executed for : >>> org.apache.stratos.autoscaler.KubernetesClusterContext@21d906d0 >>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG >>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} - >>> flag of rifReset : true flag of memoryConsumptionReset : true flag of >>> loadAverageReset : true >>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG >>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} - >>> Running scale check for [kub-cluster] : KubGrp1 [cluster] : php.php.domain >>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] INFO >>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running scaling rule >>> [kub-cluster] : KubGrp1 [cluster] : php.php.domain >>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] DEBUG >>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the >>> value, [average]: 0.0 , [gradient]: 0.0 , [second derivative]: 0.0 , [time >>> intervals]: 1 >>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] DEBUG >>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the >>> value, [average]: 28.8 , [gradient]: 0.0 , [second derivative]: 0.0 , [time >>> intervals]: 1 >>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] DEBUG >>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the >>> value, [average]: 39.0 , [gradient]: -11000.0 , [second derivative]: >>> 2000000.0 , [time intervals]: 1 >>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] INFO >>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : >>> php.php.domain [Replicas] minReplicas : 3 >>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] INFO >>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : >>> php.php.domain [Replicas] maxReplicas : 10 >>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO >>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : >>> php.php.domain [Replicas] nonTerminated : 3 >>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO >>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : >>> php.php.domain [Replicas] activeReplicas : 3 >>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO >>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : >>> php.php.domain [RequestInFlight] predicted value : 0.0 >>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO >>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : >>> php.php.domain [RequestInFlight] upper limit : 80.0 >>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO >>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : >>> php.php.domain [RequestInFlight] lower limit : 5.0 >>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO >>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : >>> php.php.domain [MemoryConsumption] predicted value : 28.799999237060547 >>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO >>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : >>> php.php.domain [MemoryConsumption] upper limit : 80.0 >>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO >>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : >>> php.php.domain [MemoryConsumption] lower limit : 15.0 >>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO >>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : >>> php.php.domain [LoadAverage] predicted value : 989039.0 >>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO >>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : >>> php.php.domain [LoadAverage] upper limit : 180.0 >>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO >>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : >>> php.php.domain [LoadAverage] lower limit : 20.0 >>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO >>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : >>> php.php.domain scale-up action : true >>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO >>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : >>> php.php.domain scale-down action : false >>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO >>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : >>> php.php.domain [LoadAverage] predicted replicas : 16484 >>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO >>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : >>> php.php.domain predicted replicas > max replicas : >>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO >>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] Decided to >>> scale-up : [cluster] : php.php.domain >>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO >>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling-up] [cluster] : >>> php.php.domain valid number of replicas to expand : 10 >>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO >>> {org.apache.stratos.autoscaler.client.cloud.controller.CloudControllerClient} >>> - Updating kubernetes replication controller via cloud controller: >>> [cluster] php.php.domain [replicas] 10 >>> TID: [0] [STRATOS] [2014-10-25 22:10:43,734] DEBUG >>> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} - >>> CloudControllerServiceImpl:updateContainers for cluster : php.php.domain >>> TID: [0] [STRATOS] [2014-10-25 22:10:43,758] DEBUG >>> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} - >>> Cloud Controller is delegating request to update a replication controller >>> php.php.domain to Kubernetes layer. >>> >>> >>> I couldn't investigate the agent side, since containers are destroyed. >>> >>> Any idea? >>> >>> Thanks. >>> >>> -- >>> Rajkumar Rajaratnam >>> Software Engineer | WSO2, Inc. >>> Mobile +94777568639 | +94783498120 >>> >> >> > > > -- > Regards, > Manula Chathurika Thantriwatte > Software Engineer > WSO2 Inc. : http://wso2.com > lean . enterprise . middleware > > email : manulac@wso2.com / manula@apache.org > phone : +94 772492511 > blog : http://manulachathurika.blogspot.com/ > > > > -- Rajkumar Rajaratnam Software Engineer | WSO2, Inc. Mobile +94777568639 | +94783498120 --001a11c20b2c1b70fd05064a9a77 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hi Chamila,

I couldn't find any lar= ge values sent by agent. There are traces for CEP sending large values. Fin= d the cep trace log here [1]


Thanks.

On Sun, Oct 26, 2014 at 8:23 AM, = Manula Chathurika Thantriwatte <manulac@wso2.com> wrote:
<= blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px= #ccc solid;padding-left:1ex">
Hi,

Shall= we enable the cartridge agent debug logs and see what are the load average= values sent from the agent. If those values are normal then we can narrow = down to the CEP.

Thanks !

On Sat,= Oct 25, 2014 at 10:43 PM, Chamila De Alwis <chamilad@wso2.com> wrote:
Hi Raj= ,

Can you inspect the CEP trace logs for the same time period= ? It will have the events published from the agent.

Regards,
Chamila de Alwis
Software Engineer | WSO2 | +94772207163
Blog: code.chamiladealwis.com



On Sat, Oct 25, 2014 at 10:39 PM, Rajk= umar Rajaratnam <rajkumarr@wso2.com> wrote:
=
Hi,

I am doing a test round with M3 with python agent. And faced this very str= ange scenario, $Subject.


TID: = [0] [STRATOS] [2014-10-25 22:10:38,693] DEBUG {org.apache.stratos.autoscale= r.KubernetesClusterContext} - =C2=A0Load average stats are reset, ready to = do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,693] DEBUG {org= .apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - =C2=A0Grad o= f load avg event: [cluster] php.php.domain [value] -2000.0
TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG {org.apache.stratos.aut= oscaler.KubernetesClusterContext} - =C2=A0Load average stats are reset, rea= dy to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [20= 14-10-25 22:10:38,694] DEBUG {org.apache.stratos.autoscaler.monitor.Kuberne= tesClusterMonitor} - =C2=A0Grad of load avg event: [cluster] php.php.domain= [value] 0.0
TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG {= org.apache.stratos.autoscaler.KubernetesClusterContext} - =C2=A0Load averag= e stats are reset, ready to do scale check [kub cluster] KubGrp1
= TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG {org.apache.stratos.auto= scaler.monitor.KubernetesClusterMonitor} - =C2=A0Avg load avg event: [clust= er] php.php.domain [value] 1.9090909
TID: [0] [STRATOS] [2014-10-= 25 22:10:38,694] DEBUG {org.apache.stratos.autoscaler.KubernetesClusterCont= ext} - =C2=A0Load average stats are reset, ready to do scale check [kub clu= ster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG = {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - =C2=A0Av= g Memory Consumption event: [cluster] php.php.domain [value] 28.718182
TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG {org.apache.strato= s.autoscaler.monitor.KubernetesClusterMonitor} - =C2=A0Second Derivation of= Memory Consumption event: [cluster] php.php.domain [value] 200000.0
<= div>TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG {org.apache.stratos.= autoscaler.monitor.KubernetesClusterMonitor} - =C2=A0Grad of load avg event= : [cluster] php.php.domain [value] 0.0
TID: [0] [STRATOS] [2014-1= 0-25 22:10:38,695] DEBUG {org.apache.stratos.autoscaler.KubernetesClusterCo= ntext} - =C2=A0Load average stats are reset, ready to do scale check [kub c= luster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBU= G {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - =C2=A0= Second Derivation of load avg event: [cluster] php.php.domain [value] 0.0
TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG {org.apache.str= atos.autoscaler.KubernetesClusterContext} - =C2=A0Load average stats are re= set, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRA= TOS] [2014-10-25 22:10:38,696] DEBUG {org.apache.stratos.autoscaler.monitor= .KubernetesClusterMonitor} - =C2=A0Second Derivation of load avg event: [cl= uster] php.php.domain [value] 0.0
TID: [0] [STRATOS] [2014-10-25 = 22:10:38,696] DEBUG {org.apache.stratos.autoscaler.KubernetesClusterContext= } - =C2=A0Load average stats are reset, ready to do scale check [kub cluste= r] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG {or= g.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - =C2=A0Avg l= oad avg event: [cluster] php.php.domain [value] 3.3636363
TID: [0= ] [STRATOS] [2014-10-25 22:10:38,696] DEBUG {org.apache.stratos.autoscaler.= KubernetesClusterContext} - =C2=A0Load average stats are reset, ready to do= scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25= 22:10:38,696] DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClust= erMonitor} - =C2=A0Grad of Memory Consumption event: [cluster] php.php.doma= in [value] 0.11670045
TID: [0] [STRATOS] [2014-10-25 22:10:38,696= ] DEBUG {org.apache.stratos.autoscaler.KubernetesClusterContext} - =C2=A0Me= mory consumption stats are reset, ready to do scale check [kub cluster] Kub= Grp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG {org.apac= he.stratos.autoscaler.monitor.KubernetesClusterMonitor} - =C2=A0Avg Memory = Consumption event: [cluster] php.php.domain [value] 28.7
TID: [0]= [STRATOS] [2014-10-25 22:10:38,697] DEBUG {org.apache.stratos.autoscaler.K= ubernetesClusterContext} - =C2=A0Memory consumption stats are reset, ready = to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-= 10-25 22:10:38,697] DEBUG {org.apache.stratos.autoscaler.monitor.Kubernetes= ClusterMonitor} - =C2=A0Second Derivation of Memory Consumption event: [clu= ster] php.php.domain [value] 0.0
TID: [0] [STRATOS] [2014-10-25 2= 2:10:38,697] DEBUG {org.apache.stratos.autoscaler.KubernetesClusterContext}= - =C2=A0Memory consumption stats are reset, ready to do scale check [kub c= luster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBU= G {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - =C2=A0= Avg Memory Consumption event: [cluster] php.php.domain [value] 28.7
TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG {org.apache.stratos.a= utoscaler.KubernetesClusterContext} - =C2=A0Memory consumption stats are re= set, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRA= TOS] [2014-10-25 22:10:38,697] DEBUG {org.apache.stratos.autoscaler.monitor= .KubernetesClusterMonitor} - =C2=A0Avg load avg event: [cluster] php.php.do= main [value] 5.0
TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEB= UG {org.apache.stratos.autoscaler.KubernetesClusterContext} - =C2=A0Load av= erage stats are reset, ready to do scale check [kub cluster] KubGrp1
<= div>TID: [0] [STRATOS] [201= 4-10-25 22:10:38,697] DEBUG {org.apache.stratos.autoscaler.monitor.Kubernet= esClusterMonitor} - =C2=A0Grad of load avg event: [cluster] php.php.domain = [value] 2000.0
TID: [0] [STRATOS] [2014-10-25 22:10:38,698= ] DEBUG {org.apache.stratos.autoscaler.KubernetesClusterContext} - =C2=A0Lo= ad average stats are reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG {org.apache.str= atos.autoscaler.monitor.KubernetesClusterMonitor} - =C2=A0Avg Memory Consum= ption event: [cluster] php.php.domain [value] 28.7
TID: [0] [STRA= TOS] [2014-10-25 22:10:38,698] DEBUG {org.apache.stratos.autoscaler.Kuberne= tesClusterContext} - =C2=A0Memory consumption stats are reset, ready to do = scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 = 22:10:38,698] DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesCluste= rMonitor} - =C2=A0Grad of Memory Consumption event: [cluster] php.php.domai= n [value] 0.0
TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG = {org.apache.stratos.autoscaler.KubernetesClusterContext} - =C2=A0Memory con= sumption stats are reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,699] DEBUG {org.apache.strat= os.autoscaler.monitor.KubernetesClusterMonitor} - =C2=A0Second Derivation o= f Memory Consumption event: [cluster] php.php.domain [value] 0.0
= TID: [0] [STRATOS] [2014-10-25 22:10:38,699] DEBUG {org.apache.stratos.auto= scaler.KubernetesClusterContext} - =C2=A0Memory consumption stats are reset= , ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,700]= DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - = =C2=A0Second Derivation of load avg event: [cluster] php.php.domain [value]= 2000000.0
TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DE= BUG {org.apache.stratos.autoscaler.KubernetesClusterContext} - =C2=A0Load a= verage stats are reset, ready to do scale check [kub cluster] KubGrp1
=
TID: [0] [STRATOS] [20= 14-10-25 22:10:38,700] DEBUG {org.apache.stratos.autoscaler.monitor.Kuberne= tesClusterMonitor} - =C2=A0Grad of load avg event: [cluster] php.php.domain= [value] 13500.0
TID: [0] [STRATOS] [2014-10-25 22:10:38,7= 00] DEBUG {org.apache.stratos.autoscaler.KubernetesClusterContext} - =C2=A0= Load average stats are reset, ready to do scale check [kub cluster] KubGrp1=
TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG {org.apache.s= tratos.autoscaler.monitor.KubernetesClusterMonitor} - =C2=A0Grad of Memory = Consumption event: [cluster] php.php.domain [value] 50.0
TID: [0]= [STRATOS] [2014-10-25 22:10:38,701] DEBUG {org.apache.stratos.autoscaler.K= ubernetesClusterContext} - =C2=A0Memory consumption stats are reset, ready = to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-= 10-25 22:10:38,701] DEBUG {org.apache.stratos.autoscaler.monitor.Kubernetes= ClusterMonitor} - =C2=A0Avg Memory Consumption event: [cluster] php.php.dom= ain [value] 28.738462
TID: [0] [STRATOS] [2014-10-25 22:10:38,701= ] DEBUG {org.apache.stratos.autoscaler.KubernetesClusterContext} - =C2=A0Me= mory consumption stats are reset, ready to do scale check [kub cluster] Kub= Grp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG {org.apac= he.stratos.autoscaler.monitor.KubernetesClusterMonitor} - =C2=A0Avg load av= g event: [cluster] php.php.domain [value] 2.3333333
TID: [0] [STR= ATOS] [2014-10-25 22:10:38,702] DEBUG {org.apache.stratos.autoscaler.Kubern= etesClusterContext} - =C2=A0Load average stats are reset, ready to do scale= check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10= :38,702] DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMoni= tor} - =C2=A0Grad of load avg event: [cluster] php.php.domain [value] 0.0
TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG {org.apache.str= atos.autoscaler.KubernetesClusterContext} - =C2=A0Load average stats are re= set, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRA= TOS] [2014-10-25 22:10:38,703] DEBUG {org.apache.stratos.autoscaler.monitor= .KubernetesClusterMonitor} - =C2=A0Grad of Memory Consumption event: [clust= er] php.php.domain [value] 0.0
TID: [0] [STRATOS] [2014-10-25 22:= 10:38,703] DEBUG {org.apache.stratos.autoscaler.KubernetesClusterContext} -= =C2=A0Memory consumption stats are reset, ready to do scale check [kub clu= ster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG = {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - =C2=A0Av= g load avg event: [cluster] php.php.domain [value] 31.0
TID: [0] = [STRATOS] [2014-10-25 22:10:38,703] DEBUG {org.apache.stratos.autoscaler.Ku= bernetesClusterContext} - =C2=A0Load average stats are reset, ready to do s= cale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 2= 2:10:38,704] DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesCluster= Monitor} - =C2=A0Grad of Memory Consumption event: [cluster] php.php.domain= [value] 0.0
TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG {= org.apache.stratos.autoscaler.KubernetesClusterContext} - =C2=A0Memory cons= umption stats are reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG {org.apache.strato= s.autoscaler.monitor.KubernetesClusterMonitor} - =C2=A0Avg load avg event: = [cluster] php.php.domain [value] 39.0
TID: [0] [STRATOS] [2014-10= -25 22:10:38,704] DEBUG {org.apache.stratos.autoscaler.KubernetesClusterCon= text} - =C2=A0Load average stats are reset, ready to do scale check [kub cl= uster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG= {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - =C2=A0A= vg load avg event: [cluster] php.php.domain [value] 39.0
TID: [0]= [STRATOS] [2014-10-25 22:10:38,705] DEBUG {org.apache.stratos.autoscaler.K= ubernetesClusterContext} - =C2=A0Load average stats are reset, ready to do = scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,706] DEBUG {org.ap= ache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - =C2=A0Grad of l= oad avg event: [cluster] php.php.domain [value] -11000.0
T= ID: [0] [STRATOS] [2014-10-25 22:10:38,706] DEBUG {org.apache.stratos.autos= caler.KubernetesClusterContext} - =C2=A0Load average stats are reset, ready= to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014= -10-25 22:10:38,708] DEBUG {org.apache.stratos.autoscaler.monitor.Kubernete= sClusterMonitor} - =C2=A0Grad of Memory Consumption event: [cluster] php.ph= p.domain [value] 0.0
TID: [0] [STRATOS] [2014-10-25 22:10:38,708]= DEBUG {org.apache.stratos.autoscaler.KubernetesClusterContext} - =C2=A0Mem= ory consumption stats are reset, ready to do scale check [kub cluster] KubG= rp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,709] DEBUG {org.apach= e.stratos.autoscaler.monitor.KubernetesClusterMonitor} - =C2=A0Avg Memory C= onsumption event: [cluster] php.php.domain [value] 28.8
TID: [0] = [STRATOS] [2014-10-25 22:10:38,709] DEBUG {org.apache.stratos.autoscaler.Ku= bernetesClusterContext} - =C2=A0Memory consumption stats are reset, ready t= o do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-1= 0-25 22:10:38,811] DEBUG {org.apache.stratos.cloud.controller.util.PodActiv= ationWatcher} - =C2=A0PodActivationWatcher running : Running
TID:= [0] [STRATOS] [2014-10-25 22:10:38,942] DEBUG {org.apache.stratos.cloud.co= ntroller.util.PodActivationWatcher} - =C2=A0PodActivationWatcher running : = Running
TID: [0] [STRATOS] [2014-10-25 22:10:38,945] DEBUG {org.a= pache.stratos.cloud.controller.util.PodActivationWatcher} - =C2=A0PodActiva= tionWatcher running : Running
TID: [0] [STRATOS] [2014-10-25 22:1= 0:43,712] DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesServiceClu= sterMonitor} - =C2=A0KubernetesServiceClusterMonitor is running.. Kubernete= sServiceClusterMonitor [ kubernetesHostClusterId=3DKubGrp1, clusterId=3Dphp= .php.domain, serviceId=3Dphp]
TID: [0] [STRATOS] [2014-10-25 22:1= 0:43,713] DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesServiceClu= sterMonitor} - =C2=A0Running min check for [kub-cluster] : KubGrp1 [cluster= ] : php.php.domain=C2=A0
TID: [0] [STRATOS] [2014-10-25 22:10:43,= 713] =C2=A0INFO {org.apache.stratos.autoscaler.rule.RuleLog} - =C2=A0Runnin= g minimum rule: [kub-cluster] KubGrp1 [cluster] php.php.domain
TI= D: [0] [STRATOS] [2014-10-25 22:10:43,714] =C2=A0INFO {org.apache.stratos.a= utoscaler.rule.RuleLog} - =C2=A0[min-check] =C2=A0[cluster] : php.php.domai= n [Replicas] nonTerminated : 3
TID: [0] [STRATOS] [2014-10-25 22:= 10:43,714] =C2=A0INFO {org.apache.stratos.autoscaler.rule.RuleLog} - =C2=A0= [min-check] =C2=A0[cluster] : php.php.domain [Replicas] minReplicas : 3
TID: [0] [STRATOS] [2014-10-25 22:10:43,715] =C2=A0INFO {org.apache.= stratos.autoscaler.rule.RuleLog} - =C2=A0Running obsolete containers rule [= kub-cluster] : KubGrp1 [cluster] : php.php.domain
TID: [0] [STRAT= OS] [2014-10-25 22:10:43,715] =C2=A0INFO {org.apache.stratos.autoscaler.rul= e.RuleLog} - =C2=A0[obsolete-check] [cluster] : php.php.domain [Replicas] o= bsoleteReplicas : 0
TID: [0] [STRATOS] [2014-10-25 22:10:43,715] = DEBUG {org.apache.stratos.autoscaler.rule.AutoscalerRuleEvaluator} - =C2=A0= Minimum check executed for : org.apache.stratos.autoscaler.KubernetesCluste= rContext@21d906d0=C2=A0
TID: [0] [STRATOS] [2014-10-25 22:10:43,7= 15] DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMo= nitor} - =C2=A0flag of rifReset : true flag of memoryConsumptionReset : tru= e flag of loadAverageReset : true
TID: [0] [STRATOS] [2014-10-25 = 22:10:43,715] DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesServic= eClusterMonitor} - =C2=A0Running scale check for [kub-cluster] : KubGrp1 [c= luster] : php.php.domain=C2=A0
TID: [0] [STRATOS] [2014-10-25 22:= 10:43,716] =C2=A0INFO {org.apache.stratos.autoscaler.rule.RuleLog} - =C2=A0= Running scaling rule [kub-cluster] : KubGrp1 [cluster] : php.php.domain
TID: [0] [STRATOS] [2014-10-25 22:10:43,716] DEBUG {org.apache.strat= os.autoscaler.rule.RuleTasksDelegator} - =C2=A0Predicting the value, [avera= ge]: 0.0 , [gradient]: 0.0 , [second derivative]: 0.0 , [time intervals]: 1= =C2=A0
TID: [0] [STRATOS] [2014-10-25 22:10:43,716] DEBUG {org.ap= ache.stratos.autoscaler.rule.RuleTasksDelegator} - =C2=A0Predicting the val= ue, [average]: 28.8 , [gradient]: 0.0 , [second derivative]: 0.0 , [time in= tervals]: 1=C2=A0
TID: [0] [STRATOS] [2014-10-25 22:10:43,717] DE= BUG {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - =C2=A0Predict= ing the value, [average]: 39.0 , [gradient]: -11000.0 , [second derivative]= : 2000000.0 , [time intervals]: 1=C2=A0
TID: [0] [STRATOS] [2014-= 10-25 22:10:43,717] =C2=A0INFO {org.apache.stratos.autoscaler.rule.RuleLog}= - =C2=A0[scaling] =C2=A0[cluster] : php.php.domain [Replicas] minReplicas = : 3
TID: [0] [STRATOS] [2014-10-25 22:10:43,717] =C2=A0INFO {org.= apache.stratos.autoscaler.rule.RuleLog} - =C2=A0[scaling] =C2=A0[cluster] := php.php.domain [Replicas] maxReplicas : 10
TID: [0] [STRATOS] [2= 014-10-25 22:10:43,718] =C2=A0INFO {org.apache.stratos.autoscaler.rule.Rule= Log} - =C2=A0[scaling] =C2=A0[cluster] : php.php.domain [Replicas] nonTermi= nated : 3
TID: [0] [STRATOS] [2014-10-25 22:10:43,718] =C2=A0INFO= {org.apache.stratos.autoscaler.rule.RuleLog} - =C2=A0[scaling] =C2=A0[clus= ter] : php.php.domain [Replicas] activeReplicas : 3
TID: [0] [STR= ATOS] [2014-10-25 22:10:43,718] =C2=A0INFO {org.apache.stratos.autoscaler.r= ule.RuleLog} - =C2=A0[scaling] =C2=A0[cluster] : php.php.domain [RequestInF= light] predicted value : 0.0
TID: [0] [STRATOS] [2014-10-25 22:10= :43,718] =C2=A0INFO {org.apache.stratos.autoscaler.rule.RuleLog} - =C2=A0[s= caling] =C2=A0[cluster] : php.php.domain [RequestInFlight] upper limit : 80= .0
TID: [0] [STRATOS] [2014-10-25 22:10:43,718] =C2=A0INFO {org.a= pache.stratos.autoscaler.rule.RuleLog} - =C2=A0[scaling] =C2=A0[cluster] : = php.php.domain [RequestInFlight] lower limit : 5.0
TID: [0] [STRA= TOS] [2014-10-25 22:10:43,718] =C2=A0INFO {org.apache.stratos.autoscaler.ru= le.RuleLog} - =C2=A0[scaling] =C2=A0[cluster] : php.php.domain [MemoryConsu= mption] predicted value : 28.799999237060547
TID: [0] [STRATOS] [= 2014-10-25 22:10:43,718] =C2=A0INFO {org.apache.stratos.autoscaler.rule.Rul= eLog} - =C2=A0[scaling] =C2=A0[cluster] : php.php.domain [MemoryConsumption= ] upper limit : 80.0
TID: [0] [STRATOS] [2014-10-25 22:10:43,718]= =C2=A0INFO {org.apache.stratos.autoscaler.rule.RuleLog} - =C2=A0[scaling] = =C2=A0[cluster] : php.php.domain [MemoryConsumption] lower limit : 15.0
TID: [0] [STRATOS] [2014-10-25 22:10:43,718] =C2=A0INFO {org.apache.= stratos.autoscaler.rule.RuleLog} - =C2=A0[scaling] =C2=A0[cluster] : php.ph= p.domain [LoadAverage] predicted value : 989039.0
TID: [0] [STRAT= OS] [2014-10-25 22:10:43,719] =C2=A0INFO {org.apache.stratos.autoscaler.rul= e.RuleLog} - =C2=A0[scaling] =C2=A0[cluster] : php.php.domain [LoadAverage]= upper limit : 180.0
TID: [0] [STRATOS] [2014-10-25 22:10:43,719]= =C2=A0INFO {org.apache.stratos.autoscaler.rule.RuleLog} - =C2=A0[scaling] = =C2=A0[cluster] : php.php.domain [LoadAverage] lower limit : 20.0
TID: [0] [STRATOS] [2014-10-25 22:10:43,719] =C2=A0INFO {org.apache.strato= s.autoscaler.rule.RuleLog} - =C2=A0[scaling] =C2=A0[cluster] : php.php.doma= in scale-up action : true
TID: [0] [STRATOS] [2014-10-25 22:10:43= ,719] =C2=A0INFO {org.apache.stratos.autoscaler.rule.RuleLog} - =C2=A0[scal= ing] =C2=A0[cluster] : php.php.domain scale-down action : false
T= ID: [0] [STRATOS] [2014-10-25 22:10:43,719] =C2=A0INFO {org.apache.stratos.= autoscaler.rule.RuleLog} - =C2=A0[scaling] =C2=A0[cluster] : php.php.domain= [LoadAverage] predicted replicas : 16484
TID: [0] [STRATOS] [201= 4-10-25 22:10:43,720] =C2=A0INFO {org.apache.stratos.autoscaler.rule.RuleLo= g} - =C2=A0[scaling] =C2=A0[cluster] : php.php.domain predicted replicas &g= t; max replicas :=C2=A0
TID: [0] [STRATOS] [2014-10-25 22:10:43,7= 20] =C2=A0INFO {org.apache.stratos.autoscaler.rule.RuleLog} - =C2=A0[scalin= g] Decided to scale-up : [cluster] : php.php.domain
TID: [0] [STR= ATOS] [2014-10-25 22:10:43,720] =C2=A0INFO {org.apache.stratos.autoscaler.r= ule.RuleLog} - =C2=A0[scaling-up] =C2=A0[cluster] : php.php.domain valid nu= mber of replicas to expand : 10
TID: [0] [STRATOS] [2014-10-25 22= :10:43,720] =C2=A0INFO {org.apache.stratos.autoscaler.client.cloud.controll= er.CloudControllerClient} - =C2=A0Updating kubernetes replication controlle= r via cloud controller: [cluster] php.php.domain [replicas] 10
TI= D: [0] [STRATOS] [2014-10-25 22:10:43,734] DEBUG {org.apache.stratos.cloud.= controller.impl.CloudControllerServiceImpl} - =C2=A0CloudControllerServiceI= mpl:updateContainers for cluster : php.php.domain
TID: [0] [STRAT= OS] [2014-10-25 22:10:43,758] DEBUG {org.apache.stratos.cloud.controller.im= pl.CloudControllerServiceImpl} - =C2=A0Cloud Controller is delegating reque= st to update a replication controller php.php.domain to Kubernetes layer.


I couldn't investigate the agent= side, since containers are destroyed.

Any idea?

Thanks.
--
= Rajkumar Rajaratnam
Software Engineer= | WSO2, Inc.




--
Regards,
<= /div>
Manula Chathurika Thantriwatte
Software Engineer
WSO2 Inc. : http:= //wso2.com
lean . enterprise . middleware

email : manulac@wso2.com / = manula@apache.org






--
Rajkumar Rajaratnam
Software Engineer | WSO2, Inc.
Mobile +94777568639 | +94783498120
--001a11c20b2c1b70fd05064a9a77--