Return-Path: X-Original-To: apmail-stratos-dev-archive@minotaur.apache.org Delivered-To: apmail-stratos-dev-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 1039217607 for ; Sun, 26 Oct 2014 02:54:17 +0000 (UTC) Received: (qmail 39046 invoked by uid 500); 26 Oct 2014 02:54:17 -0000 Delivered-To: apmail-stratos-dev-archive@stratos.apache.org Received: (qmail 38990 invoked by uid 500); 26 Oct 2014 02:54:17 -0000 Mailing-List: contact dev-help@stratos.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@stratos.apache.org Delivered-To: mailing list dev@stratos.apache.org Received: (qmail 38980 invoked by uid 99); 26 Oct 2014 02:54:16 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 26 Oct 2014 02:54:16 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_FILL_THIS_FORM_SHORT X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of manulac@wso2.com designates 209.85.214.178 as permitted sender) Received: from [209.85.214.178] (HELO mail-ob0-f178.google.com) (209.85.214.178) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 26 Oct 2014 02:53:48 +0000 Received: by mail-ob0-f178.google.com with SMTP id wm4so598515obc.37 for ; Sat, 25 Oct 2014 19:53:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=wso2.com; s=google; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=GTzA9czO9Q9rXgFvb/OO3MVx9/dw2Pt8jyed+IcUZgQ=; b=EaRd3me3Teg3PTtpJ+JmUqa6vs9DZZV2IbipPx/IP2xIl0Lo86NS4+PS4DYueuh3o3 0pps5DRmIczMjkM0vRfwpsPJa2FaQxRQK6//dz62SuS52NwRLX/juJkJCSlhRcFp5FoJ XbYWakkE+Zqm75bKVUaoOVwTZpPe971NBD+6I= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:content-type; bh=GTzA9czO9Q9rXgFvb/OO3MVx9/dw2Pt8jyed+IcUZgQ=; b=VmiSNIzUQG5//pIST8iiDPBSJOD8ruEW3Rzyx4vym8P1OJ4+SR8+6SZshf4zXTZ1Az bI1IKw2OG3A0BxtlKVk6Jo1Sdv9x+qnD+EzBdSfa5gSiLfQVNtcn1B4kjqwU4J8AIV2e D/H1U3LGhgSMQf7nMbJsKBu9VOXN6z6m8YtnJiOpQaF5NcTCuzmvl4WhJdTUsQiYj7u+ tRtAsbW1h0gzhsXtZWIpM1PWfWFKtOkm4dEKu6W3u+SKg9Nw1Q/r1V7NsjJvXHWW0O79 OV/733QXu3RR4LK32HtbQ3UZaVvrfcjKjG2K+5LxpakuqOOFW+kWdo/o5QNHb8LdgRkC x4oQ== X-Gm-Message-State: ALoCoQmeBIF0p0O9aVP7Gz9X4WWczX78b5AkXSg3cO2dcCWVAHzm+lqhsK/7kgkt9tKmKpSY24hb MIME-Version: 1.0 X-Received: by 10.202.49.142 with SMTP id x136mr11517649oix.17.1414291981334; Sat, 25 Oct 2014 19:53:01 -0700 (PDT) Received: by 10.76.7.116 with HTTP; Sat, 25 Oct 2014 19:53:01 -0700 (PDT) In-Reply-To: References: Date: Sun, 26 Oct 2014 08:23:01 +0530 Message-ID: Subject: Re: CEP sends very large values for gradient and second derivative of load average From: Manula Chathurika Thantriwatte To: "dev@stratos.apache.org" Content-Type: multipart/alternative; boundary=001a113cdcaa280e1005064a8321 X-Virus-Checked: Checked by ClamAV on apache.org --001a113cdcaa280e1005064a8321 Content-Type: text/plain; charset=UTF-8 Hi, Shall we enable the cartridge agent debug logs and see what are the load average values sent from the agent. If those values are normal then we can narrow down to the CEP. Thanks ! On Sat, Oct 25, 2014 at 10:43 PM, Chamila De Alwis wrote: > Hi Raj, > > Can you inspect the CEP trace logs for the same time period? It will have > the events published from the agent. > > > Regards, > Chamila de Alwis > Software Engineer | WSO2 | +94772207163 > Blog: code.chamiladealwis.com > > > > On Sat, Oct 25, 2014 at 10:39 PM, Rajkumar Rajaratnam > wrote: > >> Hi, >> >> I am doing a test round with M3 with python agent. And faced this very >> strange scenario, $Subject. >> >> >> TID: [0] [STRATOS] [2014-10-25 22:10:38,693] DEBUG >> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average >> stats are reset, ready to do scale check [kub cluster] KubGrp1 >> TID: [0] [STRATOS] [2014-10-25 22:10:38,693] DEBUG >> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of >> load avg event: [cluster] php.php.domain [value] -2000.0 >> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG >> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average >> stats are reset, ready to do scale check [kub cluster] KubGrp1 >> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG >> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of >> load avg event: [cluster] php.php.domain [value] 0.0 >> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG >> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average >> stats are reset, ready to do scale check [kub cluster] KubGrp1 >> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG >> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg >> load avg event: [cluster] php.php.domain [value] 1.9090909 >> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG >> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average >> stats are reset, ready to do scale check [kub cluster] KubGrp1 >> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG >> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg >> Memory Consumption event: [cluster] php.php.domain [value] 28.718182 >> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG >> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second >> Derivation of Memory Consumption event: [cluster] php.php.domain [value] >> 200000.0 >> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG >> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of >> load avg event: [cluster] php.php.domain [value] 0.0 >> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG >> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average >> stats are reset, ready to do scale check [kub cluster] KubGrp1 >> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG >> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second >> Derivation of load avg event: [cluster] php.php.domain [value] 0.0 >> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG >> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average >> stats are reset, ready to do scale check [kub cluster] KubGrp1 >> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG >> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second >> Derivation of load avg event: [cluster] php.php.domain [value] 0.0 >> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG >> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average >> stats are reset, ready to do scale check [kub cluster] KubGrp1 >> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG >> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg >> load avg event: [cluster] php.php.domain [value] 3.3636363 >> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG >> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average >> stats are reset, ready to do scale check [kub cluster] KubGrp1 >> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG >> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of >> Memory Consumption event: [cluster] php.php.domain [value] 0.11670045 >> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG >> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory >> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1 >> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG >> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg >> Memory Consumption event: [cluster] php.php.domain [value] 28.7 >> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG >> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory >> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1 >> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG >> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second >> Derivation of Memory Consumption event: [cluster] php.php.domain [value] 0.0 >> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG >> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory >> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1 >> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG >> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg >> Memory Consumption event: [cluster] php.php.domain [value] 28.7 >> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG >> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory >> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1 >> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG >> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg >> load avg event: [cluster] php.php.domain [value] 5.0 >> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG >> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average >> stats are reset, ready to do scale check [kub cluster] KubGrp1 >> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG >> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of >> load avg event: [cluster] php.php.domain [value] 2000.0 >> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG >> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average >> stats are reset, ready to do scale check [kub cluster] KubGrp1 >> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG >> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg >> Memory Consumption event: [cluster] php.php.domain [value] 28.7 >> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG >> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory >> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1 >> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG >> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of >> Memory Consumption event: [cluster] php.php.domain [value] 0.0 >> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG >> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory >> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1 >> TID: [0] [STRATOS] [2014-10-25 22:10:38,699] DEBUG >> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second >> Derivation of Memory Consumption event: [cluster] php.php.domain [value] 0.0 >> TID: [0] [STRATOS] [2014-10-25 22:10:38,699] DEBUG >> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory >> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1 >> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG >> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second >> Derivation of load avg event: [cluster] php.php.domain [value] 2000000.0 >> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG >> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average >> stats are reset, ready to do scale check [kub cluster] KubGrp1 >> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG >> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of >> load avg event: [cluster] php.php.domain [value] 13500.0 >> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG >> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average >> stats are reset, ready to do scale check [kub cluster] KubGrp1 >> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG >> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of >> Memory Consumption event: [cluster] php.php.domain [value] 50.0 >> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG >> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory >> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1 >> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG >> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg >> Memory Consumption event: [cluster] php.php.domain [value] 28.738462 >> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG >> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory >> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1 >> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG >> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg >> load avg event: [cluster] php.php.domain [value] 2.3333333 >> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG >> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average >> stats are reset, ready to do scale check [kub cluster] KubGrp1 >> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG >> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of >> load avg event: [cluster] php.php.domain [value] 0.0 >> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG >> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average >> stats are reset, ready to do scale check [kub cluster] KubGrp1 >> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG >> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of >> Memory Consumption event: [cluster] php.php.domain [value] 0.0 >> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG >> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory >> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1 >> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG >> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg >> load avg event: [cluster] php.php.domain [value] 31.0 >> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG >> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average >> stats are reset, ready to do scale check [kub cluster] KubGrp1 >> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG >> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of >> Memory Consumption event: [cluster] php.php.domain [value] 0.0 >> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG >> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory >> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1 >> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG >> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg >> load avg event: [cluster] php.php.domain [value] 39.0 >> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG >> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average >> stats are reset, ready to do scale check [kub cluster] KubGrp1 >> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG >> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg >> load avg event: [cluster] php.php.domain [value] 39.0 >> TID: [0] [STRATOS] [2014-10-25 22:10:38,705] DEBUG >> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average >> stats are reset, ready to do scale check [kub cluster] KubGrp1 >> TID: [0] [STRATOS] [2014-10-25 22:10:38,706] DEBUG >> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of >> load avg event: [cluster] php.php.domain [value] -11000.0 >> TID: [0] [STRATOS] [2014-10-25 22:10:38,706] DEBUG >> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average >> stats are reset, ready to do scale check [kub cluster] KubGrp1 >> TID: [0] [STRATOS] [2014-10-25 22:10:38,708] DEBUG >> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of >> Memory Consumption event: [cluster] php.php.domain [value] 0.0 >> TID: [0] [STRATOS] [2014-10-25 22:10:38,708] DEBUG >> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory >> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1 >> TID: [0] [STRATOS] [2014-10-25 22:10:38,709] DEBUG >> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg >> Memory Consumption event: [cluster] php.php.domain [value] 28.8 >> TID: [0] [STRATOS] [2014-10-25 22:10:38,709] DEBUG >> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory >> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1 >> TID: [0] [STRATOS] [2014-10-25 22:10:38,811] DEBUG >> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} - >> PodActivationWatcher running : Running >> TID: [0] [STRATOS] [2014-10-25 22:10:38,942] DEBUG >> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} - >> PodActivationWatcher running : Running >> TID: [0] [STRATOS] [2014-10-25 22:10:38,945] DEBUG >> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} - >> PodActivationWatcher running : Running >> TID: [0] [STRATOS] [2014-10-25 22:10:43,712] DEBUG >> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} - >> KubernetesServiceClusterMonitor is running.. >> KubernetesServiceClusterMonitor [ kubernetesHostClusterId=KubGrp1, >> clusterId=php.php.domain, serviceId=php] >> TID: [0] [STRATOS] [2014-10-25 22:10:43,713] DEBUG >> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} - >> Running min check for [kub-cluster] : KubGrp1 [cluster] : php.php.domain >> TID: [0] [STRATOS] [2014-10-25 22:10:43,713] INFO >> {org.apache.stratos.autoscaler.rule.RuleLog} - Running minimum rule: >> [kub-cluster] KubGrp1 [cluster] php.php.domain >> TID: [0] [STRATOS] [2014-10-25 22:10:43,714] INFO >> {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] [cluster] : >> php.php.domain [Replicas] nonTerminated : 3 >> TID: [0] [STRATOS] [2014-10-25 22:10:43,714] INFO >> {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] [cluster] : >> php.php.domain [Replicas] minReplicas : 3 >> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] INFO >> {org.apache.stratos.autoscaler.rule.RuleLog} - Running obsolete containers >> rule [kub-cluster] : KubGrp1 [cluster] : php.php.domain >> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] INFO >> {org.apache.stratos.autoscaler.rule.RuleLog} - [obsolete-check] [cluster] >> : php.php.domain [Replicas] obsoleteReplicas : 0 >> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG >> {org.apache.stratos.autoscaler.rule.AutoscalerRuleEvaluator} - Minimum >> check executed for : >> org.apache.stratos.autoscaler.KubernetesClusterContext@21d906d0 >> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG >> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} - >> flag of rifReset : true flag of memoryConsumptionReset : true flag of >> loadAverageReset : true >> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG >> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} - >> Running scale check for [kub-cluster] : KubGrp1 [cluster] : php.php.domain >> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] INFO >> {org.apache.stratos.autoscaler.rule.RuleLog} - Running scaling rule >> [kub-cluster] : KubGrp1 [cluster] : php.php.domain >> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] DEBUG >> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the >> value, [average]: 0.0 , [gradient]: 0.0 , [second derivative]: 0.0 , [time >> intervals]: 1 >> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] DEBUG >> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the >> value, [average]: 28.8 , [gradient]: 0.0 , [second derivative]: 0.0 , [time >> intervals]: 1 >> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] DEBUG >> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the >> value, [average]: 39.0 , [gradient]: -11000.0 , [second derivative]: >> 2000000.0 , [time intervals]: 1 >> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] INFO >> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : >> php.php.domain [Replicas] minReplicas : 3 >> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] INFO >> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : >> php.php.domain [Replicas] maxReplicas : 10 >> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO >> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : >> php.php.domain [Replicas] nonTerminated : 3 >> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO >> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : >> php.php.domain [Replicas] activeReplicas : 3 >> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO >> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : >> php.php.domain [RequestInFlight] predicted value : 0.0 >> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO >> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : >> php.php.domain [RequestInFlight] upper limit : 80.0 >> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO >> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : >> php.php.domain [RequestInFlight] lower limit : 5.0 >> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO >> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : >> php.php.domain [MemoryConsumption] predicted value : 28.799999237060547 >> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO >> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : >> php.php.domain [MemoryConsumption] upper limit : 80.0 >> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO >> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : >> php.php.domain [MemoryConsumption] lower limit : 15.0 >> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO >> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : >> php.php.domain [LoadAverage] predicted value : 989039.0 >> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO >> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : >> php.php.domain [LoadAverage] upper limit : 180.0 >> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO >> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : >> php.php.domain [LoadAverage] lower limit : 20.0 >> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO >> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : >> php.php.domain scale-up action : true >> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO >> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : >> php.php.domain scale-down action : false >> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO >> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : >> php.php.domain [LoadAverage] predicted replicas : 16484 >> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO >> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : >> php.php.domain predicted replicas > max replicas : >> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO >> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] Decided to >> scale-up : [cluster] : php.php.domain >> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO >> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling-up] [cluster] : >> php.php.domain valid number of replicas to expand : 10 >> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO >> {org.apache.stratos.autoscaler.client.cloud.controller.CloudControllerClient} >> - Updating kubernetes replication controller via cloud controller: >> [cluster] php.php.domain [replicas] 10 >> TID: [0] [STRATOS] [2014-10-25 22:10:43,734] DEBUG >> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} - >> CloudControllerServiceImpl:updateContainers for cluster : php.php.domain >> TID: [0] [STRATOS] [2014-10-25 22:10:43,758] DEBUG >> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} - >> Cloud Controller is delegating request to update a replication controller >> php.php.domain to Kubernetes layer. >> >> >> I couldn't investigate the agent side, since containers are destroyed. >> >> Any idea? >> >> Thanks. >> >> -- >> Rajkumar Rajaratnam >> Software Engineer | WSO2, Inc. >> Mobile +94777568639 | +94783498120 >> > > -- Regards, Manula Chathurika Thantriwatte Software Engineer WSO2 Inc. : http://wso2.com lean . enterprise . middleware email : manulac@wso2.com / manula@apache.org phone : +94 772492511 blog : http://manulachathurika.blogspot.com/ --001a113cdcaa280e1005064a8321 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hi,

Shall we enable the cartridge agent= debug logs and see what are the load average values sent from the agent. I= f those values are normal then we can narrow down to the CEP.
Thanks !

On Sat, Oct 25, 2014 at 10:43 PM, Chamila De Alwis <chami= lad@wso2.com> wrote:
Hi Raj,

Can you inspect the CEP trace logs fo= r the same time period? It will have the events published from the agent.

=

Regards,
Chamila de Alwis
<= span style=3D"color:rgb(153,153,153)">Software Engineer | WSO2 | +94772207163=
Bl= og: code.chami= ladealwis.com



On Sat, Oct 25, 2014 at 10:= 39 PM, Rajkumar Rajaratnam <rajkumarr@wso2.com> wrote:
<= /span>
Hi,

I am doing a test round with M3 with python agent.= And faced this very strange scenario, $Subject.

<= br>
TID: [0] [STRATOS] [2014-10-25 22:10:38,693] DEBUG {org.= apache.stratos.autoscaler.KubernetesClusterContext} - =C2=A0Load average st= ats are reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 = 22:10:38,693] DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesCluste= rMonitor} - =C2=A0Grad of load avg event: [cluster] php.php.domain [value] = -2000.0
TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG= {org.apache.stratos.autoscaler.KubernetesClusterContext} - =C2=A0Load aver= age stats are reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG {org.apache.stratos.au= toscaler.monitor.KubernetesClusterMonitor} - =C2=A0Grad of load avg event: = [cluster] php.php.domain [value] 0.0
TID: [0] [STRATOS] [2014-10-= 25 22:10:38,694] DEBUG {org.apache.stratos.autoscaler.KubernetesClusterCont= ext} - =C2=A0Load average stats are reset, ready to do scale check [kub clu= ster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG = {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - =C2=A0Av= g load avg event: [cluster] php.php.domain [value] 1.9090909
TID:= [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG {org.apache.stratos.autoscal= er.KubernetesClusterContext} - =C2=A0Load average stats are reset, ready to= do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10= -25 22:10:38,694] DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesCl= usterMonitor} - =C2=A0Avg Memory Consumption event: [cluster] php.php.domai= n [value] 28.718182
TID: [0] [STRATOS] [2014-10-25 22:10:38,695] = DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - = =C2=A0Second Derivation of Memory Consumption event: [cluster] php.php.doma= in [value] 200000.0
TID: [0] [STRATOS] [2014-10-25 22:10:38,695] = DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - = =C2=A0Grad of load avg event: [cluster] php.php.domain [value] 0.0
TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG {org.apache.stratos.au= toscaler.KubernetesClusterContext} - =C2=A0Load average stats are reset, re= ady to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2= 014-10-25 22:10:38,695] DEBUG {org.apache.stratos.autoscaler.monitor.Kubern= etesClusterMonitor} - =C2=A0Second Derivation of load avg event: [cluster] = php.php.domain [value] 0.0
TID: [0] [STRATOS] [2014-10-25 22:10:3= 8,695] DEBUG {org.apache.stratos.autoscaler.KubernetesClusterContext} - =C2= =A0Load average stats are reset, ready to do scale check [kub cluster] KubG= rp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG {org.apach= e.stratos.autoscaler.monitor.KubernetesClusterMonitor} - =C2=A0Second Deriv= ation of load avg event: [cluster] php.php.domain [value] 0.0
TID= : [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG {org.apache.stratos.autosca= ler.KubernetesClusterContext} - =C2=A0Load average stats are reset, ready t= o do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-1= 0-25 22:10:38,696] DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesC= lusterMonitor} - =C2=A0Avg load avg event: [cluster] php.php.domain [value]= 3.3636363
TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG {or= g.apache.stratos.autoscaler.KubernetesClusterContext} - =C2=A0Load average = stats are reset, ready to do scale check [kub cluster] KubGrp1
TI= D: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG {org.apache.stratos.autosc= aler.monitor.KubernetesClusterMonitor} - =C2=A0Grad of Memory Consumption e= vent: [cluster] php.php.domain [value] 0.11670045
TID: [0] [STRAT= OS] [2014-10-25 22:10:38,696] DEBUG {org.apache.stratos.autoscaler.Kubernet= esClusterContext} - =C2=A0Memory consumption stats are reset, ready to do s= cale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 2= 2:10:38,697] DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesCluster= Monitor} - =C2=A0Avg Memory Consumption event: [cluster] php.php.domain [va= lue] 28.7
TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG {org= .apache.stratos.autoscaler.KubernetesClusterContext} - =C2=A0Memory consump= tion stats are reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG {org.apache.stratos.a= utoscaler.monitor.KubernetesClusterMonitor} - =C2=A0Second Derivation of Me= mory Consumption event: [cluster] php.php.domain [value] 0.0
TID:= [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG {org.apache.stratos.autoscal= er.KubernetesClusterContext} - =C2=A0Memory consumption stats are reset, re= ady to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2= 014-10-25 22:10:38,697] DEBUG {org.apache.stratos.autoscaler.monitor.Kubern= etesClusterMonitor} - =C2=A0Avg Memory Consumption event: [cluster] php.php= .domain [value] 28.7
TID: [0] [STRATOS] [2014-10-25 22:10:38,697]= DEBUG {org.apache.stratos.autoscaler.KubernetesClusterContext} - =C2=A0Mem= ory consumption stats are reset, ready to do scale check [kub cluster] KubG= rp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG {org.apach= e.stratos.autoscaler.monitor.KubernetesClusterMonitor} - =C2=A0Avg load avg= event: [cluster] php.php.domain [value] 5.0
TID: [0] [STRATOS] [= 2014-10-25 22:10:38,697] DEBUG {org.apache.stratos.autoscaler.KubernetesClu= sterContext} - =C2=A0Load average stats are reset, ready to do scale check = [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG {org.apache.stratos= .autoscaler.monitor.KubernetesClusterMonitor} - =C2=A0Grad of load avg even= t: [cluster] php.php.domain [value] 2000.0
TID: [0] [STRAT= OS] [2014-10-25 22:10:38,698] DEBUG {org.apache.stratos.autoscaler.Kubernet= esClusterContext} - =C2=A0Load average stats are reset, ready to do scale c= heck [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:3= 8,698] DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonito= r} - =C2=A0Avg Memory Consumption event: [cluster] php.php.domain [value] 2= 8.7
TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG {org.apach= e.stratos.autoscaler.KubernetesClusterContext} - =C2=A0Memory consumption s= tats are reset, ready to do scale check [kub cluster] KubGrp1
TID= : [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG {org.apache.stratos.autosca= ler.monitor.KubernetesClusterMonitor} - =C2=A0Grad of Memory Consumption ev= ent: [cluster] php.php.domain [value] 0.0
TID: [0] [STRATOS] [201= 4-10-25 22:10:38,698] DEBUG {org.apache.stratos.autoscaler.KubernetesCluste= rContext} - =C2=A0Memory consumption stats are reset, ready to do scale che= ck [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,= 699] DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor}= - =C2=A0Second Derivation of Memory Consumption event: [cluster] php.php.d= omain [value] 0.0
TID: [0] [STRATOS] [2014-10-25 22:10:38,699] DE= BUG {org.apache.stratos.autoscaler.KubernetesClusterContext} - =C2=A0Memory= consumption stats are reset, ready to do scale check [kub cluster] KubGrp1=
TID: [0] [STRATO= S] [2014-10-25 22:10:38,700] DEBUG {org.apache.stratos.autoscaler.monitor.K= ubernetesClusterMonitor} - =C2=A0Second Derivation of load avg event: [clus= ter] php.php.domain [value] 2000000.0
TID: [0] [STRATOS] [= 2014-10-25 22:10:38,700] DEBUG {org.apache.stratos.autoscaler.KubernetesClu= sterContext} - =C2=A0Load average stats are reset, ready to do scale check = [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG {org.apache.stratos= .autoscaler.monitor.KubernetesClusterMonitor} - =C2=A0Grad of load avg even= t: [cluster] php.php.domain [value] 13500.0
TID: [0] [STRA= TOS] [2014-10-25 22:10:38,700] DEBUG {org.apache.stratos.autoscaler.Kuberne= tesClusterContext} - =C2=A0Load average stats are reset, ready to do scale = check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:= 38,701] DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonit= or} - =C2=A0Grad of Memory Consumption event: [cluster] php.php.domain [val= ue] 50.0
TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG {org.= apache.stratos.autoscaler.KubernetesClusterContext} - =C2=A0Memory consumpt= ion stats are reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG {org.apache.stratos.au= toscaler.monitor.KubernetesClusterMonitor} - =C2=A0Avg Memory Consumption e= vent: [cluster] php.php.domain [value] 28.738462
TID: [0] [STRATO= S] [2014-10-25 22:10:38,701] DEBUG {org.apache.stratos.autoscaler.Kubernete= sClusterContext} - =C2=A0Memory consumption stats are reset, ready to do sc= ale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22= :10:38,702] DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterM= onitor} - =C2=A0Avg load avg event: [cluster] php.php.domain [value] 2.3333= 333
TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG {org.apach= e.stratos.autoscaler.KubernetesClusterContext} - =C2=A0Load average stats a= re reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] = [STRATOS] [2014-10-25 22:10:38,702] DEBUG {org.apache.stratos.autoscaler.mo= nitor.KubernetesClusterMonitor} - =C2=A0Grad of load avg event: [cluster] p= hp.php.domain [value] 0.0
TID: [0] [STRATOS] [2014-10-25 22:10:38= ,702] DEBUG {org.apache.stratos.autoscaler.KubernetesClusterContext} - =C2= =A0Load average stats are reset, ready to do scale check [kub cluster] KubG= rp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG {org.apach= e.stratos.autoscaler.monitor.KubernetesClusterMonitor} - =C2=A0Grad of Memo= ry Consumption event: [cluster] php.php.domain [value] 0.0
TID: [= 0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG {org.apache.stratos.autoscaler= .KubernetesClusterContext} - =C2=A0Memory consumption stats are reset, read= y to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [201= 4-10-25 22:10:38,703] DEBUG {org.apache.stratos.autoscaler.monitor.Kubernet= esClusterMonitor} - =C2=A0Avg load avg event: [cluster] php.php.domain [val= ue] 31.0
TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG {org.= apache.stratos.autoscaler.KubernetesClusterContext} - =C2=A0Load average st= ats are reset, ready to do scale check [kub cluster] KubGrp1
TID:= [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG {org.apache.stratos.autoscal= er.monitor.KubernetesClusterMonitor} - =C2=A0Grad of Memory Consumption eve= nt: [cluster] php.php.domain [value] 0.0
TID: [0] [STRATOS] [2014= -10-25 22:10:38,704] DEBUG {org.apache.stratos.autoscaler.KubernetesCluster= Context} - =C2=A0Memory consumption stats are reset, ready to do scale chec= k [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,7= 04] DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} = - =C2=A0Avg load avg event: [cluster] php.php.domain [value] 39.0
TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG {org.apache.stratos.aut= oscaler.KubernetesClusterContext} - =C2=A0Load average stats are reset, rea= dy to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [20= 14-10-25 22:10:38,704] DEBUG {org.apache.stratos.autoscaler.monitor.Kuberne= tesClusterMonitor} - =C2=A0Avg load avg event: [cluster] php.php.domain [va= lue] 39.0
TID: [0] [STRATOS] [2014-10-25 22:10:38,705] DEBUG {org= .apache.stratos.autoscaler.KubernetesClusterContext} - =C2=A0Load average s= tats are reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25= 22:10:38,706] DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClust= erMonitor} - =C2=A0Grad of load avg event: [cluster] php.php.domain [value]= -11000.0
TID: [0] [STRATOS] [2014-10-25 22:10:38,706] DEB= UG {org.apache.stratos.autoscaler.KubernetesClusterContext} - =C2=A0Load av= erage stats are reset, ready to do scale check [kub cluster] KubGrp1
<= div>TID: [0] [STRATOS] [2014-10-25 22:10:38,708] DEBUG {org.apache.stratos.= autoscaler.monitor.KubernetesClusterMonitor} - =C2=A0Grad of Memory Consump= tion event: [cluster] php.php.domain [value] 0.0
TID: [0] [STRATO= S] [2014-10-25 22:10:38,708] DEBUG {org.apache.stratos.autoscaler.Kubernete= sClusterContext} - =C2=A0Memory consumption stats are reset, ready to do sc= ale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22= :10:38,709] DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterM= onitor} - =C2=A0Avg Memory Consumption event: [cluster] php.php.domain [val= ue] 28.8
TID: [0] [STRATOS] [2014-10-25 22:10:38,709] DEBUG {org.= apache.stratos.autoscaler.KubernetesClusterContext} - =C2=A0Memory consumpt= ion stats are reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,811] DEBUG {org.apache.stratos.cl= oud.controller.util.PodActivationWatcher} - =C2=A0PodActivationWatcher runn= ing : Running
TID: [0] [STRATOS] [2014-10-25 22:10:38,942] DEBUG = {org.apache.stratos.cloud.controller.util.PodActivationWatcher} - =C2=A0Pod= ActivationWatcher running : Running
TID: [0] [STRATOS] [2014-10-2= 5 22:10:38,945] DEBUG {org.apache.stratos.cloud.controller.util.PodActivati= onWatcher} - =C2=A0PodActivationWatcher running : Running
TID: [0= ] [STRATOS] [2014-10-25 22:10:43,712] DEBUG {org.apache.stratos.autoscaler.= monitor.KubernetesServiceClusterMonitor} - =C2=A0KubernetesServiceClusterMo= nitor is running.. KubernetesServiceClusterMonitor [ kubernetesHostClusterI= d=3DKubGrp1, clusterId=3Dphp.php.domain, serviceId=3Dphp]
TID: [0= ] [STRATOS] [2014-10-25 22:10:43,713] DEBUG {org.apache.stratos.autoscaler.= monitor.KubernetesServiceClusterMonitor} - =C2=A0Running min check for [kub= -cluster] : KubGrp1 [cluster] : php.php.domain=C2=A0
TID: [0] [ST= RATOS] [2014-10-25 22:10:43,713] =C2=A0INFO {org.apache.stratos.autoscaler.= rule.RuleLog} - =C2=A0Running minimum rule: [kub-cluster] KubGrp1 [cluster]= php.php.domain
TID: [0] [STRATOS] [2014-10-25 22:10:43,714] =C2= =A0INFO {org.apache.stratos.autoscaler.rule.RuleLog} - =C2=A0[min-check] = =C2=A0[cluster] : php.php.domain [Replicas] nonTerminated : 3
TID= : [0] [STRATOS] [2014-10-25 22:10:43,714] =C2=A0INFO {org.apache.stratos.au= toscaler.rule.RuleLog} - =C2=A0[min-check] =C2=A0[cluster] : php.php.domain= [Replicas] minReplicas : 3
TID: [0] [STRATOS] [2014-10-25 22:10:= 43,715] =C2=A0INFO {org.apache.stratos.autoscaler.rule.RuleLog} - =C2=A0Run= ning obsolete containers rule [kub-cluster] : KubGrp1 [cluster] : php.php.d= omain
TID: [0] [STRATOS] [2014-10-25 22:10:43,715] =C2=A0INFO {or= g.apache.stratos.autoscaler.rule.RuleLog} - =C2=A0[obsolete-check] [cluster= ] : php.php.domain [Replicas] obsoleteReplicas : 0
TID: [0] [STRA= TOS] [2014-10-25 22:10:43,715] DEBUG {org.apache.stratos.autoscaler.rule.Au= toscalerRuleEvaluator} - =C2=A0Minimum check executed for : org.apache.stra= tos.autoscaler.KubernetesClusterContext@21d906d0=C2=A0
TID: [0] [= STRATOS] [2014-10-25 22:10:43,715] DEBUG {org.apache.stratos.autoscaler.mon= itor.KubernetesServiceClusterMonitor} - =C2=A0flag of rifReset : true flag = of memoryConsumptionReset : true flag of loadAverageReset : true
= TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG {org.apache.stratos.auto= scaler.monitor.KubernetesServiceClusterMonitor} - =C2=A0Running scale check= for [kub-cluster] : KubGrp1 [cluster] : php.php.domain=C2=A0
TID= : [0] [STRATOS] [2014-10-25 22:10:43,716] =C2=A0INFO {org.apache.stratos.au= toscaler.rule.RuleLog} - =C2=A0Running scaling rule [kub-cluster] : KubGrp1= [cluster] : php.php.domain
TID: [0] [STRATOS] [2014-10-25 22:10:= 43,716] DEBUG {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - =C2= =A0Predicting the value, [average]: 0.0 , [gradient]: 0.0 , [second derivat= ive]: 0.0 , [time intervals]: 1=C2=A0
TID: [0] [STRATOS] [2014-10= -25 22:10:43,716] DEBUG {org.apache.stratos.autoscaler.rule.RuleTasksDelega= tor} - =C2=A0Predicting the value, [average]: 28.8 , [gradient]: 0.0 , [sec= ond derivative]: 0.0 , [time intervals]: 1=C2=A0
TID: [0] [STRATO= S] [2014-10-25 22:10:43,717] DEBUG {org.apache.stratos.autoscaler.rule.Rule= TasksDelegator} - =C2=A0Predicting the value, [average]: 39.0 , [gradient]:= -11000.0 , [second derivative]: 2000000.0 , [time intervals]: 1=C2=A0
TID: [0] [STRATOS] [2014-10-25 22:10:43,717] =C2=A0INFO {org.apache.s= tratos.autoscaler.rule.RuleLog} - =C2=A0[scaling] =C2=A0[cluster] : php.php= .domain [Replicas] minReplicas : 3
TID: [0] [STRATOS] [2014-10-25= 22:10:43,717] =C2=A0INFO {org.apache.stratos.autoscaler.rule.RuleLog} - = =C2=A0[scaling] =C2=A0[cluster] : php.php.domain [Replicas] maxReplicas : 1= 0
TID: [0] [STRATOS] [2014-10-25 22:10:43,718] =C2=A0INFO {org.ap= ache.stratos.autoscaler.rule.RuleLog} - =C2=A0[scaling] =C2=A0[cluster] : p= hp.php.domain [Replicas] nonTerminated : 3
TID: [0] [STRATOS] [20= 14-10-25 22:10:43,718] =C2=A0INFO {org.apache.stratos.autoscaler.rule.RuleL= og} - =C2=A0[scaling] =C2=A0[cluster] : php.php.domain [Replicas] activeRep= licas : 3
TID: [0] [STRATOS] [2014-10-25 22:10:43,718] =C2=A0INFO= {org.apache.stratos.autoscaler.rule.RuleLog} - =C2=A0[scaling] =C2=A0[clus= ter] : php.php.domain [RequestInFlight] predicted value : 0.0
TID= : [0] [STRATOS] [2014-10-25 22:10:43,718] =C2=A0INFO {org.apache.stratos.au= toscaler.rule.RuleLog} - =C2=A0[scaling] =C2=A0[cluster] : php.php.domain [= RequestInFlight] upper limit : 80.0
TID: [0] [STRATOS] [2014-10-2= 5 22:10:43,718] =C2=A0INFO {org.apache.stratos.autoscaler.rule.RuleLog} - = =C2=A0[scaling] =C2=A0[cluster] : php.php.domain [RequestInFlight] lower li= mit : 5.0
TID: [0] [STRATOS] [2014-10-25 22:10:43,718] =C2=A0INFO= {org.apache.stratos.autoscaler.rule.RuleLog} - =C2=A0[scaling] =C2=A0[clus= ter] : php.php.domain [MemoryConsumption] predicted value : 28.799999237060= 547
TID: [0] [STRATOS] [2014-10-25 22:10:43,718] =C2=A0INFO {org.= apache.stratos.autoscaler.rule.RuleLog} - =C2=A0[scaling] =C2=A0[cluster] := php.php.domain [MemoryConsumption] upper limit : 80.0
TID: [0] [= STRATOS] [2014-10-25 22:10:43,718] =C2=A0INFO {org.apache.stratos.autoscale= r.rule.RuleLog} - =C2=A0[scaling] =C2=A0[cluster] : php.php.domain [MemoryC= onsumption] lower limit : 15.0
TID: [0] [STRATOS] [2014-10-25 22:= 10:43,718] =C2=A0INFO {org.apache.stratos.autoscaler.rule.RuleLog} - =C2=A0= [scaling] =C2=A0[cluster] : php.php.domain [LoadAverage] predicted value : = 989039.0
TID: [0] [STRATOS] [2014-10-25 22:10:43,719] =C2=A0INFO = {org.apache.stratos.autoscaler.rule.RuleLog} - =C2=A0[scaling] =C2=A0[clust= er] : php.php.domain [LoadAverage] upper limit : 180.0
TID: [0] [= STRATOS] [2014-10-25 22:10:43,719] =C2=A0INFO {org.apache.stratos.autoscale= r.rule.RuleLog} - =C2=A0[scaling] =C2=A0[cluster] : php.php.domain [LoadAve= rage] lower limit : 20.0
TID: [0] [STRATOS] [2014-10-25 22:10:43,= 719] =C2=A0INFO {org.apache.stratos.autoscaler.rule.RuleLog} - =C2=A0[scali= ng] =C2=A0[cluster] : php.php.domain scale-up action : true
TID: = [0] [STRATOS] [2014-10-25 22:10:43,719] =C2=A0INFO {org.apache.stratos.auto= scaler.rule.RuleLog} - =C2=A0[scaling] =C2=A0[cluster] : php.php.domain sca= le-down action : false
TID: [0] [STRATOS] [2014-10-25 22:10:43,71= 9] =C2=A0INFO {org.apache.stratos.autoscaler.rule.RuleLog} - =C2=A0[scaling= ] =C2=A0[cluster] : php.php.domain [LoadAverage] predicted replicas : 16484=
TID: [0] [STRATOS] [2014-10-25 22:10:43,720] =C2=A0INFO {org.apa= che.stratos.autoscaler.rule.RuleLog} - =C2=A0[scaling] =C2=A0[cluster] : ph= p.php.domain predicted replicas > max replicas :=C2=A0
TID: [0= ] [STRATOS] [2014-10-25 22:10:43,720] =C2=A0INFO {org.apache.stratos.autosc= aler.rule.RuleLog} - =C2=A0[scaling] Decided to scale-up : [cluster] : php.= php.domain
TID: [0] [STRATOS] [2014-10-25 22:10:43,720] =C2=A0INF= O {org.apache.stratos.autoscaler.rule.RuleLog} - =C2=A0[scaling-up] =C2=A0[= cluster] : php.php.domain valid number of replicas to expand : 10
TID: [0] [STRATOS] [2014-10-25 22:10:43,720] =C2=A0INFO {org.apache.strato= s.autoscaler.client.cloud.controller.CloudControllerClient} - =C2=A0Updatin= g kubernetes replication controller via cloud controller: [cluster] php.php= .domain [replicas] 10
TID: [0] [STRATOS] [2014-10-25 22:10:43,734= ] DEBUG {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImp= l} - =C2=A0CloudControllerServiceImpl:updateContainers for cluster : php.ph= p.domain
TID: [0] [STRATOS] [2014-10-25 22:10:43,758] DEBUG {org.= apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} - =C2=A0Cl= oud Controller is delegating request to update a replication controller php= .php.domain to Kubernetes layer.


I = couldn't investigate the agent side, since containers are destroyed.

Any idea?

Thanks.

--
Rajkum= ar Rajaratnam
Software Engineer | WSO2, Inc.




--
Regards,
Manula Chathurika Thantriwatte
Soft= ware Engineer
WSO2 Inc. : http://wso2.com
lean . enterprise . middle= ware

<= div>email : manula= c@wso2.com / man= ula@apache.org
<= font color=3D"#888888">phone : +94 772492511



--001a113cdcaa280e1005064a8321--