Return-Path: X-Original-To: apmail-stratos-dev-archive@minotaur.apache.org Delivered-To: apmail-stratos-dev-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id B16261774C for ; Thu, 6 Nov 2014 08:21:01 +0000 (UTC) Received: (qmail 31200 invoked by uid 500); 6 Nov 2014 08:21:01 -0000 Delivered-To: apmail-stratos-dev-archive@stratos.apache.org Received: (qmail 31137 invoked by uid 500); 6 Nov 2014 08:21:01 -0000 Mailing-List: contact dev-help@stratos.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@stratos.apache.org Delivered-To: mailing list dev@stratos.apache.org Received: (qmail 31124 invoked by uid 99); 6 Nov 2014 08:21:00 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 Nov 2014 08:21:00 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of isuruh@wso2.com designates 209.85.217.172 as permitted sender) Received: from [209.85.217.172] (HELO mail-lb0-f172.google.com) (209.85.217.172) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 Nov 2014 08:20:54 +0000 Received: by mail-lb0-f172.google.com with SMTP id w7so462486lbi.31 for ; Thu, 06 Nov 2014 00:19:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=wso2.com; s=google; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=R4wAX4r0enjfnERdyxwroT7qcAf8ZbwWiCfcaeR3ms0=; b=egjTg7RGh9l/BRyuu2j0Zks0Qh6lx0e+odv1Dijzgjj1XB2/mXw/LFuMlSqF8np8zT bLLh00ZOMeHuo9xfsxcEuuJMCkRWx9AqT0NGq12BbZV/QBNdoOOTQ5RvKTNgUwmkRoEI 1zBVYQvTnhnW/UvXBNZP1RdkFd9NeLIvCNync= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=R4wAX4r0enjfnERdyxwroT7qcAf8ZbwWiCfcaeR3ms0=; b=LOD2zAIo1Bnbdz9mP+cXkSQI6wI5OSGEiFaWYWG7OEaKa/VgFATHvE7bilV7ORbuYz WuCvJBzYEItnaFKuMrUyo/v3VU+ibZYgcJcyFWNrL8KO+PtMhqaJiA9pdxg+BQ+OTHkp sODoXfAXZNR/e/feEJJ2N4/oImzzpSC2/HA7B/ld9IwUQ3eoRqeaYqyOo3pV0CtBO36m 8fRqV2Obc0rxZO8iNgyAHyzbepZrV5DEtiCYyDVKWFYKxFTE0HSnvO2C9lShfcaTwct9 HKhiuImi0ocEUvbRIuiPk2vHTkYsIa3DnihG+3Oin0yBjD+qOLRAhjRxuP0IN3wpOyK1 fOUA== X-Gm-Message-State: ALoCoQl+TWBQOATZ/0MruOPkNMlRSvn+Y2G9NoTmkb7L+oD+CAnsvzyrIAbYnryCYPrTY1W/nQJm MIME-Version: 1.0 X-Received: by 10.152.45.1 with SMTP id i1mr3195501lam.7.1415261985540; Thu, 06 Nov 2014 00:19:45 -0800 (PST) Sender: isuruh@wso2.com Received: by 10.112.223.44 with HTTP; Thu, 6 Nov 2014 00:19:45 -0800 (PST) In-Reply-To: References: Date: Thu, 6 Nov 2014 13:49:45 +0530 X-Google-Sender-Auth: -h2lz-I2xX3E1XtWE69SfU5tVMk Message-ID: Subject: Re: [Fixed] CEP sends very large values for gradient and second derivative of load average From: Isuru Haththotuwa To: dev Cc: Lakmal Warusawithana , Chamila De Alwis , Rajkumar Rajaratnam Content-Type: multipart/alternative; boundary=001a11c27c84e9aa6305072c5bf3 X-Virus-Checked: Checked by ClamAV on apache.org --001a11c27c84e9aa6305072c5bf3 Content-Type: text/plain; charset=UTF-8 Thanks a lot Nirmal for the detailed explanation. It makes sense for me, as an effort to decrease the risk of false alarms, such as triggering scale up, due to input data (in this case, events) not being sufficient to make a correct decision. On Thu, Nov 6, 2014 at 11:38 AM, Imesh Gunaratne wrote: > On Thu, Nov 6, 2014 at 9:22 AM, Rajkumar Rajaratnam > wrote: > >> Hi, >> >> No Imesh. There are no difference between *taking the the difference and >> dividing it by 1000* and *dividing it by 1000 and taking the difference* >> . >> > > Raj: I agree, in this scenario it makes no difference, however when we are > applying a fomular the unit conversion should happen prior to applying > values. > > Still I see a problem with this correction, take the same example: > Gradient: -999.9999999999998 Last val: 7.000000000000001 First val: 12.0 > Time Gap: 5 t1: 1415213232152 t2: 1415213232157 > > *Manual calulation:* > The value difference: 7 - 12 = -5 > Time difference: 1415213232157 - 1415213232152 = 5 ms => 5/1000 = 0.005 sec > *Gradient = - 5/0.005 = -1000* > > *According to the current fix:* > long millisecondsForASecond = 1000; > long tGap = t2 - t1 > millisecondsForASecond ? t2 - t1 : > millisecondsForASecond; > double gradient = 0.0; > if (tGap > 0) { > gradient = ((lastVal - firstVal) * millisecondsForASecond) / > tGap; > } > > t1: 1415213232152 t2: 1415213232157 firstVal: 12 lastVall: 7 > Gradient = -5 * 1000 / 1000 = - 5 > > Nirmal: Can you please justify this? > > Thanks > AFAIU, we cannot compare these two calculations. The manual calculation is the standard way of calculating the gradient, which would give correct answers based on how correct the input values are. It can also give false values, when in short term there seem to be a couple spikes spanning across a small time gap, but if you take a larger time frame, there is no much difference. That it why you are getting the gradient as a large value in the manual calculation. AFAIU Nirmal's fix attempts to address this problem. In the real scenario, we do not want the system to scale up if there are two spikes very close together, but in the larger time window the values are normal. AFAIU, this is what the proposed fix covers. Of course, it might not be a perfect solution, but AFAIU this is will do for now. > > On Thu, Nov 6, 2014 at 11:31 AM, Nirmal Fernando > wrote: > >> >> >> On Thu, Nov 6, 2014 at 6:05 AM, Gayan Gunarathne wrote: >> >>> AFAIU this is not related with the gradient calculation formula that we >>> used. But It is related with the how we calculate gradient when the >>> denominator become a small number.IMO when the divisor (timeInSeconds) >>> approaches to the small number we do need to normalize the result. >>> >>> IMO we do have max and min scale up and down values. So based on that we >>> can calculate the gradient on that value range.Simply if the denominator >>> approaches zero result approaches to infinity. >>> >>> I think that's what Nirmal doing here by making the lower threshold as >>> "millisecondsForASecond" >>> >> >> You got it! thanks. >> >>> >>> >>> long tGap = t2 - t1 > millisecondsForASecond ? t2 - t1 : >>> millisecondsForASecond; >>> double gradient = 0.0; >>> if (tGap > 0) { >>> gradient = ((lastVal - firstVal) * millisecondsForASecond) / >>> tGap; >>> } >>> >>> >>> Thanks, >>> Gayan >>> >>> On Thu, Nov 6, 2014 at 9:22 AM, Rajkumar Rajaratnam >>> wrote: >>> >>>> Hi, >>>> >>>> On Thu, Nov 6, 2014 at 9:03 AM, Imesh Gunaratne >>>> wrote: >>>> >>>>> Hi Nirmal, >>>>> >>>>> I do not think the milliseconds to seconds convertion is correct here. >>>>> >>>>> As I see we are taking the difference of two timestamp values and then >>>>> dividing it by 1000. The corect way might be to first divide each value by >>>>> 1000 and then take the difference. >>>>> >>>> >>>> No Imesh. There are no difference between *taking the the difference >>>> and dividing it by 1000* and *dividing it by 1000 and taking the >>>> difference*. >>>> >>>> >>>>> On the other hand we might not need to convert these values to seconds >>>>> since we are taking a time difference and calculating a gradient. >>>>> >>>>> I did a quick test with the following sample: >>>>> Gradient: -999.9999999999998 Last val: 7.000000000000001 First val: >>>>> 12.0 Time Gap: 5 t1: 1415213232152 t2: 1415213232157 >>>>> >>>>> *According previous code:* >>>>> long tGap = t2 - t1; >>>>> double gradient = 0.0; >>>>> if (tGap > 0) { >>>>> gradient = ((lastVal - firstVal) * 1000) / tGap; >>>>> } >>>>> >>>>> t1: 1415213232152 t2: 1415213232157 firstVal: 12 lastVall: 7 >>>>> gradient: -1000.0 >>>>> >>>>> *According to your fix:* >>>>> long millisecondsForASecond = 1000; >>>>> long tGap = t2 - t1 > millisecondsForASecond ? t2 - t1 : >>>>> millisecondsForASecond; >>>>> double gradient = 0.0; >>>>> if (tGap > 0) { >>>>> gradient = ((lastVal - firstVal) * millisecondsForASecond) >>>>> / tGap; >>>>> } >>>>> >>>>> t1: 1415213232152 t2: 1415213232157 firstVal: 12 lastVall: 7 >>>>> gradient: -5.0 >>>>> >>>>> *According to an online gradient calculator:* >>>>> gradient: -1 >>>>> >>>>> http://www.calculator.net/slope-calculator.html?type=1&x11=1415213232152&y11=12&x12=1415213232157&y12=7&x=27&y=19 >>>>> >>>>> According to the online gradient calculator (assuming their >>>>> calculation is correct), the calculation in your fix is not correct. I >>>>> believe the logic should be simple as follows: >>>>> >>>>> long tGap = t2 - t1; >>>>> double gradient = 0.0; >>>>> if (tGap > 0) { >>>>> gradient = ((lastVal - firstVal)) / tGap; >>>>> } >>>>> >>>>> t1: 1415213232152 t2: 1415213232157 firstVal: 12 lastVall: 7 >>>>> gradient: -1.0 >>>>> >>>> >>>> And AFAIK, standard way to calculate gradient = (v1-v2)/timeInSeconds. >>>> >>>> Well, we can divide it by what ever time units we wants, but the >>>> result's unit will be different. >>>> >>>> In Nirmal's case, he get will get *stats/seconds*. In your case, you >>>> will get *stats/milliseconds* >>>> >>>> And AS prediction is based on *seconds*, CEP should use *seconds* for >>>> the calculation. >>>> >>>> wdyt? >>>> >>>> Thanks. >>>> >>>> >>>>> Thanks >>>>> >>>>> >>>>> On Thu, Nov 6, 2014 at 1:32 AM, Nirmal Fernando < >>>>> nirmal070125@gmail.com> wrote: >>>>> >>>>>> Hi Guys, >>>>>> >>>>>> So, I got a chance to reproduce and analyze this issue. >>>>>> >>>>>> How we calculate gradient of two events? >>>>>> >>>>>> Say the events are; e1(t1,v1) and e2(t2,v2) >>>>>> >>>>>> tx - time in *milliseconds* when xth event occurred >>>>>> vx - value (memory, cpu etc.) that xth event carries >>>>>> >>>>>> *time gap = t(2-1) = t2 -t1 milliseconds* >>>>>> time gap in seconds = t(2-1) = (t2 - t1)/1000 seconds >>>>>> >>>>>> Hence, >>>>>> *Gradient = (v2 - v1) / t(2-1) = ( (v2 - v1) * 1000 ) / (t2 - t1)* >>>>>> >>>>>> I've enabled debug logs for CEP extension; >>>>>> >>>>>> log4j.logger.org.apache.stratos.cep.extension=DEBUG >>>>>> >>>>>> Please find the following 3 logs extracted from the debug logs; >>>>>> >>>>>> =================================================================== >>>>>> TID: [0] [STRATOS] [2014-11-05 19:47:27,073] DEBUG >>>>>> {org.apache.stratos.cep.extension.SecondDerivativeFinderWindowProcessor} - >>>>>> Gradient: -0.1996007984031936 Last val: 9.0 First val: 12.0 *Time >>>>>> Gap: 15030* t1: 1415213202095 t2: 1415213217125 hash: 155426542 >>>>>> >>>>>> TID: [0] [STRATOS] [2014-11-05 19:47:27,073] DEBUG >>>>>> {org.apache.stratos.cep.extension.SecondDerivativeFinderWindowProcessor} - Gradient: >>>>>> -999.9999999999998 Last val: 7.000000000000001 First val: 12.0 *Time >>>>>> Gap: 5* t1: 1415213232152 t2: 1415213232157 hash: 155426542 >>>>>> >>>>>> TID: [0] [STRATOS] [2014-11-05 19:47:27,074] DEBUG >>>>>> {org.apache.stratos.cep.extension.SecondDerivativeFinderWindowProcessor} - >>>>>> Gradient: -44.34884666437174 Last val: -999.9999999999998 First val: >>>>>> -0.1996007984031936 *Time Gap: 22544* t1: 1415213209610 t2: >>>>>> 1415213232154 hash: 155426542 >>>>>> =================================================================== >>>>>> >>>>>> So, as you can see the reason behind a large value is when the time >>>>>> gap between two subjected events is less than 1 second. This could happen >>>>>> since events are coming from different asynchronous agents and also when >>>>>> there are less number of events. >>>>>> >>>>>> So, the fix I propose is a very simple one and it will not compromise >>>>>> anything AFAIS. >>>>>> >>>>>> Fix is to calculate time gap as follows; >>>>>> *____ t2 - >>>>>> t1* >>>>>> *| yes?* >>>>>> *time gap = t(2-1) = t2 -t1 > 1000 ---* >>>>>> *|____ 1000* >>>>>> >>>>>> I have tested this and works fine. >>>>>> >>>>>> >>>>>> >>>>>> ---------- Forwarded message ---------- >>>>>> From: Manula Chathurika Thantriwatte >>>>>> Date: Tue, Oct 28, 2014 at 5:16 AM >>>>>> Subject: Re: CEP sends very large values for gradient and second >>>>>> derivative of load average >>>>>> To: "dev@stratos.apache.org" >>>>>> >>>>>> >>>>>> Hi Raj, >>>>>> >>>>>> Yes it has. But it's very easy to find the values from the agent with >>>>>> the time stamp. Then we can search that time stamp in the CEP trace logs >>>>>> and compare the summarized values. >>>>>> >>>>>> Thanks ! >>>>>> >>>>>> On Mon, Oct 27, 2014 at 9:23 PM, Rajkumar Rajaratnam < >>>>>> rajkumarr@wso2.com> wrote: >>>>>> >>>>>>> >>>>>>> >>>>>>> On Mon, Oct 27, 2014 at 7:56 PM, Manula Chathurika Thantriwatte < >>>>>>> manulac@wso2.com> wrote: >>>>>>> >>>>>>>> Hi Raj, >>>>>>>> >>>>>>>> Shall we enable the cartridge agent debug logs. Then we can compare >>>>>>>> both CEP and agent logs with the time stamp. Then we can narrow down where >>>>>>>> the actual problem is. >>>>>>>> >>>>>>> >>>>>>> Manula, cep-trace log have the values sent by cartridge agent also >>>>>>> right? >>>>>>> >>>>>>> >>>>>>>> >>>>>>>> Thanks ! >>>>>>>> >>>>>>>> On Mon, Oct 27, 2014 at 5:06 PM, Rajkumar Rajaratnam < >>>>>>>> rajkumarr@wso2.com> wrote: >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Mon, Oct 27, 2014 at 12:53 PM, Manula Chathurika Thantriwatte < >>>>>>>>> manulac@wso2.com> wrote: >>>>>>>>> >>>>>>>>>> Hi Raj, >>>>>>>>>> >>>>>>>>>> Is this happen frequently or intermittently ? Once I go though >>>>>>>>>> the logs I'm unable to find the high input values for load average. But >>>>>>>>>> output have high values. >>>>>>>>>> >>>>>>>>>> 22:10:38,698 [-] [Siddhi-Scheduler-pool-10-thread-1] INFO >>>>>>>>>> EVENT_TRACE_LOGGER TenantId=-1234 : Output Event Adaptor : >>>>>>>>>> JMSOutputAdaptor, sent >>>>>>>>>> >>>>>>>>>> {"org.apache.stratos.messaging.event.health.stat.SecondDerivativeOfLoadAverageEvent":{"message":{"clusterId":"php.php.domain","networkPartitionId":"","value":"2000000.0"}}} >>>>>>>>>> >>>>>>>>> >>>>>>>>> This is happening intermittently. Experienced now too. This is a >>>>>>>>> serious issue. We need to somehow sort this out. >>>>>>>>> >>>>>>>>> >>>>>>>>>> >>>>>>>>>> Also networkPartitionId is empty. In the current execution plans >>>>>>>>>> it took networkPartitionId for the calculation. IMO we need to update the >>>>>>>>>> CEP execution plans for 4.1.0. >>>>>>>>>> >>>>>>>>>> Thanks ! >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Sun, Oct 26, 2014 at 8:49 AM, Manula Chathurika Thantriwatte < >>>>>>>>>> manulac@wso2.com> wrote: >>>>>>>>>> >>>>>>>>>>> Hi, >>>>>>>>>>> >>>>>>>>>>> Thanks Raj. I'll go through them. >>>>>>>>>>> >>>>>>>>>>> Thanks ! >>>>>>>>>>> >>>>>>>>>>> On Sun, Oct 26, 2014 at 8:33 AM, Rajkumar Rajaratnam < >>>>>>>>>>> rajkumarr@wso2.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi Manula, >>>>>>>>>>>> >>>>>>>>>>>> You can see those values in the same log I attached in the >>>>>>>>>>>> previous mail. >>>>>>>>>>>> >>>>>>>>>>>> Thanks. >>>>>>>>>>>> >>>>>>>>>>>> On Sun, Oct 26, 2014 at 8:29 AM, Rajkumar Rajaratnam < >>>>>>>>>>>> rajkumarr@wso2.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hi Chamila, >>>>>>>>>>>>> >>>>>>>>>>>>> I couldn't find any large values sent by agent. There are >>>>>>>>>>>>> traces for CEP sending large values. Find the cep trace log here [1] >>>>>>>>>>>>> >>>>>>>>>>>>> 1. >>>>>>>>>>>>> https://drive.google.com/file/d/0B1haIleqJMHIS3FFVFpfa0JlbVE/view?usp=sharing >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks. >>>>>>>>>>>>> >>>>>>>>>>>>> On Sun, Oct 26, 2014 at 8:23 AM, Manula Chathurika >>>>>>>>>>>>> Thantriwatte wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>> >>>>>>>>>>>>>> Shall we enable the cartridge agent debug logs and see what >>>>>>>>>>>>>> are the load average values sent from the agent. If those values are normal >>>>>>>>>>>>>> then we can narrow down to the CEP. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks ! >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Sat, Oct 25, 2014 at 10:43 PM, Chamila De Alwis < >>>>>>>>>>>>>> chamilad@wso2.com> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi Raj, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Can you inspect the CEP trace logs for the same time period? >>>>>>>>>>>>>>> It will have the events published from the agent. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>>> Chamila de Alwis >>>>>>>>>>>>>>> Software Engineer | WSO2 | +94772207163 >>>>>>>>>>>>>>> Blog: code.chamiladealwis.com >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Sat, Oct 25, 2014 at 10:39 PM, Rajkumar Rajaratnam < >>>>>>>>>>>>>>> rajkumarr@wso2.com> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I am doing a test round with M3 with python agent. And >>>>>>>>>>>>>>>> faced this very strange scenario, $Subject. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,693] DEBUG >>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average >>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1 >>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,693] DEBUG >>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of >>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] -2000.0 >>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG >>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average >>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1 >>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG >>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of >>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 0.0 >>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG >>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average >>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1 >>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG >>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg >>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 1.9090909 >>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG >>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average >>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1 >>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG >>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg >>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.718182 >>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG >>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second >>>>>>>>>>>>>>>> Derivation of Memory Consumption event: [cluster] php.php.domain [value] >>>>>>>>>>>>>>>> 200000.0 >>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG >>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of >>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 0.0 >>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG >>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average >>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1 >>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG >>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second >>>>>>>>>>>>>>>> Derivation of load avg event: [cluster] php.php.domain [value] 0.0 >>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG >>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average >>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1 >>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG >>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second >>>>>>>>>>>>>>>> Derivation of load avg event: [cluster] php.php.domain [value] 0.0 >>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG >>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average >>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1 >>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG >>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg >>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 3.3636363 >>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG >>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average >>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1 >>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG >>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of >>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.11670045 >>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG >>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory >>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1 >>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG >>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg >>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7 >>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG >>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory >>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1 >>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG >>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second >>>>>>>>>>>>>>>> Derivation of Memory Consumption event: [cluster] php.php.domain [value] 0.0 >>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG >>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory >>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1 >>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG >>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg >>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7 >>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG >>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory >>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1 >>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG >>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg >>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 5.0 >>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG >>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average >>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1 >>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG >>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of >>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 2000.0 >>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG >>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average >>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1 >>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG >>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg >>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.7 >>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG >>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory >>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1 >>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG >>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of >>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0 >>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG >>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory >>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1 >>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,699] DEBUG >>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second >>>>>>>>>>>>>>>> Derivation of Memory Consumption event: [cluster] php.php.domain [value] 0.0 >>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,699] DEBUG >>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory >>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1 >>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG >>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Second >>>>>>>>>>>>>>>> Derivation of load avg event: [cluster] php.php.domain [value] 2000000.0 >>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG >>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average >>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1 >>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG >>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of >>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 13500.0 >>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG >>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average >>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1 >>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG >>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of >>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 50.0 >>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG >>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory >>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1 >>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG >>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg >>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.738462 >>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG >>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory >>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1 >>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG >>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg >>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 2.3333333 >>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG >>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average >>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1 >>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG >>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of >>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 0.0 >>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG >>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average >>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1 >>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG >>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of >>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0 >>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG >>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory >>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1 >>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG >>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg >>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 31.0 >>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG >>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average >>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1 >>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG >>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of >>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0 >>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG >>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory >>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1 >>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG >>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg >>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 39.0 >>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG >>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average >>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1 >>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG >>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg >>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] 39.0 >>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,705] DEBUG >>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average >>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1 >>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,706] DEBUG >>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of >>>>>>>>>>>>>>>> load avg event: [cluster] php.php.domain [value] -11000.0 >>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,706] DEBUG >>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average >>>>>>>>>>>>>>>> stats are reset, ready to do scale check [kub cluster] KubGrp1 >>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,708] DEBUG >>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad of >>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 0.0 >>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,708] DEBUG >>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory >>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1 >>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,709] DEBUG >>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg >>>>>>>>>>>>>>>> Memory Consumption event: [cluster] php.php.domain [value] 28.8 >>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,709] DEBUG >>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory >>>>>>>>>>>>>>>> consumption stats are reset, ready to do scale check [kub cluster] KubGrp1 >>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,811] DEBUG >>>>>>>>>>>>>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} - >>>>>>>>>>>>>>>> PodActivationWatcher running : Running >>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,942] DEBUG >>>>>>>>>>>>>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} - >>>>>>>>>>>>>>>> PodActivationWatcher running : Running >>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:38,945] DEBUG >>>>>>>>>>>>>>>> {org.apache.stratos.cloud.controller.util.PodActivationWatcher} - >>>>>>>>>>>>>>>> PodActivationWatcher running : Running >>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,712] DEBUG >>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} - >>>>>>>>>>>>>>>> KubernetesServiceClusterMonitor is running.. >>>>>>>>>>>>>>>> KubernetesServiceClusterMonitor [ kubernetesHostClusterId=KubGrp1, >>>>>>>>>>>>>>>> clusterId=php.php.domain, serviceId=php] >>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,713] DEBUG >>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} - >>>>>>>>>>>>>>>> Running min check for [kub-cluster] : KubGrp1 [cluster] : php.php.domain >>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,713] INFO >>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running minimum rule: >>>>>>>>>>>>>>>> [kub-cluster] KubGrp1 [cluster] php.php.domain >>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,714] INFO >>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] [cluster] : >>>>>>>>>>>>>>>> php.php.domain [Replicas] nonTerminated : 3 >>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,714] INFO >>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] [cluster] : >>>>>>>>>>>>>>>> php.php.domain [Replicas] minReplicas : 3 >>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] INFO >>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running obsolete containers >>>>>>>>>>>>>>>> rule [kub-cluster] : KubGrp1 [cluster] : php.php.domain >>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] INFO >>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [obsolete-check] [cluster] >>>>>>>>>>>>>>>> : php.php.domain [Replicas] obsoleteReplicas : 0 >>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG >>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.AutoscalerRuleEvaluator} - Minimum >>>>>>>>>>>>>>>> check executed for : >>>>>>>>>>>>>>>> org.apache.stratos.autoscaler.KubernetesClusterContext@21d906d0 >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG >>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} - >>>>>>>>>>>>>>>> flag of rifReset : true flag of memoryConsumptionReset : true flag of >>>>>>>>>>>>>>>> loadAverageReset : true >>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG >>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} - >>>>>>>>>>>>>>>> Running scale check for [kub-cluster] : KubGrp1 [cluster] : php.php.domain >>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] INFO >>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - Running scaling rule >>>>>>>>>>>>>>>> [kub-cluster] : KubGrp1 [cluster] : php.php.domain >>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] DEBUG >>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the >>>>>>>>>>>>>>>> value, [average]: 0.0 , [gradient]: 0.0 , [second derivative]: 0.0 , [time >>>>>>>>>>>>>>>> intervals]: 1 >>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,716] DEBUG >>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the >>>>>>>>>>>>>>>> value, [average]: 28.8 , [gradient]: 0.0 , [second derivative]: 0.0 , [time >>>>>>>>>>>>>>>> intervals]: 1 >>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] DEBUG >>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the >>>>>>>>>>>>>>>> value, [average]: 39.0 , [gradient]: -11000.0 , [second derivative]: >>>>>>>>>>>>>>>> 2000000.0 , [time intervals]: 1 >>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] INFO >>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : >>>>>>>>>>>>>>>> php.php.domain [Replicas] minReplicas : 3 >>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,717] INFO >>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : >>>>>>>>>>>>>>>> php.php.domain [Replicas] maxReplicas : 10 >>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO >>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : >>>>>>>>>>>>>>>> php.php.domain [Replicas] nonTerminated : 3 >>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO >>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : >>>>>>>>>>>>>>>> php.php.domain [Replicas] activeReplicas : 3 >>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO >>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : >>>>>>>>>>>>>>>> php.php.domain [RequestInFlight] predicted value : 0.0 >>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO >>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : >>>>>>>>>>>>>>>> php.php.domain [RequestInFlight] upper limit : 80.0 >>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO >>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : >>>>>>>>>>>>>>>> php.php.domain [RequestInFlight] lower limit : 5.0 >>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO >>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : >>>>>>>>>>>>>>>> php.php.domain [MemoryConsumption] predicted value : 28.799999237060547 >>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO >>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : >>>>>>>>>>>>>>>> php.php.domain [MemoryConsumption] upper limit : 80.0 >>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO >>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : >>>>>>>>>>>>>>>> php.php.domain [MemoryConsumption] lower limit : 15.0 >>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,718] INFO >>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : >>>>>>>>>>>>>>>> php.php.domain [LoadAverage] predicted value : 989039.0 >>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO >>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : >>>>>>>>>>>>>>>> php.php.domain [LoadAverage] upper limit : 180.0 >>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO >>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : >>>>>>>>>>>>>>>> php.php.domain [LoadAverage] lower limit : 20.0 >>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO >>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : >>>>>>>>>>>>>>>> php.php.domain scale-up action : true >>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO >>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : >>>>>>>>>>>>>>>> php.php.domain scale-down action : false >>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,719] INFO >>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : >>>>>>>>>>>>>>>> php.php.domain [LoadAverage] predicted replicas : 16484 >>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO >>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : >>>>>>>>>>>>>>>> php.php.domain predicted replicas > max replicas : >>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO >>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] Decided to >>>>>>>>>>>>>>>> scale-up : [cluster] : php.php.domain >>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO >>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling-up] [cluster] : >>>>>>>>>>>>>>>> php.php.domain valid number of replicas to expand : 10 >>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,720] INFO >>>>>>>>>>>>>>>> {org.apache.stratos.autoscaler.client.cloud.controller.CloudControllerClient} >>>>>>>>>>>>>>>> - Updating kubernetes replication controller via cloud controller: >>>>>>>>>>>>>>>> [cluster] php.php.domain [replicas] 10 >>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,734] DEBUG >>>>>>>>>>>>>>>> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} - >>>>>>>>>>>>>>>> CloudControllerServiceImpl:updateContainers for cluster : php.php.domain >>>>>>>>>>>>>>>> TID: [0] [STRATOS] [2014-10-25 22:10:43,758] DEBUG >>>>>>>>>>>>>>>> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} - >>>>>>>>>>>>>>>> Cloud Controller is delegating request to update a replication controller >>>>>>>>>>>>>>>> php.php.domain to Kubernetes layer. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I couldn't investigate the agent side, since containers are >>>>>>>>>>>>>>>> destroyed. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Any idea? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thanks. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>> Rajkumar Rajaratnam >>>>>>>>>>>>>>>> Software Engineer | WSO2, Inc. >>>>>>>>>>>>>>>> Mobile +94777568639 | +94783498120 >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>> Manula Chathurika Thantriwatte >>>>>>>>>>>>>> Software Engineer >>>>>>>>>>>>>> WSO2 Inc. : http://wso2.com >>>>>>>>>>>>>> lean . enterprise . middleware >>>>>>>>>>>>>> >>>>>>>>>>>>>> email : manulac@wso2.com / manula@apache.org >>>>>>>>>>>>>> phone : +94 772492511 >>>>>>>>>>>>>> blog : http://manulachathurika.blogspot.com/ >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> Rajkumar Rajaratnam >>>>>>>>>>>>> Software Engineer | WSO2, Inc. >>>>>>>>>>>>> Mobile +94777568639 | +94783498120 >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> Rajkumar Rajaratnam >>>>>>>>>>>> Software Engineer | WSO2, Inc. >>>>>>>>>>>> Mobile +94777568639 | +94783498120 >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> Regards, >>>>>>>>>>> Manula Chathurika Thantriwatte >>>>>>>>>>> Software Engineer >>>>>>>>>>> WSO2 Inc. : http://wso2.com >>>>>>>>>>> lean . enterprise . middleware >>>>>>>>>>> >>>>>>>>>>> email : manulac@wso2.com / manula@apache.org >>>>>>>>>>> phone : +94 772492511 >>>>>>>>>>> blog : http://manulachathurika.blogspot.com/ >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Regards, >>>>>>>>>> Manula Chathurika Thantriwatte >>>>>>>>>> Software Engineer >>>>>>>>>> WSO2 Inc. : http://wso2.com >>>>>>>>>> lean . enterprise . middleware >>>>>>>>>> >>>>>>>>>> email : manulac@wso2.com / manula@apache.org >>>>>>>>>> phone : +94 772492511 >>>>>>>>>> blog : http://manulachathurika.blogspot.com/ >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Rajkumar Rajaratnam >>>>>>>>> Software Engineer | WSO2, Inc. >>>>>>>>> Mobile +94777568639 | +94783498120 >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Regards, >>>>>>>> Manula Chathurika Thantriwatte >>>>>>>> Software Engineer >>>>>>>> WSO2 Inc. : http://wso2.com >>>>>>>> lean . enterprise . middleware >>>>>>>> >>>>>>>> email : manulac@wso2.com / manula@apache.org >>>>>>>> phone : +94 772492511 >>>>>>>> blog : http://manulachathurika.blogspot.com/ >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Rajkumar Rajaratnam >>>>>>> Software Engineer | WSO2, Inc. >>>>>>> Mobile +94777568639 | +94783498120 >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Regards, >>>>>> Manula Chathurika Thantriwatte >>>>>> Software Engineer >>>>>> WSO2 Inc. : http://wso2.com >>>>>> lean . enterprise . middleware >>>>>> >>>>>> email : manulac@wso2.com / manula@apache.org >>>>>> phone : +94 772492511 >>>>>> blog : http://manulachathurika.blogspot.com/ >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Best Regards, >>>>>> Nirmal >>>>>> >>>>>> Nirmal Fernando. >>>>>> PPMC Member & Committer of Apache Stratos, >>>>>> Senior Software Engineer, WSO2 Inc. >>>>>> >>>>>> Blog: http://nirmalfdo.blogspot.com/ >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Imesh Gunaratne >>>>> >>>>> Technical Lead, WSO2 >>>>> Committer & PMC Member, Apache Stratos >>>>> >>>> >>>> >>>> >>>> -- >>>> Raj >>>> >>> >>> >>> >>> -- >>> >>> Gayan Gunarathne >>> Technical Lead >>> WSO2 Inc. (http://wso2.com) >>> email : gayang@wso2.com | mobile : +94 766819985 >>> >>> >> >> >> >> -- >> Best Regards, >> Nirmal >> >> Nirmal Fernando. >> PPMC Member & Committer of Apache Stratos, >> Senior Software Engineer, WSO2 Inc. >> >> Blog: http://nirmalfdo.blogspot.com/ >> > > > > -- > Imesh Gunaratne > > Technical Lead, WSO2 > Committer & PMC Member, Apache Stratos > > -- > Thanks and Regards, > > Isuru H. > +94 716 358 048* * > > > * * > > > --001a11c27c84e9aa6305072c5bf3 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Thanks a lot Nirmal for the detailed explanation. It makes= sense for me, as an effort to decrease the risk of false alarms, such as t= riggering scale up, due to input data=C2=A0 (in this case, events) not bein= g sufficient to make a correct decision.

On Thu, Nov 6, 2014 at 11:38 AM, Imesh Gun= aratne <imesh@apache.org> wrote:
On Thu, Nov 6, 2014 at 9:22 AM, Rajkumar Rajar= atnam=C2=A0<rajkumarr@wso2.com>=C2=A0wrote:
Hi,

No Imesh. There are no differe= nce between=C2=A0taking the the difference and dividing it by 1000= =C2=A0and=C2=A0dividing it by 1000 and taking the difference.=C2=A0<= /div>

Raj: I= agree, in this scenario it makes no difference, however when we are applyi= ng a fomular the unit conversion should happen prior to applying values.

Still I see a problem with this correction, take the= same example:
Gradient: -999.9999999999998 Last val: 7.= 000000000000001 First val: 12.0 Time Gap: 5 t1: 1415213232152 t2: 141521323= 2157

Manual calulation:
<= div>The value difference: 7 - 12 =3D -5
Time difference: 14152132= 32157 - 1415213232152 =3D 5 ms =3D> 5/1000 =3D 0.005 sec
Gr= adient =3D - 5/0.005 =3D -1000

According to= the current fix:
long millisecondsForASecond =3D= 1000;
=C2=A0 =C2=A0 =C2=A0 =C2=A0 long tGap =3D t2 - t1 > mil= lisecondsForASecond ? t2 - t1 : millisecondsForASecond;
=C2=A0 = =C2=A0 =C2=A0 =C2=A0 double gradient =3D 0.0;
=C2=A0 =C2=A0 =C2= =A0 =C2=A0 if (tGap > 0) {
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 gradient =3D ((lastVal - firstVal) * millisecondsForASecond) / tGap;=
=C2=A0 =C2=A0 =C2=A0 =C2=A0 }

t1: 1415213232152 t2: 1415213232157 firstVal: 12 lastVall: 7
Gradient =3D -5 * 1000 / 1000 =3D - 5

Ni= rmal: Can you please justify this?

Thanks
AFAIU, we cannot compare these two calcula= tions. The manual calculation is the standard way of calculating the gradie= nt, which would give correct answers based on how correct the input values = are. It can also give false values, when in short term there seem to be a c= ouple spikes spanning across a small time gap, but if you take a larger tim= e frame, there is no much difference. That it why you are getting the gradi= ent as a large value in the manual calculation.

AFAIU Ni= rmal's fix attempts to address this problem. In the real scenario, we d= o not want the system to scale up if there are two spikes very close togeth= er, but in the larger time window the values are normal. AFAIU, this is wha= t the proposed fix covers. Of course, it might not be a perfect solution, b= ut AFAIU this is will do for now.
=

On Thu,= Nov 6, 2014 at 11:31 AM, Nirmal Fernando <nirmal070125@gmail.com= > wrote:

<= div class=3D"gmail_extra">
On Thu, Nov = 6, 2014 at 6:05 AM, Gayan Gunarathne <gayang@wso2.com> wrote:<= br>
AFAIU this is not relate= d with the gradient calculation formula that we used. But It is related wit= h the how we calculate gradient when the denominator become a small number.= IMO when the divisor (timeInSeconds) approaches to the small number we do n= eed to normalize the result.

IMO we do have max and min scale up and= down values. So based on that we can calculate the gradient on that value = range.Simply if the denominator approaches zero result approaches to infini= ty.

I think that's what Nirmal doing here by making the lower th= reshold as "millisecondsForASecond"

You got it! thanks.


=C2=A0 =C2=A0 =C2=A0 =C2=A0 long tGap= =3D t2 - t1 > millisecondsForASecond ? t2 - t1 : millisecondsForASecond= ;
=C2=A0 =C2=A0 =C2=A0 =C2=A0 double gradient =3D 0.0;
=C2=A0 =C2=A0 = =C2=A0 =C2=A0 if (tGap > 0) {
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 gradient =3D ((lastVal - firstVal) * millisecondsForASecond) / tGap;=C2=A0 =C2=A0 =C2=A0 =C2=A0 }


Thanks,
Gayan

On Thu, Nov= 6, 2014 at 9:22 AM, Rajkumar Rajaratnam <rajkumarr@wso2.com> wrote:
Hi,

On Thu, Nov 6= , 2014 at 9:03 AM, Imesh Gunaratne <imesh@apache.org> wrote:<= br>
Hi Nirmal,

I do not think the milliseconds to seconds convertion is correct here= .=C2=A0

As I see we are taking the difference of t= wo timestamp values and then dividing it by 1000. The corect way might be t= o first divide each value by 1000 and then take the difference.=C2=A0
=

No Imesh. There are no differ= ence between taking the the difference and dividing it by 1000 and <= b>dividing it by 1000 and taking the difference.


On th= e other hand we might not need to convert these values to seconds since we = are taking a time difference and calculating a gradient.

=
I did a quick test with the following sample:
Gr= adient: -999.9999999999998 Last val: 7.000000000000001 First val: 12.0 Time= Gap: 5 t1: 1415213232152 t2: 1415213232157

According previous code:
long tGap =3D t2 - t1;
=C2=A0 =C2=A0 =C2=A0 =C2=A0 double gradient =3D 0.0;
=C2=A0= =C2=A0 =C2=A0 =C2=A0 if (tGap > 0) {
=C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 gradient =3D ((lastVal - firstVal) * 1000) / tGap;
<= div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 }

t1: 1= 415213232152 t2: 1415213232157 firstVal: 12 lastVall: 7
gradient:= -1000.0

According to your fix:
=
long millisecondsForASecond =3D 1000;
=C2=A0 =C2=A0 =C2= =A0 =C2=A0 long tGap =3D t2 - t1 > millisecondsForASecond ? t2 - t1 : mi= llisecondsForASecond;
=C2=A0 =C2=A0 =C2=A0 =C2=A0 double gradient= =3D 0.0;
=C2=A0 =C2=A0 =C2=A0 =C2=A0 if (tGap > 0) {
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 gradient =3D ((lastVal - firstV= al) * millisecondsForASecond) / tGap;
=C2=A0 =C2=A0 =C2=A0 =C2=A0= }

t1: 1415213232152 t2: 1415213232157 = firstVal: 12 lastVall: 7
gradient: -5.0

According to an online gradient calculator:
gradient= : -1

According to the online gradient calculator (assuming their calcul= ation is correct), the calculation in your fix is not correct. I believe th= e logic should be simple as follows:

long tGa= p =3D t2 - t1;
=C2=A0 =C2=A0 =C2=A0 =C2=A0 double gradient =3D 0.= 0;
=C2=A0 =C2=A0 =C2=A0 =C2=A0 if (tGap > 0) {
=C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 gradient =3D ((lastVal - firstVal)) = / tGap;
=C2=A0 =C2=A0 =C2=A0 =C2=A0 }

<= div>
t1: 1415213232152 t2: 1415213232157 firstVal: 12 lastVall: 7
=
gradient: -1.0

And AFAIK, standard way to calculate gradient =3D (v1-v2)/timeInSeco= nds.

Well, we can divide it by what ever time units we w= ants, but the result's unit will be different.

In Ni= rmal's case, he get will get stats/seconds. In your case, you wi= ll get stats/milliseconds

And AS prediction is bas= ed on seconds, CEP should use seconds for the calculation.
wdyt?

Thanks.


Thanks


=
On Thu, Nov 6, 2014 at 1:32 AM, Nirmal Fernando = <nirmal070125@gmail.com> wrote:
Hi Guys,

So, I go= t a chance to reproduce and analyze this issue.=C2=A0

<= div>How we calculate gradient of two events?=C2=A0

Say the events are; e1(t1,v1) and e2(t2,v2)=C2=A0

tx - time in milliseconds when xth event occurred
vx - va= lue (memory, cpu etc.) that xth event carries

t= ime gap =3D t(2-1) =3D t2 -t1 milliseconds
time gap in second= s =3D t(2-1) =3D (t2 - t1)/1000 seconds

Hence,
Gradient =3D (v2 - v1) =C2=A0/ t(2-1) =C2=A0=3D ( (v2 - v1) * 100= 0 ) / (t2 - t1)

I've enabled debug logs fo= r CEP extension;

log4j.logger= .org.apache.stratos.cep.extension=3DDEBUG


Plea= se find the following 3 logs extracted from the debug logs;

<= /div>
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
<= div>TID: [0] [STRATOS] [2014-11-05 19:47:27,073] DEBUG {org.apache.stratos.= cep.extension.SecondDerivativeFinderWindowProcessor} - =C2=A0Gradient: -0.1= 996007984031936 Last val: 9.0 First val: 12.0 Time Gap: 15030 t1: 14= 15213202095 t2: 1415213217125 hash: 155426542

TID:= [0] [STRATOS] [2014-11-05 19:47:27,073] DEBUG {org.apache.stratos.cep.exte= nsion.SecondDerivativeFinderWindowProcessor} - =C2=A0Gradient: -999.9999999999998 Last val: 7.0= 00000000000001 First val: 12.0 Time Gap: 5 t1: 1415213232152 t2: 141= 5213232157 hash: 155426542

TID: [0] [STRATOS] [201= 4-11-05 19:47:27,074] DEBUG {org.apache.stratos.cep.extension.SecondDerivat= iveFinderWindowProcessor} - =C2=A0Gradient: -44.34884666437174 Last val: -9= 99.9999999999998 First val: -0.1996007984031936 Time Gap: 22544 t1: = 1415213209610 t2: 1415213232154 hash: 155426542
=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D

So,= as you can see the reason behind a large value is when the time gap betwee= n two subjected events is less than 1 second. This could happen since event= s are coming from different asynchronous agents and also when there are les= s number of events.

So, the fix I propose is a ver= y simple one and it will not compromise anything AFAIS.

Fix is to calculate time gap as follows;
=C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 ____ t2 - t1
=C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 | yes?
time gap =3D = t(2-1) =3D t2 -t1 > 1000 ---
=C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 |____ 1000

= I have tested this and works fine.



<= div class=3D"gmail_quote">---------- Forwarded message ----------
From: = Manula Chathurika Thantriwatte <manulac@= wso2.com>
Date: Tue, Oct 28, 2014 at 5:16 AM
Subject: R= e: CEP sends very large values for gradient and second derivative of load a= verage
To: "dev@stratos.apache.org" <dev@stratos.apache.org>


Hi Raj,

Yes it has. But it's very easy to= find the values from the agent with the time stamp. Then we can search tha= t time stamp in the CEP trace logs and compare the summarized values.
=

Thanks !

On Mon, Oct 27, 2014 at 9:23 PM, Rajkumar = Rajaratnam <rajkumarr@wso2.com> wrote:
=


On Mon, Oct 27, 2014 at 7:56 PM, Manula Chathurika Thantriwatte <= span dir=3D"ltr"><= manulac@wso2.com> wrote:
Hi Raj,

Shall we enable the cartridge age= nt debug logs. Then we can compare both CEP and agent logs with the time st= amp. Then we can narrow down where the actual problem is.

Manula, cep-trace log have the values sen= t by cartridge agent also right?=C2=A0
=C2=A0

Thanks !
=

On Mon, Oct = 27, 2014 at 5:06 PM, Rajkumar Rajaratnam <rajkumarr@wso2.com> wrote:


On Mon, Oct 27, 2014 at 12:53 PM, M= anula Chathurika Thantriwatte <manulac@wso2.com> wrote:
Hi Raj,

Is this happen f= requently or intermittently ? Once I go though the logs I'm unable to f= ind the high input values for load average. But output have high values.

22:10:38,698 [-] [Siddhi-Scheduler-pool-10-thread-1]= =C2=A0INFO EVENT_TRACE_LOGGER TenantId=3D-1234 : Output Event Adaptor : JM= SOutputAdaptor, sent
{"org.apache.stratos.messaging.event.he= alth.stat.SecondDerivativeOfLoadAverageEvent":{"message":{&q= uot;clusterId":"php.php.domain","networkPartitionId&quo= t;:"","value":"2000000.0"}}}

This is happening intermittently. Experi= enced now too. This is a serious issue. We need to somehow sort this out.
=C2=A0=C2=A0

Also networkPartitionId is empty. In the current execution = plans it took networkPartitionId for the calculation. IMO we need to update= the CEP execution plans for 4.1.0.

Thanks !
=


On Sun, Oct 26, 2014 at 8:49 AM, Manula Chathurika Thantriwat= te <manulac@wso2.com> wrote:
Hi,

Thanks Raj. I'll go through them.

Thanks !
On Sun, Oct 26, 2014 at 8:33 AM, Rajkumar Raja= ratnam <rajkumarr@wso2.com> wrote:
=
Hi Manula,

You can see those values in = the same log I attached in the previous mail.

Than= ks.

On Sun, Oct 26, 2014 at 8:29 AM, Rajkumar Rajaratnam <rajkumarr@w= so2.com> wrote:
Hi Chami= la,

I couldn't find any large values sent by agent. = There are traces for CEP sending large values. Find the cep trace log here = [1]


Thanks.

On Sun, Oct 26, 2014 at 8:23 AM,= Manula Chathurika Thantriwatte <manulac@wso2.com> wrote:
=
Hi,

Shall we enable th= e cartridge agent debug logs and see what are the load average values sent = from the agent. If those values are normal then we can narrow down to the C= EP.

Thanks !

On Sat, Oct 25, 2014 at 10:43 PM,= Chamila De Alwis <chamilad@wso2.com> wrote:
Hi Raj,

Can you inspect the CEP tr= ace logs for the same time period? It will have the events published from t= he agent.


Regards,
Chamila de Alwis=
Software = Engineer | WSO2 | +947722071= 63
Blog: code.chamiladealwis.com



On Sat, Oct 25, 2014 at 10:39 PM, Rajk= umar Rajaratnam <rajkumarr@wso2.com> wrote:
=
Hi,

I am doing a = test round with M3 with python agent. And faced this very strange scenario,= $Subject.


TID: [0] [STRATOS] = [2014-10-25 22:10:38,693] DEBUG {org.apache.stratos.autoscaler.KubernetesCl= usterContext} - =C2=A0Load average stats are reset, ready to do scale check= [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,693] DEBUG {org.apache.strato= s.autoscaler.monitor.KubernetesClusterMonitor} - =C2=A0Grad of load avg eve= nt: [cluster] php.php.domain [value] -2000.0
TID: [0] [STR= ATOS] [2014-10-25 22:10:38,694] DEBUG {org.apache.stratos.autoscaler.Kubern= etesClusterContext} - =C2=A0Load average stats are reset, ready to do scale= check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10= :38,694] DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMoni= tor} - =C2=A0Grad of load avg event: [cluster] php.php.domain [value] 0.0
TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG {org.apache.str= atos.autoscaler.KubernetesClusterContext} - =C2=A0Load average stats are re= set, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRA= TOS] [2014-10-25 22:10:38,694] DEBUG {org.apache.stratos.autoscaler.monitor= .KubernetesClusterMonitor} - =C2=A0Avg load avg event: [cluster] php.php.do= main [value] 1.9090909
TID: [0] [STRATOS] [2014-10-25 22:10:38,69= 4] DEBUG {org.apache.stratos.autoscaler.KubernetesClusterContext} - =C2=A0L= oad average stats are reset, ready to do scale check [kub cluster] KubGrp1<= /div>
TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG {org.apache.st= ratos.autoscaler.monitor.KubernetesClusterMonitor} - =C2=A0Avg Memory Consu= mption event: [cluster] php.php.domain [value] 28.718182
TID: [0]= [STRATOS] [2014-10-25 22:10:38,695] DEBUG {org.apache.stratos.autoscaler.m= onitor.KubernetesClusterMonitor} - =C2=A0Second Derivation of Memory Consum= ption event: [cluster] php.php.domain [value] 200000.0
TID: [0] [= STRATOS] [2014-10-25 22:10:38,695] DEBUG {org.apache.stratos.autoscaler.mon= itor.KubernetesClusterMonitor} - =C2=A0Grad of load avg event: [cluster] ph= p.php.domain [value] 0.0
TID: [0] [STRATOS] [2014-10-25 22:10:38,= 695] DEBUG {org.apache.stratos.autoscaler.KubernetesClusterContext} - =C2= =A0Load average stats are reset, ready to do scale check [kub cluster] KubG= rp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG {org.apach= e.stratos.autoscaler.monitor.KubernetesClusterMonitor} - =C2=A0Second Deriv= ation of load avg event: [cluster] php.php.domain [value] 0.0
TID= : [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG {org.apache.stratos.autosca= ler.KubernetesClusterContext} - =C2=A0Load average stats are reset, ready t= o do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-1= 0-25 22:10:38,696] DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesC= lusterMonitor} - =C2=A0Second Derivation of load avg event: [cluster] php.p= hp.domain [value] 0.0
TID: [0] [STRATOS] [2014-10-25 22:10:38,696= ] DEBUG {org.apache.stratos.autoscaler.KubernetesClusterContext} - =C2=A0Lo= ad average stats are reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG {org.apache.str= atos.autoscaler.monitor.KubernetesClusterMonitor} - =C2=A0Avg load avg even= t: [cluster] php.php.domain [value] 3.3636363
TID: [0] [STRATOS] = [2014-10-25 22:10:38,696] DEBUG {org.apache.stratos.autoscaler.KubernetesCl= usterContext} - =C2=A0Load average stats are reset, ready to do scale check= [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,69= 6] DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -= =C2=A0Grad of Memory Consumption event: [cluster] php.php.domain [value] 0= .11670045
TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG {org= .apache.stratos.autoscaler.KubernetesClusterContext} - =C2=A0Memory consump= tion stats are reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG {org.apache.stratos.a= utoscaler.monitor.KubernetesClusterMonitor} - =C2=A0Avg Memory Consumption = event: [cluster] php.php.domain [value] 28.7
TID: [0] [STRATOS] [= 2014-10-25 22:10:38,697] DEBUG {org.apache.stratos.autoscaler.KubernetesClu= sterContext} - =C2=A0Memory consumption stats are reset, ready to do scale = check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:= 38,697] DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonit= or} - =C2=A0Second Derivation of Memory Consumption event: [cluster] php.ph= p.domain [value] 0.0
TID: [0] [STRATOS] [2014-10-25 22:10:38,697]= DEBUG {org.apache.stratos.autoscaler.KubernetesClusterContext} - =C2=A0Mem= ory consumption stats are reset, ready to do scale check [kub cluster] KubG= rp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG {org.apach= e.stratos.autoscaler.monitor.KubernetesClusterMonitor} - =C2=A0Avg Memory C= onsumption event: [cluster] php.php.domain [value] 28.7
TID: [0] = [STRATOS] [2014-10-25 22:10:38,697] DEBUG {org.apache.stratos.autoscaler.Ku= bernetesClusterContext} - =C2=A0Memory consumption stats are reset, ready t= o do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-1= 0-25 22:10:38,697] DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesC= lusterMonitor} - =C2=A0Avg load avg event: [cluster] php.php.domain [value]= 5.0
TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG {org.apac= he.stratos.autoscaler.KubernetesClusterContext} - =C2=A0Load average stats = are reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:1= 0:38,697] DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMon= itor} - =C2=A0Grad of load avg event: [cluster] php.php.domain [value] 2000= .0
TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG {org= .apache.stratos.autoscaler.KubernetesClusterContext} - =C2=A0Load average s= tats are reset, ready to do scale check [kub cluster] KubGrp1
TID= : [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG {org.apache.stratos.autosca= ler.monitor.KubernetesClusterMonitor} - =C2=A0Avg Memory Consumption event:= [cluster] php.php.domain [value] 28.7
TID: [0] [STRATOS] [2014-1= 0-25 22:10:38,698] DEBUG {org.apache.stratos.autoscaler.KubernetesClusterCo= ntext} - =C2=A0Memory consumption stats are reset, ready to do scale check = [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,698= ] DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - = =C2=A0Grad of Memory Consumption event: [cluster] php.php.domain [value] 0.= 0
TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG {org.apache.= stratos.autoscaler.KubernetesClusterContext} - =C2=A0Memory consumption sta= ts are reset, ready to do scale check [kub cluster] KubGrp1
TID: = [0] [STRATOS] [2014-10-25 22:10:38,699] DEBUG {org.apache.stratos.autoscale= r.monitor.KubernetesClusterMonitor} - =C2=A0Second Derivation of Memory Con= sumption event: [cluster] php.php.domain [value] 0.0
TID: [0] [ST= RATOS] [2014-10-25 22:10:38,699] DEBUG {org.apache.stratos.autoscaler.Kuber= netesClusterContext} - =C2=A0Memory consumption stats are reset, ready to d= o scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG {org.= apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - =C2=A0Second = Derivation of load avg event: [cluster] php.php.domain [value] 2000000.0
TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG {org.apac= he.stratos.autoscaler.KubernetesClusterContext} - =C2=A0Load average stats = are reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:1= 0:38,700] DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMon= itor} - =C2=A0Grad of load avg event: [cluster] php.php.domain [value] 1350= 0.0
TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG {or= g.apache.stratos.autoscaler.KubernetesClusterContext} - =C2=A0Load average = stats are reset, ready to do scale check [kub cluster] KubGrp1
TI= D: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG {org.apache.stratos.autosc= aler.monitor.KubernetesClusterMonitor} - =C2=A0Grad of Memory Consumption e= vent: [cluster] php.php.domain [value] 50.0
TID: [0] [STRATOS] [2= 014-10-25 22:10:38,701] DEBUG {org.apache.stratos.autoscaler.KubernetesClus= terContext} - =C2=A0Memory consumption stats are reset, ready to do scale c= heck [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:3= 8,701] DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonito= r} - =C2=A0Avg Memory Consumption event: [cluster] php.php.domain [value] 2= 8.738462
TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG {org.= apache.stratos.autoscaler.KubernetesClusterContext} - =C2=A0Memory consumpt= ion stats are reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG {org.apache.stratos.au= toscaler.monitor.KubernetesClusterMonitor} - =C2=A0Avg load avg event: [clu= ster] php.php.domain [value] 2.3333333
TID: [0] [STRATOS] [2014-1= 0-25 22:10:38,702] DEBUG {org.apache.stratos.autoscaler.KubernetesClusterCo= ntext} - =C2=A0Load average stats are reset, ready to do scale check [kub c= luster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBU= G {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - =C2=A0= Grad of load avg event: [cluster] php.php.domain [value] 0.0
TID:= [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG {org.apache.stratos.autoscal= er.KubernetesClusterContext} - =C2=A0Load average stats are reset, ready to= do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10= -25 22:10:38,703] DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesCl= usterMonitor} - =C2=A0Grad of Memory Consumption event: [cluster] php.php.d= omain [value] 0.0
TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DE= BUG {org.apache.stratos.autoscaler.KubernetesClusterContext} - =C2=A0Memory= consumption stats are reset, ready to do scale check [kub cluster] KubGrp1=
TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG {org.apache.s= tratos.autoscaler.monitor.KubernetesClusterMonitor} - =C2=A0Avg load avg ev= ent: [cluster] php.php.domain [value] 31.0
TID: [0] [STRATOS] [20= 14-10-25 22:10:38,703] DEBUG {org.apache.stratos.autoscaler.KubernetesClust= erContext} - =C2=A0Load average stats are reset, ready to do scale check [k= ub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,704] = DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - = =C2=A0Grad of Memory Consumption event: [cluster] php.php.domain [value] 0.= 0
TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG {org.apache.= stratos.autoscaler.KubernetesClusterContext} - =C2=A0Memory consumption sta= ts are reset, ready to do scale check [kub cluster] KubGrp1
TID: = [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG {org.apache.stratos.autoscale= r.monitor.KubernetesClusterMonitor} - =C2=A0Avg load avg event: [cluster] p= hp.php.domain [value] 39.0
TID: [0] [STRATOS] [2014-10-25 22:10:3= 8,704] DEBUG {org.apache.stratos.autoscaler.KubernetesClusterContext} - =C2= =A0Load average stats are reset, ready to do scale check [kub cluster] KubG= rp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG {org.apach= e.stratos.autoscaler.monitor.KubernetesClusterMonitor} - =C2=A0Avg load avg= event: [cluster] php.php.domain [value] 39.0
TID: [0] [STRATOS] = [2014-10-25 22:10:38,705] DEBUG {org.apache.stratos.autoscaler.KubernetesCl= usterContext} - =C2=A0Load average stats are reset, ready to do scale check= [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,706] DEBUG {org.apache.strato= s.autoscaler.monitor.KubernetesClusterMonitor} - =C2=A0Grad of load avg eve= nt: [cluster] php.php.domain [value] -11000.0
TID: [0] [ST= RATOS] [2014-10-25 22:10:38,706] DEBUG {org.apache.stratos.autoscaler.Kuber= netesClusterContext} - =C2=A0Load average stats are reset, ready to do scal= e check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:1= 0:38,708] DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMon= itor} - =C2=A0Grad of Memory Consumption event: [cluster] php.php.domain [v= alue] 0.0
TID: [0] [STRATOS] [2014-10-25 22:10:38,708] DEBUG {org= .apache.stratos.autoscaler.KubernetesClusterContext} - =C2=A0Memory consump= tion stats are reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,709] DEBUG {org.apache.stratos.a= utoscaler.monitor.KubernetesClusterMonitor} - =C2=A0Avg Memory Consumption = event: [cluster] php.php.domain [value] 28.8
TID: [0] [STRATOS] [= 2014-10-25 22:10:38,709] DEBUG {org.apache.stratos.autoscaler.KubernetesClu= sterContext} - =C2=A0Memory consumption stats are reset, ready to do scale = check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:= 38,811] DEBUG {org.apache.stratos.cloud.controller.util.PodActivationWatche= r} - =C2=A0PodActivationWatcher running : Running
TID: [0] [STRAT= OS] [2014-10-25 22:10:38,942] DEBUG {org.apache.stratos.cloud.controller.ut= il.PodActivationWatcher} - =C2=A0PodActivationWatcher running : Running
TID: [0] [STRATOS] [2014-10-25 22:10:38,945] DEBUG {org.apache.strat= os.cloud.controller.util.PodActivationWatcher} - =C2=A0PodActivationWatcher= running : Running
TID: [0] [STRATOS] [2014-10-25 22:10:43,712] D= EBUG {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor= } - =C2=A0KubernetesServiceClusterMonitor is running.. KubernetesServiceClu= sterMonitor [ kubernetesHostClusterId=3DKubGrp1, clusterId=3Dphp.php.domain= , serviceId=3Dphp]
TID: [0] [STRATOS] [2014-10-25 22:10:43,713] D= EBUG {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor= } - =C2=A0Running min check for [kub-cluster] : KubGrp1 [cluster] : php.php= .domain=C2=A0
TID: [0] [STRATOS] [2014-10-25 22:10:43,713] =C2=A0= INFO {org.apache.stratos.autoscaler.rule.RuleLog} - =C2=A0Running minimum r= ule: [kub-cluster] KubGrp1 [cluster] php.php.domain
TID: [0] [STR= ATOS] [2014-10-25 22:10:43,714] =C2=A0INFO {org.apache.stratos.autoscaler.r= ule.RuleLog} - =C2=A0[min-check] =C2=A0[cluster] : php.php.domain [Replicas= ] nonTerminated : 3
TID: [0] [STRATOS] [2014-10-25 22:10:43,714] = =C2=A0INFO {org.apache.stratos.autoscaler.rule.RuleLog} - =C2=A0[min-check]= =C2=A0[cluster] : php.php.domain [Replicas] minReplicas : 3
TID:= [0] [STRATOS] [2014-10-25 22:10:43,715] =C2=A0INFO {org.apache.stratos.aut= oscaler.rule.RuleLog} - =C2=A0Running obsolete containers rule [kub-cluster= ] : KubGrp1 [cluster] : php.php.domain
TID: [0] [STRATOS] [2014-1= 0-25 22:10:43,715] =C2=A0INFO {org.apache.stratos.autoscaler.rule.RuleLog} = - =C2=A0[obsolete-check] [cluster] : php.php.domain [Replicas] obsoleteRepl= icas : 0
TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG {org.= apache.stratos.autoscaler.rule.AutoscalerRuleEvaluator} - =C2=A0Minimum che= ck executed for : org.apache.stratos.autoscaler.KubernetesClusterContext@21= d906d0=C2=A0
TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG {= org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} - = =C2=A0flag of rifReset : true flag of memoryConsumptionReset : true flag of= loadAverageReset : true
TID: [0] [STRATOS] [2014-10-25 22:10:43,= 715] DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterM= onitor} - =C2=A0Running scale check for [kub-cluster] : KubGrp1 [cluster] := php.php.domain=C2=A0
TID: [0] [STRATOS] [2014-10-25 22:10:43,716= ] =C2=A0INFO {org.apache.stratos.autoscaler.rule.RuleLog} - =C2=A0Running s= caling rule [kub-cluster] : KubGrp1 [cluster] : php.php.domain
TI= D: [0] [STRATOS] [2014-10-25 22:10:43,716] DEBUG {org.apache.stratos.autosc= aler.rule.RuleTasksDelegator} - =C2=A0Predicting the value, [average]: 0.0 = , [gradient]: 0.0 , [second derivative]: 0.0 , [time intervals]: 1=C2=A0
TID: [0] [STRATOS] [2014-10-25 22:10:43,716] DEBUG {org.apache.stra= tos.autoscaler.rule.RuleTasksDelegator} - =C2=A0Predicting the value, [aver= age]: 28.8 , [gradient]: 0.0 , [second derivative]: 0.0 , [time intervals]:= 1=C2=A0
TID: [0] [STRATOS] [2014-10-25 22:10:43,717] DEBUG {org.= apache.stratos.autoscaler.rule.RuleTasksDelegator} - =C2=A0Predicting the v= alue, [average]: 39.0 , [gradient]: -11000.0 , [second derivative]: 2000000= .0 , [time intervals]: 1=C2=A0
TID: [0] [STRATOS] [2014-10-25 22:= 10:43,717] =C2=A0INFO {org.apache.stratos.autoscaler.rule.RuleLog} - =C2=A0= [scaling] =C2=A0[cluster] : php.php.domain [Replicas] minReplicas : 3
=
TID: [0] [STRATOS] [2014-10-25 22:10:43,717] =C2=A0INFO {org.apache.st= ratos.autoscaler.rule.RuleLog} - =C2=A0[scaling] =C2=A0[cluster] : php.php.= domain [Replicas] maxReplicas : 10
TID: [0] [STRATOS] [2014-10-25= 22:10:43,718] =C2=A0INFO {org.apache.stratos.autoscaler.rule.RuleLog} - = =C2=A0[scaling] =C2=A0[cluster] : php.php.domain [Replicas] nonTerminated := 3
TID: [0] [STRATOS] [2014-10-25 22:10:43,718] =C2=A0INFO {org.a= pache.stratos.autoscaler.rule.RuleLog} - =C2=A0[scaling] =C2=A0[cluster] : = php.php.domain [Replicas] activeReplicas : 3
TID: [0] [STRATOS] [= 2014-10-25 22:10:43,718] =C2=A0INFO {org.apache.stratos.autoscaler.rule.Rul= eLog} - =C2=A0[scaling] =C2=A0[cluster] : php.php.domain [RequestInFlight] = predicted value : 0.0
TID: [0] [STRATOS] [2014-10-25 22:10:43,718= ] =C2=A0INFO {org.apache.stratos.autoscaler.rule.RuleLog} - =C2=A0[scaling]= =C2=A0[cluster] : php.php.domain [RequestInFlight] upper limit : 80.0
TID: [0] [STRATOS] [2014-10-25 22:10:43,718] =C2=A0INFO {org.apache.s= tratos.autoscaler.rule.RuleLog} - =C2=A0[scaling] =C2=A0[cluster] : php.php= .domain [RequestInFlight] lower limit : 5.0
TID: [0] [STRATOS] [2= 014-10-25 22:10:43,718] =C2=A0INFO {org.apache.stratos.autoscaler.rule.Rule= Log} - =C2=A0[scaling] =C2=A0[cluster] : php.php.domain [MemoryConsumption]= predicted value : 28.799999237060547
TID: [0] [STRATOS] [2014-10= -25 22:10:43,718] =C2=A0INFO {org.apache.stratos.autoscaler.rule.RuleLog} -= =C2=A0[scaling] =C2=A0[cluster] : php.php.domain [MemoryConsumption] upper= limit : 80.0
TID: [0] [STRATOS] [2014-10-25 22:10:43,718] =C2=A0= INFO {org.apache.stratos.autoscaler.rule.RuleLog} - =C2=A0[scaling] =C2=A0[= cluster] : php.php.domain [MemoryConsumption] lower limit : 15.0
= TID: [0] [STRATOS] [2014-10-25 22:10:43,718] =C2=A0INFO {org.apache.stratos= .autoscaler.rule.RuleLog} - =C2=A0[scaling] =C2=A0[cluster] : php.php.domai= n [LoadAverage] predicted value : 989039.0
TID: [0] [STRATOS] [20= 14-10-25 22:10:43,719] =C2=A0INFO {org.apache.stratos.autoscaler.rule.RuleL= og} - =C2=A0[scaling] =C2=A0[cluster] : php.php.domain [LoadAverage] upper = limit : 180.0
TID: [0] [STRATOS] [2014-10-25 22:10:43,719] =C2=A0= INFO {org.apache.stratos.autoscaler.rule.RuleLog} - =C2=A0[scaling] =C2=A0[= cluster] : php.php.domain [LoadAverage] lower limit : 20.0
TID: [= 0] [STRATOS] [2014-10-25 22:10:43,719] =C2=A0INFO {org.apache.stratos.autos= caler.rule.RuleLog} - =C2=A0[scaling] =C2=A0[cluster] : php.php.domain scal= e-up action : true
TID: [0] [STRATOS] [2014-10-25 22:10:43,719] = =C2=A0INFO {org.apache.stratos.autoscaler.rule.RuleLog} - =C2=A0[scaling] = =C2=A0[cluster] : php.php.domain scale-down action : false
TID: [= 0] [STRATOS] [2014-10-25 22:10:43,719] =C2=A0INFO {org.apache.stratos.autos= caler.rule.RuleLog} - =C2=A0[scaling] =C2=A0[cluster] : php.php.domain [Loa= dAverage] predicted replicas : 16484
TID: [0] [STRATOS] [2014-10-= 25 22:10:43,720] =C2=A0INFO {org.apache.stratos.autoscaler.rule.RuleLog} - = =C2=A0[scaling] =C2=A0[cluster] : php.php.domain predicted replicas > ma= x replicas :=C2=A0
TID: [0] [STRATOS] [2014-10-25 22:10:43,720] = =C2=A0INFO {org.apache.stratos.autoscaler.rule.RuleLog} - =C2=A0[scaling] D= ecided to scale-up : [cluster] : php.php.domain
TID: [0] [STRATOS= ] [2014-10-25 22:10:43,720] =C2=A0INFO {org.apache.stratos.autoscaler.rule.= RuleLog} - =C2=A0[scaling-up] =C2=A0[cluster] : php.php.domain valid number= of replicas to expand : 10
TID: [0] [STRATOS] [2014-10-25 22:10:= 43,720] =C2=A0INFO {org.apache.stratos.autoscaler.client.cloud.controller.C= loudControllerClient} - =C2=A0Updating kubernetes replication controller vi= a cloud controller: [cluster] php.php.domain [replicas] 10
TID: [= 0] [STRATOS] [2014-10-25 22:10:43,734] DEBUG {org.apache.stratos.cloud.cont= roller.impl.CloudControllerServiceImpl} - =C2=A0CloudControllerServiceImpl:= updateContainers for cluster : php.php.domain
TID: [0] [STRATOS] = [2014-10-25 22:10:43,758] DEBUG {org.apache.stratos.cloud.controller.impl.C= loudControllerServiceImpl} - =C2=A0Cloud Controller is delegating request t= o update a replication controller php.php.domain to Kubernetes layer.
=


I couldn't investigate the agent sid= e, since containers are destroyed.

Any idea?
=

Thanks.

--
Rajkumar Rajaratnam
Software Engineer | W= SO2, Inc.




<= font color=3D"#888888">--
=
= Regards,
Manula = Chathurika Thantriwatte
Software Engineer
WSO2 = Inc. : http://wso2.com
lean . enterprise . middleware

<= /font>
<= span style=3D"color:rgb(153,153,153)">blog : http://manulachathurika.blogspot.com/






--
Rajkumar Rajaratnam
Software Engineer | WSO2, Inc.



--
=
Rajkumar Rajaratnam
Software Engineer | WSO2, Inc.<= /font>



--
=
<= div>
Regards,
Manula Chathurika Thantriwatte
Software Engineer
WSO2 Inc. : http://wso2.com
lean . enterpr= ise . middleware




--
=
<= div>
Regards,
Manula Chathurika Thantriwatte
Software Engineer
WSO2 Inc. : http://wso2.com
lean . enterpr= ise . middleware



=

--
Rajkumar Rajaratnam
Software = Engineer | WSO2, Inc.



--
Regards,
Manula Chathurika Thantriwatte
Soft= ware Engineer
WSO2 Inc. : http://wso2.com
lean . enterprise . middle= ware

<= div>email : manula= c@wso2.com / man= ula@apache.org
<= font color=3D"#888888">phone : +94 772492511



=



--
Manul= a Chathurika Thantriwatte
Software Engineer
WSO= 2 Inc. : http://wso2.com<= br>
<= font color=3D"#888888">lean . enterprise . middleware

<= font color=3D"#888888">phone : +94 772492511






--
Best Regards,
Nirma= l

Nirmal Fernando.
PPMC Member & Committer of Apache Stratos,=
Senior Software Engineer, WSO2 Inc.

Blog:=C2=A0<= a href=3D"http://nirmalfdo.blogspot.com/" target=3D"_blank">http://nirmalfd= o.blogspot.com/


<= span>--
Imesh Gunaratne
Technical Lead, WSO2
Committer &am= p; PMC Member, Apache Stratos
=


--
Raj
<= /div>



<= font color=3D"#888888">--

Gayan Gunarathne
Technical Lead
WSO2 Inc. <= /span>(http://wso2.com)
email=C2= =A0 : gayang@wso2.com<= /a>=C2=A0 | m<= font color=3D"#888888">obile : +94 766819985
=C2=A0
<= /font>



--
Best Regards,
Nirmal

Nirmal Ferna= ndo.
PPMC Member & Committer of Apache Stratos,
Senior Software E= ngineer, WSO2 Inc.




--
<= div dir=3D"ltr">Imesh Gunaratne

Technical Lead, WSO2
Committer & PMC Member, Apache Stratos

--
Thanks= and Regards,

Isuru H.
+94 716 358 048




--001a11c27c84e9aa6305072c5bf3--