Return-Path: X-Original-To: apmail-stratos-dev-archive@minotaur.apache.org Delivered-To: apmail-stratos-dev-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id A475617A21 for ; Mon, 13 Oct 2014 09:22:18 +0000 (UTC) Received: (qmail 42224 invoked by uid 500); 13 Oct 2014 09:22:18 -0000 Delivered-To: apmail-stratos-dev-archive@stratos.apache.org Received: (qmail 42173 invoked by uid 500); 13 Oct 2014 09:22:18 -0000 Mailing-List: contact dev-help@stratos.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@stratos.apache.org Delivered-To: mailing list dev@stratos.apache.org Received: (qmail 42163 invoked by uid 99); 13 Oct 2014 09:22:18 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Oct 2014 09:22:18 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_FILL_THIS_FORM_SHORT X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of manulac@wso2.com designates 209.85.218.43 as permitted sender) Received: from [209.85.218.43] (HELO mail-oi0-f43.google.com) (209.85.218.43) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Oct 2014 09:22:12 +0000 Received: by mail-oi0-f43.google.com with SMTP id u20so12354536oif.16 for ; Mon, 13 Oct 2014 02:21:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=wso2.com; s=google; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=4pqqvARbLfyHGNyOZocdcnJ38heY1x2qgylDEVYJWnY=; b=ShqSZNp2OMoHXbDIIljHrRtTbC82BGgkQKXSr9GhnJpBW+pemD0ylwa14/Jk9WNUl4 k4dVh4uu95AMVO2cDOPyqpnNQe3Z1nIs3yoSgteAih6g9Ls7JdPBlZ6AK+G+GUD/WOGd eZHcKjtRh3rolwTaCc+luPgn+QXnzS/vUk4RU= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:content-type; bh=4pqqvARbLfyHGNyOZocdcnJ38heY1x2qgylDEVYJWnY=; b=nJ9QhH39PI8qCwMelabVng/KPlAOUn+IoWZHb9YaR3ODJ7tZ3nn5G6CzhbaMjFxrfn uPKHTnKGvXe8MjmqTa1yCtgMOsgUcNlCOlszlIpp8YaTA17p6wysQi0zCZ7WBJFqwCxK Niq1Q9GVf489YxlxUJrOSM+0KPVmE+cgqI+tIgvsMIGtE+zSE6oa7od3Zr7lyrU3JLmQ dEGMiAvSs5SRTv/TOBZiQaG/INKfJzhQUdMxzidNOs+SLmhpOoUH/zMTs0Vmae+VmMoJ 6Uoz2uA9X2jNPg7YSvwbNvQMokiRCIB4HWRgTtiGjbSeLc/TrVXX4Dze0tEwhEjw1XuC KzYg== X-Gm-Message-State: ALoCoQn0tHs1bE7O1L0BbTnGn0CKW4fYwqHy4TqP21LemUPhKO/bIy0UJ3pcVRZkIW9SgAHiL1e/ MIME-Version: 1.0 X-Received: by 10.182.110.129 with SMTP id ia1mr20915751obb.4.1413192111634; Mon, 13 Oct 2014 02:21:51 -0700 (PDT) Received: by 10.76.71.166 with HTTP; Mon, 13 Oct 2014 02:21:51 -0700 (PDT) In-Reply-To: References: Date: Mon, 13 Oct 2014 14:51:51 +0530 Message-ID: Subject: Re: CEP removed a terminated member from memberTimeStampMap, but complaining it as a faulty member From: Manula Chathurika Thantriwatte To: "dev@stratos.apache.org" Content-Type: multipart/alternative; boundary=089e0112ce22d0836c05054a6d74 X-Virus-Checked: Checked by ClamAV on apache.org --089e0112ce22d0836c05054a6d74 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Hi Raj, It seems like [1] get executed again. memberTimeStampMap is a concurrent hash map. Seems that CEPTopologyEventReceiver unable to remove the element in the memberTimeStampMap. [1] https://github.com/apache/stratos/blob/master/extensions/cep/stratos-cep-ex= tension/src/main/java/org/apache/stratos/cep/extension/FaultHandlingWindowP= rocessor.java#L220 Thanks ! On Mon, Oct 13, 2014 at 11:48 AM, Rajkumar Rajaratnam wrote: > Hi, > > CEP extension FaultHandlingWindowProcessor removing a terminated member > from the map. But after a while, it is complaining it as a faulty member. > Here is the log. MemberId highlighted. > > TID: [0] [STRATOS] [2014-10-13 11:15:22,409] DEBUG > {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the > value, [average]: 0.0 , [gradient]: 0.0 , [second derivative]: 0.0 , [tim= e > intervals]: 1 > TID: [0] [STRATOS] [2014-10-13 11:15:22,410] DEBUG > {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the > value, [average]: 41.63388 , [gradient]: -0.013520682 , [second > derivative]: 0.0013238431 , [time intervals]: 1 > TID: [0] [STRATOS] [2014-10-13 11:15:22,410] DEBUG > {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the > value, [average]: 99.9 , [gradient]: -2.0885546 , [second derivative]: > 0.03460533 , [time intervals]: 1 > TID: [0] [STRATOS] [2014-10-13 11:15:22,411] INFO > {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : > myphp.php.domain [Replicas] minReplicas : 2 > TID: [0] [STRATOS] [2014-10-13 11:15:22,411] INFO > {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : > myphp.php.domain [Replicas] maxReplicas : 20 > TID: [0] [STRATOS] [2014-10-13 11:15:22,411] INFO > {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : > myphp.php.domain [Replicas] nonTerminated : 5 > TID: [0] [STRATOS] [2014-10-13 11:15:22,411] INFO > {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : > myphp.php.domain [Replicas] activeReplicas : 5 > TID: [0] [STRATOS] [2014-10-13 11:15:22,412] DEBUG > {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : > myphp.php.domain [RequestInFlight] resetted ? : false > TID: [0] [STRATOS] [2014-10-13 11:15:22,412] DEBUG > {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : > myphp.php.domain [RequestInFlight] predicted value : 0.0 > TID: [0] [STRATOS] [2014-10-13 11:15:22,412] DEBUG > {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : > myphp.php.domain [RequestInFlight] upper limit : 80.0 > TID: [0] [STRATOS] [2014-10-13 11:15:22,412] DEBUG > {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : > myphp.php.domain [RequestInFlight] lower limit : 5.0 > TID: [0] [STRATOS] [2014-10-13 11:15:22,412] DEBUG > {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : > myphp.php.domain [MemoryConsumption] resetted ? : true > TID: [0] [STRATOS] [2014-10-13 11:15:22,413] INFO > {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : > myphp.php.domain [MemoryConsumption] predicted value : 41.62102324969601 > TID: [0] [STRATOS] [2014-10-13 11:15:22,413] INFO > {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : > myphp.php.domain [MemoryConsumption] upper limit : 80.0 > TID: [0] [STRATOS] [2014-10-13 11:15:22,413] INFO > {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : > myphp.php.domain [MemoryConsumption] lower limit : 15.0 > TID: [0] [STRATOS] [2014-10-13 11:15:22,415] DEBUG > {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : > myphp.php.domain [LoadAverage] resetted ? : true > TID: [0] [STRATOS] [2014-10-13 11:15:22,415] INFO > {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : > myphp.php.domain [LoadAverage] predicted value : 97.82874980941415 > TID: [0] [STRATOS] [2014-10-13 11:15:22,415] INFO > {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : > myphp.php.domain [LoadAverage] upper limit : 180.0 > TID: [0] [STRATOS] [2014-10-13 11:15:22,415] INFO > {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : > myphp.php.domain [LoadAverage] lower limit : 20.0 > TID: [0] [STRATOS] [2014-10-13 11:15:22,416] INFO > {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : > myphp.php.domain scale-up action : false > TID: [0] [STRATOS] [2014-10-13 11:15:22,416] INFO > {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : > myphp.php.domain scale-down action : true > TID: [0] [STRATOS] [2014-10-13 11:15:22,416] INFO > {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] Decided to > scale-down [cluster] : myphp.php.domain > TID: [0] [STRATOS] [2014-10-13 11:15:22,417] INFO > {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling-down] [cluster]= : > myphp.php.domain shrink the cluster to minReplicas : 2 > TID: [0] [STRATOS] [2014-10-13 11:15:22,417] INFO > {org.apache.stratos.autoscaler.client.cloud.controller.CloudControllerCli= ent} > - Updating kubernetes replication controller via cloud controller: > [cluster] myphp.php.domain [replicas] 2 > TID: [0] [STRATOS] [2014-10-13 11:15:22,425] DEBUG > {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} - > CloudControllerServiceImpl:updateContainers for cluster : myphp.php.doma= in > TID: [0] [STRATOS] [2014-10-13 11:15:22,448] DEBUG > {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} - > Cloud Controller is delegating request to update a replication controlle= r > myphp.php.domain to Kubernetes layer. > TID: [0] [STRATOS] [2014-10-13 11:15:22,506] DEBUG > {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} - > Cloud Controller successfully updated the controller myphp.php.domain vi= a > Kubernetes layer. > TID: [0] [STRATOS] [2014-10-13 11:15:22,531] DEBUG > {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} - > Pods Count: 5 for cluster: myphp.php.domain > TID: [0] [STRATOS] [2014-10-13 11:15:22,548] DEBUG > {org.apache.stratos.cloud.controller.util.PodActivationWatcher} - > PodActivationWatcher running : Running > TID: [0] [STRATOS] [2014-10-13 11:15:22,561] DEBUG > {org.apache.stratos.cloud.controller.util.PodActivationWatcher} - > PodActivationWatcher running : Running > TID: [0] [STRATOS] [2014-10-13 11:15:22,563] DEBUG > {org.apache.stratos.cloud.controller.util.PodActivationWatcher} - > PodActivationWatcher running : Running > TID: [0] [STRATOS] [2014-10-13 11:15:22,655] DEBUG > {org.apache.stratos.cloud.controller.util.PodActivationWatcher} - > PodActivationWatcher running : Running > TID: [0] [STRATOS] [2014-10-13 11:15:22,657] DEBUG > {org.apache.stratos.cloud.controller.util.PodActivationWatcher} - > PodActivationWatcher running : Running > TID: [0] [STRATOS] [2014-10-13 11:15:27,538] DEBUG > {org.apache.stratos.cloud.controller.util.PodActivationWatcher} - > PodActivationWatcher running : Running > TID: [0] [STRATOS] [2014-10-13 11:15:27,539] ERROR > {org.apache.stratos.kubernetes.client.KubernetesApiClient} - Pod > [b446c0dd-529b-11e4-acbb-08002794b041] doesn't exist. > TID: [0] [STRATOS] [2014-10-13 11:15:27,539] ERROR > {org.apache.stratos.cloud.controller.util.PodActivationWatcher} - > Container Activation Watcher Failed.. Cause: Pod > [b446c0dd-529b-11e4-acbb-08002794b041] doesn't exist. > TID: [0] [STRATOS] [2014-10-13 11:15:27,539] DEBUG > {org.apache.stratos.cloud.controller.util.PodActivationWatcher} - > org.apache.stratos.kubernetes.client.exceptions.KubernetesClientExceptio= n: > Pod [b446c0dd-529b-11e4-acbb-08002794b041] doesn't exist. > TID: [0] [STRATOS] [2014-10-13 11:15:27,539] ERROR > {org.apache.stratos.kubernetes.client.KubernetesApiClient} - Pod > [b4468fb6-529b-11e4-acbb-08002794b041] doesn't exist. > TID: [0] [STRATOS] [2014-10-13 11:15:27,540] ERROR > {org.apache.stratos.cloud.controller.util.PodActivationWatcher} - > Container Activation Watcher Failed.. Cause: Pod > [b4468fb6-529b-11e4-acbb-08002794b041] doesn't exist. > TID: [0] [STRATOS] [2014-10-13 11:15:27,540] DEBUG > {org.apache.stratos.cloud.controller.util.PodActivationWatcher} - > org.apache.stratos.kubernetes.client.exceptions.KubernetesClientExceptio= n: > Pod [b4468fb6-529b-11e4-acbb-08002794b041] doesn't exist. > TID: [0] [STRATOS] [2014-10-13 11:15:27,628] ERROR > {org.apache.stratos.kubernetes.client.KubernetesApiClient} - Pod > [3ccf4e4e-529a-11e4-acbb-08002794b041] doesn't exist. > TID: [0] [STRATOS] [2014-10-13 11:15:27,628] ERROR > {org.apache.stratos.cloud.controller.util.PodActivationWatcher} - > Container Activation Watcher Failed.. Cause: Pod > [3ccf4e4e-529a-11e4-acbb-08002794b041] doesn't exist. > TID: [0] [STRATOS] [2014-10-13 11:15:27,628] DEBUG > {org.apache.stratos.cloud.controller.util.PodActivationWatcher} - > org.apache.stratos.kubernetes.client.exceptions.KubernetesClientExceptio= n: > Pod [3ccf4e4e-529a-11e4-acbb-08002794b041] doesn't exist. > TID: [0] [STRATOS] [2014-10-13 11:15:27,630] DEBUG > {org.apache.stratos.cloud.controller.util.PodActivationWatcher} - > PodActivationWatcher running : Running > TID: [0] [STRATOS] [2014-10-13 11:15:32,542] ERROR > {org.apache.stratos.kubernetes.client.KubernetesApiClient} - Pod > [b4468fb6-529b-11e4-acbb-08002794b041] doesn't exist. > TID: [0] [STRATOS] [2014-10-13 11:15:32,542] ERROR > {org.apache.stratos.cloud.controller.util.PodActivationWatcher} - > Container Activation Watcher Failed.. Cause: Pod > [b4468fb6-529b-11e4-acbb-08002794b041] doesn't exist. > TID: [0] [STRATOS] [2014-10-13 11:15:32,542] DEBUG > {org.apache.stratos.cloud.controller.util.PodActivationWatcher} - > org.apache.stratos.kubernetes.client.exceptions.KubernetesClientExceptio= n: > Pod [b4468fb6-529b-11e4-acbb-08002794b041] doesn't exist. > TID: [0] [STRATOS] [2014-10-13 11:15:32,549] DEBUG > {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} - > Pods Count: 2 for cluster: myphp.php.domain > TID: [0] [STRATOS] [2014-10-13 11:15:32,549] DEBUG > {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} - > Pods created : 2 for cluster : myphp.php.domain > TID: [0] [STRATOS] [2014-10-13 11:15:32,551] DEBUG > {org.apache.stratos.cloud.controller.topology.TopologyManager} - Write > lock acquired > TID: [0] [STRATOS] [2014-10-13 11:15:32,551] DEBUG > {org.apache.stratos.cloud.controller.topology.TopologyManager} - Updatin= g > topology > TID: [0] [STRATOS] [2014-10-13 11:15:32,553] DEBUG > {org.apache.stratos.cloud.controller.util.PodActivationWatcher} - > PodActivationWatcher running : Running > TID: [0] [STRATOS] [2014-10-13 11:15:32,554] ERROR > {org.apache.stratos.kubernetes.client.KubernetesApiClient} - Pod > [b446c0dd-529b-11e4-acbb-08002794b041] doesn't exist. > TID: [0] [STRATOS] [2014-10-13 11:15:32,554] ERROR > {org.apache.stratos.cloud.controller.util.PodActivationWatcher} - > Container Activation Watcher Failed.. Cause: Pod > [b446c0dd-529b-11e4-acbb-08002794b041] doesn't exist. > TID: [0] [STRATOS] [2014-10-13 11:15:32,555] DEBUG > {org.apache.stratos.cloud.controller.util.PodActivationWatcher} - > org.apache.stratos.kubernetes.client.exceptions.KubernetesClientExceptio= n: > Pod [b446c0dd-529b-11e4-acbb-08002794b041] doesn't exist. > TID: [0] [STRATOS] [2014-10-13 11:15:32,564] DEBUG > {org.apache.stratos.cloud.controller.topology.TopologyManager} - Topolog= y > updated: > {"serviceMap":{"php":{"serviceName":"php","serviceType":"SingleTenant","c= lusterIdClusterMap":{"myphp.php.domain":{"serviceName":"php","clusterId":"m= yphp.php.domain","autoscalePolicyName":"economy","hostNames":[" > myphp.apachestratos.org > "],"isLbCluster":false,"isKubernetesCluster":true,"memberMap":{"3ccf4e4e-= 529a-11e4-acbb-08002794b041":{"serviceName":"php","clusterId":"myphp.php.do= main","memberId":"3ccf4e4e-529a-11e4-acbb-08002794b041","portMap":{"8280":{= "protocol":"http","value":80,"proxy":8280}},"memberPublicIp":"172.17.8.101"= ,"status":"Activated","memberIp":"172.17.8.101","properties":{"ALLOCATED_SE= RVICE_HOST_PORT":"4500"}},"b446c0dd-529b-11e4-acbb-08002794b041":{"serviceN= ame":"php","clusterId":"myphp.php.domain","memberId":"b446c0dd-529b-11e4-ac= bb-08002794b041","portMap":{"8280":{"protocol":"http","value":80,"proxy":82= 80}},"memberPublicIp":"172.17.8.101","status":"Activated","memberIp":"172.1= 7.8.101","properties":{}},"b44650cb-529b-11e4-acbb-08002794b041":{"serviceN= ame":"php","clusterId":"myphp.php.domain","memberId":"b44650cb-529b-11e4-ac= bb-08002794b041","portMap":{"8280":{"protocol":"http","value":80,"proxy":82= 80}},"memberPublicIp":"172.17.8.101","status":"Activated","memberIp":"172.1= 7.8.101","properties":{}},"3ccfb8bd-529a-11e4-acbb-08002794b041":{"serviceN= ame":"php","clusterId":"myphp.php.domain","memberId":"3ccfb8bd-529a-11e4-ac= bb-08002794b041","portMap":{"8280":{"protocol":"http","value":80,"proxy":82= 80}},"memberPublicIp":"172.17.8.101","status":"Activated","memberIp":"172.1= 7.8.101","properties":{"ALLOCATED_SERVICE_HOST_PORT":"4500"}}},"status":"Cr= eated","properties":{"payload_parameter.APP_PATH":"/var/www","KUBERNETES_RE= PLICAS_MAX":"20","KUBERNETES_CLUSTER_ID":"KubGrp1","payload_parameter.LOG_L= EVEL":"DEBUG","KUBERNETES_REPLICAS_MIN":"2","payload_parameter.MB_IP":"172.= 17.42.1","payload_parameter.CEP_IP":"172.17.42.1","payload_parameter.MB_POR= T":"1883","payload_parameter.CEP_PORT":"7611"}}},"portMap":{"8280":{"protoc= ol":"http","value":80,"proxy":8280}},"properties":{}}},"initialized":false} > TID: [0] [STRATOS] [2014-10-13 11:15:32,564] DEBUG > {org.apache.stratos.cloud.controller.topology.TopologyManager} - Write > lock released > TID: [0] [STRATOS] [2014-10-13 11:15:32,564] INFO > {org.apache.stratos.cloud.controller.topology.TopologyEventPublisher} - = Publishing > member terminated event: [service] php [cluster] myphp.php.domain > [network-partition] null [partition] null [member] > b4468fb6-529b-11e4-acbb-08002794b041 > TID: [0] [STRATOS] [2014-10-13 11:15:32,575] INFO > {org.apache.stratos.manager.topology.receiver.StratosManagerTopologyEvent= Receiver} > - [MemberTerminatedEventListener] Received: class > org.apache.stratos.messaging.event.topology.MemberTerminatedEvent > TID: [0] [STRATOS] [2014-10-13 11:15:32,580] DEBUG > {org.apache.stratos.manager.topology.receiver.StratosManagerTopologyEvent= Receiver} > - Removed the terminated member with id > b4468fb6-529b-11e4-acbb-08002794b041 from the cluster > TID: [0] [STRATOS] [2014-10-13 11:15:32,580] DEBUG > {org.apache.stratos.manager.topology.model.TopologyClusterInformationMode= l} > - Adding cluster [myphp.php.domain] > TID: [0] [STRATOS] [2014-10-13 11:15:32,580] INFO > {org.apache.stratos.messaging.message.processor.topology.MemberTerminated= MessageProcessor} > - Member terminated: [service] php [cluster] myphp.php.domain [member] > b4468fb6-529b-11e4-acbb-08002794b041 > TID: [0] [STRATOS] [2014-10-13 11:15:32,580] DEBUG > {org.apache.stratos.cloud.controller.runtime.FasterLookUpDataHolder} - > MemberContext [id]: b4468fb6-529b-11e4-acbb-08002794b041 removed from > information model. > TID: [0] [STRATOS] [2014-10-13 11:15:32,581] INFO > {org.apache.stratos.cloud.controller.runtime.FasterLookUpDataHolder} - > Scheduled Pod Activation Watcher task canceled. > TID: [0] [STRATOS] [2014-10-13 11:15:32,581] WARN > {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Membe= r > is in the wrong list and it is removed from active members list > TID: [0] [STRATOS] [2014-10-13 11:15:32,582] INFO > {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Membe= r > stat context has been removed successfully: [member] > b4468fb6-529b-11e4-acbb-08002794b041 > TID: [0] [STRATOS] [2014-10-13 11:15:32,582] WARN > {org.apache.stratos.messaging.message.processor.topology.MemberTerminated= MessageProcessor} > - Member already terminated: [service] php [cluster] myphp.php.domain > [member] b4468fb6-529b-11e4-acbb-08002794b041 > TID: [0] [STRATOS] [2014-10-13 11:15:32,583] INFO > {org.apache.stratos.cep.extension.CEPTopologyEventReceiver} - Member > [member id] b4468fb6-529b-11e4-acbb-08002794b041 was removed from the tim= e > stamp map. > TID: [0] [STRATOS] [2014-10-13 11:15:32,583] WARN > {org.apache.stratos.messaging.message.processor.topology.MemberTerminated= MessageProcessor} > - Member already terminated: [service] php [cluster] myphp.php.domain > [member] b4468fb6-529b-11e4-acbb-08002794b041 > TID: [0] [STRATOS] [2014-10-13 11:15:32,593] DEBUG > {org.apache.stratos.cloud.controller.topology.TopologyManager} - Write > lock acquired > TID: [0] [STRATOS] [2014-10-13 11:15:32,593] DEBUG > {org.apache.stratos.cloud.controller.topology.TopologyManager} - Updatin= g > topology > TID: [0] [STRATOS] [2014-10-13 11:15:32,603] DEBUG > {org.apache.stratos.cloud.controller.topology.TopologyManager} - Topolog= y > updated: > {"serviceMap":{"php":{"serviceName":"php","serviceType":"SingleTenant","c= lusterIdClusterMap":{"myphp.php.domain":{"serviceName":"php","clusterId":"m= yphp.php.domain","autoscalePolicyName":"economy","hostNames":[" > myphp.apachestratos.org > "],"isLbCluster":false,"isKubernetesCluster":true,"memberMap":{"3ccf4e4e-= 529a-11e4-acbb-08002794b041":{"serviceName":"php","clusterId":"myphp.php.do= main","memberId":"3ccf4e4e-529a-11e4-acbb-08002794b041","portMap":{"8280":{= "protocol":"http","value":80,"proxy":8280}},"memberPublicIp":"172.17.8.101"= ,"status":"Activated","memberIp":"172.17.8.101","properties":{"ALLOCATED_SE= RVICE_HOST_PORT":"4500"}},"b44650cb-529b-11e4-acbb-08002794b041":{"serviceN= ame":"php","clusterId":"myphp.php.domain","memberId":"b44650cb-529b-11e4-ac= bb-08002794b041","portMap":{"8280":{"protocol":"http","value":80,"proxy":82= 80}},"memberPublicIp":"172.17.8.101","status":"Activated","memberIp":"172.1= 7.8.101","properties":{}},"3ccfb8bd-529a-11e4-acbb-08002794b041":{"serviceN= ame":"php","clusterId":"myphp.php.domain","memberId":"3ccfb8bd-529a-11e4-ac= bb-08002794b041","portMap":{"8280":{"protocol":"http","value":80,"proxy":82= 80}},"memberPublicIp":"172.17.8.101","status":"Activated","memberIp":"172.1= 7.8.101","properties":{"ALLOCATED_SERVICE_HOST_PORT":"4500"}}},"status":"Cr= eated","properties":{"payload_parameter.APP_PATH":"/var/www","KUBERNETES_RE= PLICAS_MAX":"20","KUBERNETES_CLUSTER_ID":"KubGrp1","payload_parameter.LOG_L= EVEL":"DEBUG","KUBERNETES_REPLICAS_MIN":"2","payload_parameter.MB_IP":"172.= 17.42.1","payload_parameter.CEP_IP":"172.17.42.1","payload_parameter.MB_POR= T":"1883","payload_parameter.CEP_PORT":"7611"}}},"portMap":{"8280":{"protoc= ol":"http","value":80,"proxy":8280}},"properties":{}}},"initialized":false} > TID: [0] [STRATOS] [2014-10-13 11:15:32,603] DEBUG > {org.apache.stratos.cloud.controller.topology.TopologyManager} - Write > lock released > TID: [0] [STRATOS] [2014-10-13 11:15:32,603] INFO > {org.apache.stratos.cloud.controller.topology.TopologyEventPublisher} - > Publishing member terminated event: [service] php [cluster] > myphp.php.domain [network-partition] null [partition] null [member] > b446c0dd-529b-11e4-acbb-08002794b041 > TID: [0] [STRATOS] [2014-10-13 11:15:32,607] INFO > {org.apache.stratos.cep.extension.CEPTopologyEventReceiver} - Member > [member id] b446c0dd-529b-11e4-acbb-08002794b041 was removed from the tim= e > stamp map. > TID: [0] [STRATOS] [2014-10-13 11:15:32,608] INFO > {org.apache.stratos.messaging.message.processor.topology.MemberTerminated= MessageProcessor} > - Member terminated: [service] php [cluster] myphp.php.domain [member] > b446c0dd-529b-11e4-acbb-08002794b041 > TID: [0] [STRATOS] [2014-10-13 11:15:32,609] INFO > {org.apache.stratos.manager.topology.receiver.StratosManagerTopologyEvent= Receiver} > - [MemberTerminatedEventListener] Received: class > org.apache.stratos.messaging.event.topology.MemberTerminatedEvent > TID: [0] [STRATOS] [2014-10-13 11:15:32,609] DEBUG > {org.apache.stratos.manager.topology.model.TopologyClusterInformationMode= l} > - Adding cluster [myphp.php.domain] > TID: [0] [STRATOS] [2014-10-13 11:15:32,609] WARN > {org.apache.stratos.messaging.message.processor.topology.MemberTerminated= MessageProcessor} > - Member already terminated: [service] php [cluster] myphp.php.domain > [member] b446c0dd-529b-11e4-acbb-08002794b041 > TID: [0] [STRATOS] [2014-10-13 11:15:32,609] WARN > {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Membe= r > is in the wrong list and it is removed from active members list > TID: [0] [STRATOS] [2014-10-13 11:15:32,609] INFO > {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Membe= r > stat context has been removed successfully: [member] > b446c0dd-529b-11e4-acbb-08002794b041 > TID: [0] [STRATOS] [2014-10-13 11:15:32,610] DEBUG > {org.apache.stratos.cloud.controller.runtime.FasterLookUpDataHolder} - > MemberContext [id]: b446c0dd-529b-11e4-acbb-08002794b041 removed from > information model. > TID: [0] [STRATOS] [2014-10-13 11:15:32,610] WARN > {org.apache.stratos.messaging.message.processor.topology.MemberTerminated= MessageProcessor} > - Member already terminated: [service] php [cluster] myphp.php.domain > [member] b446c0dd-529b-11e4-acbb-08002794b041 > TID: [0] [STRATOS] [2014-10-13 11:15:32,610] INFO > {org.apache.stratos.cloud.controller.runtime.FasterLookUpDataHolder} - > Scheduled Pod Activation Watcher task canceled. > TID: [0] [STRATOS] [2014-10-13 11:15:32,623] DEBUG > {org.apache.stratos.cloud.controller.topology.TopologyManager} - Write > lock acquired > TID: [0] [STRATOS] [2014-10-13 11:15:32,624] DEBUG > {org.apache.stratos.cloud.controller.topology.TopologyManager} - Updatin= g > topology > TID: [0] [STRATOS] [2014-10-13 11:15:32,627] ERROR > {org.apache.stratos.kubernetes.client.KubernetesApiClient} - Pod > [3ccf4e4e-529a-11e4-acbb-08002794b041] doesn't exist. > TID: [0] [STRATOS] [2014-10-13 11:15:32,627] ERROR > {org.apache.stratos.cloud.controller.util.PodActivationWatcher} - > Container Activation Watcher Failed.. Cause: Pod > [3ccf4e4e-529a-11e4-acbb-08002794b041] doesn't exist. > TID: [0] [STRATOS] [2014-10-13 11:15:32,628] DEBUG > {org.apache.stratos.cloud.controller.util.PodActivationWatcher} - > org.apache.stratos.kubernetes.client.exceptions.KubernetesClientExceptio= n: > Pod [3ccf4e4e-529a-11e4-acbb-08002794b041] doesn't exist. > TID: [0] [STRATOS] [2014-10-13 11:15:32,633] DEBUG > {org.apache.stratos.cloud.controller.util.PodActivationWatcher} - > PodActivationWatcher running : Running > TID: [0] [STRATOS] [2014-10-13 11:15:32,642] DEBUG > {org.apache.stratos.cloud.controller.topology.TopologyManager} - Topolog= y > updated: > {"serviceMap":{"php":{"serviceName":"php","serviceType":"SingleTenant","c= lusterIdClusterMap":{"myphp.php.domain":{"serviceName":"php","clusterId":"m= yphp.php.domain","autoscalePolicyName":"economy","hostNames":[" > myphp.apachestratos.org > "],"isLbCluster":false,"isKubernetesCluster":true,"memberMap":{"b44650cb-= 529b-11e4-acbb-08002794b041":{"serviceName":"php","clusterId":"myphp.php.do= main","memberId":"b44650cb-529b-11e4-acbb-08002794b041","portMap":{"8280":{= "protocol":"http","value":80,"proxy":8280}},"memberPublicIp":"172.17.8.101"= ,"status":"Activated","memberIp":"172.17.8.101","properties":{}},"3ccfb8bd-= 529a-11e4-acbb-08002794b041":{"serviceName":"php","clusterId":"myphp.php.do= main","memberId":"3ccfb8bd-529a-11e4-acbb-08002794b041","portMap":{"8280":{= "protocol":"http","value":80,"proxy":8280}},"memberPublicIp":"172.17.8.101"= ,"status":"Activated","memberIp":"172.17.8.101","properties":{"ALLOCATED_SE= RVICE_HOST_PORT":"4500"}}},"status":"Created","properties":{"payload_parame= ter.APP_PATH":"/var/www","KUBERNETES_REPLICAS_MAX":"20","KUBERNETES_CLUSTER= _ID":"KubGrp1","payload_parameter.LOG_LEVEL":"DEBUG","KUBERNETES_REPLICAS_M= IN":"2","payload_parameter.MB_IP":"172.17.42.1","payload_parameter.CEP_IP":= "172.17.42.1","payload_parameter.MB_PORT":"1883","payload_parameter.CEP_POR= T":"7611"}}},"portMap":{"8280":{"protocol":"http","value":80,"proxy":8280}}= ,"properties":{}}},"initialized":false} > TID: [0] [STRATOS] [2014-10-13 11:15:32,642] DEBUG > {org.apache.stratos.cloud.controller.topology.TopologyManager} - Write > lock released > TID: [0] [STRATOS] [2014-10-13 11:15:32,642] INFO > {org.apache.stratos.cloud.controller.topology.TopologyEventPublisher} - > Publishing member terminated event: [service] php [cluster] > myphp.php.domain [network-partition] null [partition] null [member] > 3ccf4e4e-529a-11e4-acbb-08002794b041 > TID: [0] [STRATOS] [2014-10-13 11:15:32,650] INFO > {org.apache.stratos.cep.extension.CEPTopologyEventReceiver} - Member > [member id] 3ccf4e4e-529a-11e4-acbb-08002794b041 was removed from the tim= e > stamp map. > TID: [0] [STRATOS] [2014-10-13 11:15:32,652] INFO > {org.apache.stratos.messaging.message.processor.topology.MemberTerminated= MessageProcessor} > - Member terminated: [service] php [cluster] myphp.php.domain [member] > 3ccf4e4e-529a-11e4-acbb-08002794b041 > TID: [0] [STRATOS] [2014-10-13 11:15:32,653] INFO > {org.apache.stratos.manager.topology.receiver.StratosManagerTopologyEvent= Receiver} > - [MemberTerminatedEventListener] Received: class > org.apache.stratos.messaging.event.topology.MemberTerminatedEvent > TID: [0] [STRATOS] [2014-10-13 11:15:32,653] DEBUG > {org.apache.stratos.manager.topology.model.TopologyClusterInformationMode= l} > - Adding cluster [myphp.php.domain] > TID: [0] [STRATOS] [2014-10-13 11:15:32,653] WARN > {org.apache.stratos.messaging.message.processor.topology.MemberTerminated= MessageProcessor} > - Member already terminated: [service] php [cluster] myphp.php.domain > [member] 3ccf4e4e-529a-11e4-acbb-08002794b041 > TID: [0] [STRATOS] [2014-10-13 11:15:32,654] WARN > {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Membe= r > is in the wrong list and it is removed from active members list > TID: [0] [STRATOS] [2014-10-13 11:15:32,654] DEBUG > {org.apache.stratos.cloud.controller.runtime.FasterLookUpDataHolder} - > MemberContext [id]: 3ccf4e4e-529a-11e4-acbb-08002794b041 removed from > information model. > TID: [0] [STRATOS] [2014-10-13 11:15:32,654] INFO > {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Membe= r > stat context has been removed successfully: [member] > 3ccf4e4e-529a-11e4-acbb-08002794b041 > TID: [0] [STRATOS] [2014-10-13 11:15:32,655] WARN > {org.apache.stratos.messaging.message.processor.topology.MemberTerminated= MessageProcessor} > - Member already terminated: [service] php [cluster] myphp.php.domain > [member] 3ccf4e4e-529a-11e4-acbb-08002794b041 > TID: [0] [STRATOS] [2014-10-13 11:15:32,654] INFO > {org.apache.stratos.cloud.controller.runtime.FasterLookUpDataHolder} - > Scheduled Pod Activation Watcher task canceled. > TID: [0] [STRATOS] [2014-10-13 11:15:32,696] INFO > {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} - > Kubernetes entities are successfully starting up. [MemberContext > [memberId=3Db4468fb6-529b-11e4-acbb-08002794b041, nodeId=3Dnull, > instanceId=3Dnull, clusterId=3Dmyphp.php.domain, partition=3Dnull, > cartridgeType=3Dphp, privateIpAddress=3D172.17.8.100, > publicIpAddress=3D172.17.8.100, allocatedIpAddress=3Dnull, > initTime=3D1413178952532, lbClusterId=3Dnull, networkPartitionId=3Dnull, > properties=3DProperties [properties=3D[Property > [name=3DALLOCATED_SERVICE_HOST_PORT, value=3Dnull]]]], MemberContext > [memberId=3Db446c0dd-529b-11e4-acbb-08002794b041, nodeId=3Dnull, > instanceId=3Dnull, clusterId=3Dmyphp.php.domain, partition=3Dnull, > cartridgeType=3Dphp, privateIpAddress=3D172.17.8.101, > publicIpAddress=3D172.17.8.101, allocatedIpAddress=3Dnull, > initTime=3D1413178952532, lbClusterId=3Dnull, networkPartitionId=3Dnull, > properties=3DProperties [properties=3D[Property > [name=3DALLOCATED_SERVICE_HOST_PORT, value=3Dnull]]]], MemberContext > [memberId=3D3ccf4e4e-529a-11e4-acbb-08002794b041, nodeId=3Dnull, > instanceId=3Dnull, clusterId=3Dmyphp.php.domain, partition=3Dnull, > cartridgeType=3Dphp, privateIpAddress=3D172.17.8.101, > publicIpAddress=3D172.17.8.101, allocatedIpAddress=3Dnull, > initTime=3D1413178322623, lbClusterId=3Dnull, networkPartitionId=3Dnull, > properties=3DProperties [properties=3D[Property > [name=3DALLOCATED_SERVICE_HOST_PORT, value=3D4500]]]]] > TID: [0] [STRATOS] [2014-10-13 11:15:32,709] DEBUG > {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Scaling down, > terminated the member with id b4468fb6-529b-11e4-acbb-08002794b041 in > cluster myphp.php.domain > TID: [0] [STRATOS] [2014-10-13 11:15:32,709] DEBUG > {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Scaling down, > terminated the member with id b446c0dd-529b-11e4-acbb-08002794b041 in > cluster myphp.php.domain > TID: [0] [STRATOS] [2014-10-13 11:15:32,709] DEBUG > {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Scaling down, > terminated the member with id 3ccf4e4e-529a-11e4-acbb-08002794b041 in > cluster myphp.php.domain > TID: [0] [STRATOS] [2014-10-13 11:15:32,709] DEBUG > {org.apache.stratos.autoscaler.rule.AutoscalerRuleEvaluator} - Scale che= ck > executed for : > org.apache.stratos.autoscaler.KubernetesClusterContext@29d32c84 > TID: [0] [STRATOS] [2014-10-13 11:16:01,002] DEBUG > {org.apache.stratos.manager.publisher.TenantSynzhronizerTask} - Publishi= ng > complete tenant event > TID: [0] [STRATOS] [2014-10-13 11:16:01,006] DEBUG > {org.apache.stratos.cloud.controller.topology.TopologySynchronizerTask} - > Executing topology synchronization task > TID: [0] [STRATOS] [2014-10-13 11:16:01,006] DEBUG > {org.apache.stratos.cloud.controller.topology.TopologyEventPublisher} - > Publishing complete topology event > TID: [0] [STRATOS] [2014-10-13 11:16:22,230] DEBUG > {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad = of > load avg event: [cluster] myphp.php.domain [value] -0.70397747 > TID: [0] [STRATOS] [2014-10-13 11:16:22,232] DEBUG > {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad = of > Memory Consumption event: [cluster] myphp.php.domain [value] -0.46528494 > TID: [0] [STRATOS] [2014-10-13 11:16:22,234] DEBUG > {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg > Memory Consumption event: [cluster] myphp.php.domain [value] 27.538944 > TID: [0] [STRATOS] [2014-10-13 11:16:22,236] DEBUG > {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Secon= d > Derivation of load avg event: [cluster] myphp.php.domain [value] > -0.0012131054 > TID: [0] [STRATOS] [2014-10-13 11:16:22,236] DEBUG > {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Secon= d > Derivation of Memory Consumption event: [cluster] myphp.php.domain [value= ] > 0.021650476 > TID: [0] [STRATOS] [2014-10-13 11:16:22,236] DEBUG > {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory > consumption stats are reset, ready to do scale check [kub cluster] KubGrp= 1 > TID: [0] [STRATOS] [2014-10-13 11:16:22,237] DEBUG > {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg > load avg event: [cluster] myphp.php.domain [value] 53.7 > TID: [0] [STRATOS] [2014-10-13 11:16:22,237] DEBUG > {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average > stats are reset, ready to do scale check [kub cluster] KubGrp1 > TID: [0] [STRATOS] [2014-10-13 11:16:22,425] DEBUG > {org.apache.stratos.autoscaler.message.receiver.health.AutoscalerHealthSt= atEventReceiver} > - Member not found in the Topology: [member] > 3ccf4e4e-529a-11e4-acbb-08002794b041 > TID: [0] [STRATOS] [2014-10-13 11:16:22,426] DEBUG > {org.apache.stratos.autoscaler.message.receiver.health.AutoscalerHealthSt= atEventReceiver} > - Member not found in the Topology: [member] > 3ccf4e4e-529a-11e4-acbb-08002794b041 > TID: [0] [STRATOS] [2014-10-13 11:16:22,426] DEBUG > {org.apache.stratos.autoscaler.message.receiver.health.AutoscalerHealthSt= atEventReceiver} > - Member not found in the Topology: [member] > 3ccf4e4e-529a-11e4-acbb-08002794b041 > TID: [0] [STRATOS] [2014-10-13 11:16:22,427] DEBUG > {org.apache.stratos.autoscaler.message.receiver.health.AutoscalerHealthSt= atEventReceiver} > - Member not found in the Topology: [member] > 3ccf4e4e-529a-11e4-acbb-08002794b041 > TID: [0] [STRATOS] [2014-10-13 11:16:22,427] DEBUG > {org.apache.stratos.autoscaler.message.receiver.health.AutoscalerHealthSt= atEventReceiver} > - Member not found in the Topology: [member] > 3ccf4e4e-529a-11e4-acbb-08002794b041 > TID: [0] [STRATOS] [2014-10-13 11:16:52,406] DEBUG > {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} - > KubernetesServiceClusterMonitor is running.. > KubernetesServiceClusterMonitor [ kubernetesHostClusterId=3DKubGrp1, > clusterId=3Dmyphp.php.domain, serviceId=3Dphp] > TID: [0] [STRATOS] [2014-10-13 11:16:52,407] DEBUG > {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} - > Running min check for [kub-cluster] : KubGrp1 [cluster] : myphp.php.doma= in > TID: [0] [STRATOS] [2014-10-13 11:16:52,408] INFO > {org.apache.stratos.autoscaler.rule.RuleLog} - Running minimum rule: > [kub-cluster] KubGrp1 [cluster] myphp.php.domain > TID: [0] [STRATOS] [2014-10-13 11:16:52,408] INFO > {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] [cluster] : > myphp.php.domain [Replicas] nonTerminated : 2 > TID: [0] [STRATOS] [2014-10-13 11:16:52,409] INFO > {org.apache.stratos.autoscaler.rule.RuleLog} - [min-check] [cluster] : > myphp.php.domain [Replicas] minReplicas : 2 > TID: [0] [STRATOS] [2014-10-13 11:16:52,409] INFO > {org.apache.stratos.autoscaler.rule.RuleLog} - Running obsolete containe= rs > rule [kub-cluster] : KubGrp1 [cluster] : myphp.php.domain > TID: [0] [STRATOS] [2014-10-13 11:16:52,410] INFO > {org.apache.stratos.autoscaler.rule.RuleLog} - [obsolete-check] [cluster= ] > : myphp.php.domain [Replicas] obsoleteReplicas : 0 > TID: [0] [STRATOS] [2014-10-13 11:16:52,411] DEBUG > {org.apache.stratos.autoscaler.rule.AutoscalerRuleEvaluator} - Minimum > check executed for : > org.apache.stratos.autoscaler.KubernetesClusterContext@29d32c84 > TID: [0] [STRATOS] [2014-10-13 11:16:52,411] DEBUG > {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} - > flag of rifReset : false flag of memoryConsumptionReset : true flag of > loadAverageReset : true > TID: [0] [STRATOS] [2014-10-13 11:16:52,412] DEBUG > {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} - > Running scale check for [kub-cluster] : KubGrp1 [cluster] : > myphp.php.domain > TID: [0] [STRATOS] [2014-10-13 11:16:52,412] INFO > {org.apache.stratos.autoscaler.rule.RuleLog} - Running scaling rule > [kub-cluster] : KubGrp1 [cluster] : myphp.php.domain > TID: [0] [STRATOS] [2014-10-13 11:16:52,414] DEBUG > {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the > value, [average]: 0.0 , [gradient]: 0.0 , [second derivative]: 0.0 , [tim= e > intervals]: 1 > TID: [0] [STRATOS] [2014-10-13 11:16:52,415] DEBUG > {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the > value, [average]: 27.538944 , [gradient]: -0.46528494 , [second > derivative]: 0.021650476 , [time intervals]: 1 > TID: [0] [STRATOS] [2014-10-13 11:16:52,416] DEBUG > {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Predicting the > value, [average]: 53.7 , [gradient]: -0.70397747 , [second derivative]: > -0.0012131054 , [time intervals]: 1 > TID: [0] [STRATOS] [2014-10-13 11:16:52,417] INFO > {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : > myphp.php.domain [Replicas] minReplicas : 2 > TID: [0] [STRATOS] [2014-10-13 11:16:52,417] INFO > {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : > myphp.php.domain [Replicas] maxReplicas : 20 > TID: [0] [STRATOS] [2014-10-13 11:16:52,417] INFO > {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : > myphp.php.domain [Replicas] nonTerminated : 2 > TID: [0] [STRATOS] [2014-10-13 11:16:52,417] INFO > {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : > myphp.php.domain [Replicas] activeReplicas : 2 > TID: [0] [STRATOS] [2014-10-13 11:16:52,417] DEBUG > {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : > myphp.php.domain [RequestInFlight] resetted ? : false > TID: [0] [STRATOS] [2014-10-13 11:16:52,417] DEBUG > {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : > myphp.php.domain [RequestInFlight] predicted value : 0.0 > TID: [0] [STRATOS] [2014-10-13 11:16:52,418] DEBUG > {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : > myphp.php.domain [RequestInFlight] upper limit : 80.0 > TID: [0] [STRATOS] [2014-10-13 11:16:52,418] DEBUG > {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : > myphp.php.domain [RequestInFlight] lower limit : 5.0 > TID: [0] [STRATOS] [2014-10-13 11:16:52,418] DEBUG > {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : > myphp.php.domain [MemoryConsumption] resetted ? : true > TID: [0] [STRATOS] [2014-10-13 11:16:52,418] INFO > {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : > myphp.php.domain [MemoryConsumption] predicted value : 27.084485135041177 > TID: [0] [STRATOS] [2014-10-13 11:16:52,418] INFO > {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : > myphp.php.domain [MemoryConsumption] upper limit : 80.0 > TID: [0] [STRATOS] [2014-10-13 11:16:52,419] INFO > {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : > myphp.php.domain [MemoryConsumption] lower limit : 15.0 > TID: [0] [STRATOS] [2014-10-13 11:16:52,419] DEBUG > {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : > myphp.php.domain [LoadAverage] resetted ? : true > TID: [0] [STRATOS] [2014-10-13 11:16:52,419] INFO > {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : > myphp.php.domain [LoadAverage] predicted value : 52.9954185327515 > TID: [0] [STRATOS] [2014-10-13 11:16:52,419] INFO > {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : > myphp.php.domain [LoadAverage] upper limit : 180.0 > TID: [0] [STRATOS] [2014-10-13 11:16:52,420] INFO > {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : > myphp.php.domain [LoadAverage] lower limit : 20.0 > TID: [0] [STRATOS] [2014-10-13 11:16:52,420] INFO > {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : > myphp.php.domain scale-up action : false > TID: [0] [STRATOS] [2014-10-13 11:16:52,420] INFO > {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : > myphp.php.domain scale-down action : true > TID: [0] [STRATOS] [2014-10-13 11:16:52,421] INFO > {org.apache.stratos.autoscaler.rule.RuleLog} - [scaling] [cluster] : > myphp.php.domain Even if all predicted stats < upperLimit, still > activeReplicas <=3D minReplicas ; hence not scaling down > TID: [0] [STRATOS] [2014-10-13 11:16:52,421] DEBUG > {org.apache.stratos.autoscaler.rule.AutoscalerRuleEvaluator} - Scale che= ck > executed for : > org.apache.stratos.autoscaler.KubernetesClusterContext@29d32c84 > TID: [0] [STRATOS] [2014-10-13 11:17:01,001] DEBUG > {org.apache.stratos.manager.publisher.TenantSynzhronizerTask} - Publishi= ng > complete tenant event > TID: [0] [STRATOS] [2014-10-13 11:17:01,002] DEBUG > {org.apache.stratos.cloud.controller.topology.TopologySynchronizerTask} - > Executing topology synchronization task > TID: [0] [STRATOS] [2014-10-13 11:17:01,002] DEBUG > {org.apache.stratos.cloud.controller.topology.TopologyEventPublisher} - > Publishing complete topology event > TID: [0] [STRATOS] [2014-10-13 11:17:16,141] INFO > {org.apache.stratos.cep.extension.FaultHandlingWindowProcessor} - Faulty > member detected [member-id] 3ccf4e4e-529a-11e4-acbb-08002794b041 with [la= st > time-stamp] 1413179137678 [time-out] 60000 milliseconds > TID: [0] [STRATOS] [2014-10-13 11:17:16,141] ERROR > {org.apache.stratos.cep.extension.FaultHandlingWindowProcessor} - Failed > to publish member fault event. Member having [member-id] > 3ccf4e4e-529a-11e4-acbb-08002794b041 does not exist in topology > TID: [0] [STRATOS] [2014-10-13 11:17:22,231] DEBUG > {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg > load avg event: [cluster] myphp.php.domain [value] 23.75 > TID: [0] [STRATOS] [2014-10-13 11:17:22,232] DEBUG > {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad = of > load avg event: [cluster] myphp.php.domain [value] -0.2682943 > TID: [0] [STRATOS] [2014-10-13 11:17:22,236] DEBUG > {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Avg > Memory Consumption event: [cluster] myphp.php.domain [value] 17.602432 > TID: [0] [STRATOS] [2014-10-13 11:17:22,237] DEBUG > {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Secon= d > Derivation of load avg event: [cluster] myphp.php.domain [value] > -0.0045045386 > TID: [0] [STRATOS] [2014-10-13 11:17:22,237] DEBUG > {org.apache.stratos.autoscaler.KubernetesClusterContext} - Load average > stats are reset, ready to do scale check [kub cluster] KubGrp1 > TID: [0] [STRATOS] [2014-10-13 11:17:22,237] DEBUG > {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Grad = of > Memory Consumption event: [cluster] myphp.php.domain [value] 0.0016795773 > TID: [0] [STRATOS] [2014-10-13 11:17:22,238] DEBUG > {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - Secon= d > Derivation of Memory Consumption event: [cluster] myphp.php.domain [value= ] > 2.3589536E-4 > TID: [0] [STRATOS] [2014-10-13 11:17:22,238] DEBUG > {org.apache.stratos.autoscaler.KubernetesClusterContext} - Memory > consumption stats are reset, ready to do scale check [kub cluster] KubGrp= 1 > TID: [0] [STRATOS] [2014-10-13 11:18:01,000] DEBUG > {org.apache.stratos.manager.publisher.TenantSynzhronizerTask} - Publishi= ng > complete tenant event > TID: [0] [STRATOS] [2014-10-13 11:18:01,001] DEBUG > {org.apache.stratos.cloud.controller.topology.TopologySynchronizerTask} - > Executing topology synchronization task > TID: [0] [STRATOS] [2014-10-13 11:18:01,001] DEBUG > {org.apache.stratos.cloud.controller.topology.TopologyEventPublisher} - > Publishing complete topology event > TID: [0] [STRATOS] [2014-10-13 11:18:16,142] INFO > {org.apache.stratos.cep.extension.FaultHandlingWindowProcessor} - Faulty > member detected [member-id] 3ccf4e4e-529a-11e4-acbb-08002794b041 with [la= st > time-stamp] 1413179137678 [time-out] 60000 milliseconds > TID: [0] [STRATOS] [2014-10-13 11:18:16,144] ERROR > {org.apache.stratos.cep.extension.FaultHandlingWindowProcessor} - Failed > to publish member fault event. Member having [member-id] > 3ccf4e4e-529a-11e4-acbb-08002794b041 does not exist in topology > > Thanks. > > -- > Rajkumar Rajaratnam > Software Engineer | WSO2, Inc. > Mobile +94777568639 | +94783498120 > --=20 Regards, Manula Chathurika Thantriwatte Software Engineer WSO2 Inc. : http://wso2.com lean . enterprise . middleware email : manulac@wso2.com / manula@apache.org phone : +94 772492511 blog : http://manulachathurika.blogspot.com/ --089e0112ce22d0836c05054a6d74 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hi Raj,

It seems like [1] get executed = again.=C2=A0memberTimeStampMap is a concurrent hash map. Seems that=C2=A0CE= PTopologyEventReceiver unable to remove the element in the memberTimeStampM= ap.


Thanks !

On Mon, Oct 13, 2014 at 11:48 = AM, Rajkumar Rajaratnam <rajkumarr@wso2.com> wrote:
Hi,

CEP exte= nsion=C2=A0FaultHandlingWindowProcessor removing a terminated member from t= he map. But after a while, it is complaining it as a faulty member. Here is= the log. MemberId highligh= ted.

TID: [0] [STRATOS] [2014-10-13 11= :15:22,409] DEBUG {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} -= =C2=A0Predicting the value, [average]: 0.0 , [gradient]: 0.0 , [second der= ivative]: 0.0 , [time intervals]: 1=C2=A0
TID: [0] [STRATOS] [201= 4-10-13 11:15:22,410] DEBUG {org.apache.stratos.autoscaler.rule.RuleTasksDe= legator} - =C2=A0Predicting the value, [average]: 41.63388 , [gradient]: -0= .013520682 , [second derivative]: 0.0013238431 , [time intervals]: 1=C2=A0<= /div>
TID: [0] [STRATOS] [2014-10-13 11:15:22,410] DEBUG {org.apache.st= ratos.autoscaler.rule.RuleTasksDelegator} - =C2=A0Predicting the value, [av= erage]: 99.9 , [gradient]: -2.0885546 , [second derivative]: 0.03460533 , [= time intervals]: 1=C2=A0
TID: [0] [STRATOS] [2014-10-13 11:15:22,= 411] =C2=A0INFO {org.apache.stratos.autoscaler.rule.RuleLog} - =C2=A0[scali= ng] =C2=A0[cluster] : myphp.php.domain [Replicas] minReplicas : 2
TID: [0] [STRATOS] [2014-10-13 11:15:22,411] =C2=A0INFO {org.apache.strato= s.autoscaler.rule.RuleLog} - =C2=A0[scaling] =C2=A0[cluster] : myphp.php.do= main [Replicas] maxReplicas : 20
TID: [0] [STRATOS] [2014-10-13 1= 1:15:22,411] =C2=A0INFO {org.apache.stratos.autoscaler.rule.RuleLog} - =C2= =A0[scaling] =C2=A0[cluster] : myphp.php.domain [Replicas] nonTerminated : = 5
TID: [0] [STRATOS] [2014-10-13 11:15:22,411] =C2=A0INFO {org.ap= ache.stratos.autoscaler.rule.RuleLog} - =C2=A0[scaling] =C2=A0[cluster] : m= yphp.php.domain [Replicas] activeReplicas : 5
TID: [0] [STRATOS] = [2014-10-13 11:15:22,412] DEBUG {org.apache.stratos.autoscaler.rule.RuleLog= } - =C2=A0[scaling] =C2=A0[cluster] : myphp.php.domain [RequestInFlight] re= setted ? : false
TID: [0] [STRATOS] [2014-10-13 11:15:22,412] DEB= UG {org.apache.stratos.autoscaler.rule.RuleLog} - =C2=A0[scaling] =C2=A0[cl= uster] : myphp.php.domain [RequestInFlight] predicted value : 0.0
TID: [0] [STRATOS] [2014-10-13 11:15:22,412] DEBUG {org.apache.stratos.aut= oscaler.rule.RuleLog} - =C2=A0[scaling] =C2=A0[cluster] : myphp.php.domain = [RequestInFlight] upper limit : 80.0
TID: [0] [STRATOS] [2014-10-= 13 11:15:22,412] DEBUG {org.apache.stratos.autoscaler.rule.RuleLog} - =C2= =A0[scaling] =C2=A0[cluster] : myphp.php.domain [RequestInFlight] lower lim= it : 5.0
TID: [0] [STRATOS] [2014-10-13 11:15:22,412] DEBUG {org.= apache.stratos.autoscaler.rule.RuleLog} - =C2=A0[scaling] =C2=A0[cluster] := myphp.php.domain [MemoryConsumption] resetted ? : true
TID: [0] = [STRATOS] [2014-10-13 11:15:22,413] =C2=A0INFO {org.apache.stratos.autoscal= er.rule.RuleLog} - =C2=A0[scaling] =C2=A0[cluster] : myphp.php.domain [Memo= ryConsumption] predicted value : 41.62102324969601
TID: [0] [STRA= TOS] [2014-10-13 11:15:22,413] =C2=A0INFO {org.apache.stratos.autoscaler.ru= le.RuleLog} - =C2=A0[scaling] =C2=A0[cluster] : myphp.php.domain [MemoryCon= sumption] upper limit : 80.0
TID: [0] [STRATOS] [2014-10-13 11:15= :22,413] =C2=A0INFO {org.apache.stratos.autoscaler.rule.RuleLog} - =C2=A0[s= caling] =C2=A0[cluster] : myphp.php.domain [MemoryConsumption] lower limit = : 15.0
TID: [0] [STRATOS] [2014-10-13 11:15:22,415] DEBUG {org.ap= ache.stratos.autoscaler.rule.RuleLog} - =C2=A0[scaling] =C2=A0[cluster] : m= yphp.php.domain [LoadAverage] resetted ? : true
TID: [0] [STRATOS= ] [2014-10-13 11:15:22,415] =C2=A0INFO {org.apache.stratos.autoscaler.rule.= RuleLog} - =C2=A0[scaling] =C2=A0[cluster] : myphp.php.domain [LoadAverage]= predicted value : 97.82874980941415
TID: [0] [STRATOS] [2014-10-= 13 11:15:22,415] =C2=A0INFO {org.apache.stratos.autoscaler.rule.RuleLog} - = =C2=A0[scaling] =C2=A0[cluster] : myphp.php.domain [LoadAverage] upper limi= t : 180.0
TID: [0] [STRATOS] [2014-10-13 11:15:22,415] =C2=A0INFO= {org.apache.stratos.autoscaler.rule.RuleLog} - =C2=A0[scaling] =C2=A0[clus= ter] : myphp.php.domain [LoadAverage] lower limit : 20.0
TID: [0]= [STRATOS] [2014-10-13 11:15:22,416] =C2=A0INFO {org.apache.stratos.autosca= ler.rule.RuleLog} - =C2=A0[scaling] =C2=A0[cluster] : myphp.php.domain scal= e-up action : false
TID: [0] [STRATOS] [2014-10-13 11:15:22,416] = =C2=A0INFO {org.apache.stratos.autoscaler.rule.RuleLog} - =C2=A0[scaling] = =C2=A0[cluster] : myphp.php.domain scale-down action : true
TID: = [0] [STRATOS] [2014-10-13 11:15:22,416] =C2=A0INFO {org.apache.stratos.auto= scaler.rule.RuleLog} - =C2=A0[scaling] Decided to scale-down [cluster] : my= php.php.domain
TID: [0] [STRATOS] [2014-10-13 11:15:22,417] =C2= =A0INFO {org.apache.stratos.autoscaler.rule.RuleLog} - =C2=A0[scaling-down]= =C2=A0[cluster] : myphp.php.domain shrink the cluster to minReplicas : 2
TID: [0] [STRATOS] [2014-10-13 11:15:22,417] =C2=A0INFO {org.apach= e.stratos.autoscaler.client.cloud.controller.CloudControllerClient} - =C2= =A0Updating kubernetes replication controller via cloud controller: [cluste= r] myphp.php.domain [replicas] 2
TID: [0] [STRATOS] [2014-10-13 1= 1:15:22,425] DEBUG {org.apache.stratos.cloud.controller.impl.CloudControlle= rServiceImpl} - =C2=A0CloudControllerServiceImpl:updateContainers for clust= er : myphp.php.domain
TID: [0] [STRATOS] [2014-10-13 11:15:22,448= ] DEBUG {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImp= l} - =C2=A0Cloud Controller is delegating request to update a replication c= ontroller myphp.php.domain to Kubernetes layer.
TID: [0] [STRATOS= ] [2014-10-13 11:15:22,506] DEBUG {org.apache.stratos.cloud.controller.impl= .CloudControllerServiceImpl} - =C2=A0Cloud Controller successfully updated = the controller myphp.php.domain via Kubernetes layer.
TID: [0] [S= TRATOS] [2014-10-13 11:15:22,531] DEBUG {org.apache.stratos.cloud.controlle= r.impl.CloudControllerServiceImpl} - =C2=A0Pods Count: 5 for cluster: myphp= .php.domain
TID: [0] [STRATOS] [2014-10-13 11:15:22,548] DEBUG {o= rg.apache.stratos.cloud.controller.util.PodActivationWatcher} - =C2=A0PodAc= tivationWatcher running : Running
TID: [0] [STRATOS] [2014-10-13 = 11:15:22,561] DEBUG {org.apache.stratos.cloud.controller.util.PodActivation= Watcher} - =C2=A0PodActivationWatcher running : Running
TID: [0] = [STRATOS] [2014-10-13 11:15:22,563] DEBUG {org.apache.stratos.cloud.control= ler.util.PodActivationWatcher} - =C2=A0PodActivationWatcher running : Runni= ng
TID: [0] [STRATOS] [2014-10-13 11:15:22,655] DEBUG {org.apache= .stratos.cloud.controller.util.PodActivationWatcher} - =C2=A0PodActivationW= atcher running : Running
TID: [0] [STRATOS] [2014-10-13 11:15:22,= 657] DEBUG {org.apache.stratos.cloud.controller.util.PodActivationWatcher} = - =C2=A0PodActivationWatcher running : Running
TID: [0] [STRATOS]= [2014-10-13 11:15:27,538] DEBUG {org.apache.stratos.cloud.controller.util.= PodActivationWatcher} - =C2=A0PodActivationWatcher running : Running
<= div>TID: [0] [STRATOS] [2014-10-13 11:15:27,539] ERROR {org.apache.stratos.= kubernetes.client.KubernetesApiClient} - =C2=A0Pod [b446c0dd-529b-11e4-acbb= -08002794b041] doesn't exist.
TID: [0] [STRATOS] [2014-10-13 = 11:15:27,539] ERROR {org.apache.stratos.cloud.controller.util.PodActivation= Watcher} - =C2=A0Container Activation Watcher Failed.. Cause: Pod [b446c0dd= -529b-11e4-acbb-08002794b041] doesn't exist.
TID: [0] [STRATO= S] [2014-10-13 11:15:27,539] DEBUG {org.apache.stratos.cloud.controller.uti= l.PodActivationWatcher} - =C2=A0org.apache.stratos.kubernetes.client.except= ions.KubernetesClientException: Pod [b446c0dd-529b-11e4-acbb-08002794b041] = doesn't exist.
TID: [0] [STRATOS] [2014-10-13 11:15:27,539] E= RROR {org.apache.stratos.kubernetes.client.KubernetesApiClient} - =C2=A0Pod= [b4468fb6-529b-11e4-acbb-08002794b041] doesn't exist.
TID: [= 0] [STRATOS] [2014-10-13 11:15:27,540] ERROR {org.apache.stratos.cloud.cont= roller.util.PodActivationWatcher} - =C2=A0Container Activation Watcher Fail= ed.. Cause: Pod [b4468fb6-529b-11e4-acbb-08002794b041] doesn't exist.
TID: [0] [STRATOS] [2014-10-13 11:15:27,540] DEBUG {org.apache.str= atos.cloud.controller.util.PodActivationWatcher} - =C2=A0org.apache.stratos= .kubernetes.client.exceptions.KubernetesClientException: Pod [b4468fb6-529b= -11e4-acbb-08002794b041] doesn't exist.
TID: [0] [STRATOS] [2= 014-10-13 11:15:27,628] ERROR {org.apache.stratos.kubernetes.client.Kuberne= tesApiClient} - =C2=A0Pod [3ccf4e4e-529a-11e4-acbb-08002794b041] doesn'= t exist.
TID: [0] [STRATOS] [2014-10-13 11:15:27,628] ERROR {org.= apache.stratos.cloud.controller.util.PodActivationWatcher} - =C2=A0Containe= r Activation Watcher Failed.. Cause: Pod [3ccf4e4e-529a-11e4-acbb-08002794b= 041] doesn't exist.
TID: [0] [STRATOS] [2014-10-13 11:15:27,6= 28] DEBUG {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -= =C2=A0org.apache.stratos.kubernetes.client.exceptions.KubernetesClientExce= ption: Pod [3ccf4e4e-529a-11e4-acbb-08002794b041] doesn't exist.
<= div>TID: [0] [STRATOS] [2014-10-13 11:15:27,630] DEBUG {org.apache.stratos.= cloud.controller.util.PodActivationWatcher} - =C2=A0PodActivationWatcher ru= nning : Running
TID: [0] [STRATOS] [2014-10-13 11:15:32,542] ERRO= R {org.apache.stratos.kubernetes.client.KubernetesApiClient} - =C2=A0Pod [b= 4468fb6-529b-11e4-acbb-08002794b041] doesn't exist.
TID: [0] = [STRATOS] [2014-10-13 11:15:32,542] ERROR {org.apache.stratos.cloud.control= ler.util.PodActivationWatcher} - =C2=A0Container Activation Watcher Failed.= . Cause: Pod [b4468fb6-529b-11e4-acbb-08002794b041] doesn't exist.
TID: [0] [STRATOS] [2014-10-13 11:15:32,542] DEBUG {org.apache.strato= s.cloud.controller.util.PodActivationWatcher} - =C2=A0org.apache.stratos.ku= bernetes.client.exceptions.KubernetesClientException: Pod [b4468fb6-529b-11= e4-acbb-08002794b041] doesn't exist.
TID: [0] [STRATOS] [2014= -10-13 11:15:32,549] DEBUG {org.apache.stratos.cloud.controller.impl.CloudC= ontrollerServiceImpl} - =C2=A0Pods Count: 2 for cluster: myphp.php.domain
TID: [0] [STRATOS] [2014-10-13 11:15:32,549] DEBUG {org.apache.str= atos.cloud.controller.impl.CloudControllerServiceImpl} - =C2=A0Pods created= : 2 for cluster : myphp.php.domain
TID: [0] [STRATOS] [2014-10-1= 3 11:15:32,551] DEBUG {org.apache.stratos.cloud.controller.topology.Topolog= yManager} - =C2=A0Write lock acquired
TID: [0] [STRATOS] [2014-10= -13 11:15:32,551] DEBUG {org.apache.stratos.cloud.controller.topology.Topol= ogyManager} - =C2=A0Updating topology
TID: [0] [STRATOS] [2014-10= -13 11:15:32,553] DEBUG {org.apache.stratos.cloud.controller.util.PodActiva= tionWatcher} - =C2=A0PodActivationWatcher running : Running
TID: = [0] [STRATOS] [2014-10-13 11:15:32,554] ERROR {org.apache.stratos.kubernete= s.client.KubernetesApiClient} - =C2=A0Pod [b446c0dd-529b-11e4-acbb-08002794= b041] doesn't exist.
TID: [0] [STRATOS] [2014-10-13 11:15:32,= 554] ERROR {org.apache.stratos.cloud.controller.util.PodActivationWatcher} = - =C2=A0Container Activation Watcher Failed.. Cause: Pod [b446c0dd-529b-11e= 4-acbb-08002794b041] doesn't exist.
TID: [0] [STRATOS] [2014-= 10-13 11:15:32,555] DEBUG {org.apache.stratos.cloud.controller.util.PodActi= vationWatcher} - =C2=A0org.apache.stratos.kubernetes.client.exceptions.Kube= rnetesClientException: Pod [b446c0dd-529b-11e4-acbb-08002794b041] doesn'= ;t exist.
TID: [0] [STRATOS] [2014-10-13 11:15:32,564] DEBUG {org= .apache.stratos.cloud.controller.topology.TopologyManager} - =C2=A0Topology= updated: {"serviceMap":{"php":{"serviceName"= :"php","serviceType":"SingleTenant","clu= sterIdClusterMap":{"myphp.php.domain":{"serviceName&quo= t;:"php","clusterId":"myphp.php.domain","= ;autoscalePolicyName":"economy","hostNames":["= ;myphp.apaches= tratos.org"],"isLbCluster":false,"isKubernetesClust= er":true,"memberMap":{"3ccf4e4e-529a-11e4-acbb-08002794= b041":{"serviceName":"php","clusterId":&= quot;myphp.php.domain","memberId":"3ccf4e4e-529a-11e4-a= cbb-08002794b041","portMap":{"8280":{"protoco= l":"http","value":80,"proxy":8280}},&quo= t;memberPublicIp":"172.17.8.101","status":"Ac= tivated","memberIp":"172.17.8.101","propertie= s":{"ALLOCATED_SERVICE_HOST_PORT":"4500"}},"b= 446c0dd-529b-11e4-acbb-08002794b041":{"serviceName":"ph= p","clusterId":"myphp.php.domain","memberId&q= uot;:"b446c0dd-529b-11e4-acbb-08002794b041","portMap":{= "8280":{"protocol":"http","value":8= 0,"proxy":8280}},"memberPublicIp":"172.17.8.101&qu= ot;,"status":"Activated","memberIp":"172= .17.8.101","properties":{}},"b44650cb-529b-11e4-acbb-08= 002794b041":{"serviceName":"php","clusterId&q= uot;:"myphp.php.domain","memberId":"b44650cb-529b-= 11e4-acbb-08002794b041","portMap":{"8280":{"p= rotocol":"http","value":80,"proxy":8280}= },"memberPublicIp":"172.17.8.101","status":&q= uot;Activated","memberIp":"172.17.8.101","pro= perties":{}},"3ccfb8bd-529a-11e4-acbb-08002794b041":{"s= erviceName":"php","clusterId":"myphp.php.doma= in","memberId":"3ccfb8bd-529a-11e4-acbb-08002794b041&qu= ot;,"portMap":{"8280":{"protocol":"http&= quot;,"value":80,"proxy":8280}},"memberPublicIp&qu= ot;:"172.17.8.101","status":"Activated","= ;memberIp":"172.17.8.101","properties":{"ALLO= CATED_SERVICE_HOST_PORT":"4500"}}},"status":"= Created","properties":{"payload_parameter.APP_PATH"= ;:"/var/www","KUBERNETES_REPLICAS_MAX":"20",&= quot;KUBERNETES_CLUSTER_ID":"KubGrp1","payload_paramete= r.LOG_LEVEL":"DEBUG","KUBERNETES_REPLICAS_MIN":&qu= ot;2","payload_parameter.MB_IP":"172.17.42.1",&quo= t;payload_parameter.CEP_IP":"172.17.42.1","payload_para= meter.MB_PORT":"1883","payload_parameter.CEP_PORT"= :"7611"}}},"portMap":{"8280":{"protocol&= quot;:"http","value":80,"proxy":8280}},"= properties":{}}},"initialized":false}
TID: [0] [ST= RATOS] [2014-10-13 11:15:32,564] DEBUG {org.apache.stratos.cloud.controller= .topology.TopologyManager} - =C2=A0Write lock released
TID: [0] [= STRATOS] [2014-10-13 11:15:32,564] =C2=A0INFO {org.apache.stratos.cloud.con= troller.topology.TopologyEventPublisher} - =C2=A0Pu= blishing member terminated event: [service] php [cluster] myphp.php.domain = [network-partition] null [partition] null [member] b4468fb6-529b-11e4-acbb-= 08002794b041
TID: [0] [STRATOS] [2014-10-13 11:15:32,575] = =C2=A0INFO {org.apache.stratos.manager.topology.receiver.StratosManagerTopo= logyEventReceiver} - =C2=A0[MemberTerminatedEventListener] Received: class = org.apache.stratos.messaging.event.topology.MemberTerminatedEvent
TID: [0] [STRATOS] [2014-10-13 11:15:32,580] DEBUG {org.apache.stratos.man= ager.topology.receiver.StratosManagerTopologyEventReceiver} - =C2=A0Removed= the terminated member with id b4468fb6-529b-11e4-acbb-08002794b041 from th= e cluster
TID: [0] [STRATOS] [2014-10-13 11:15:32,580] DEBUG {org= .apache.stratos.manager.topology.model.TopologyClusterInformationModel} - = =C2=A0 Adding cluster [myphp.php.domain]=C2=A0
TID: [0] [STRATOS]= [2014-10-13 11:15:32,580] =C2=A0INFO {org.apache.stratos.messaging.message= .processor.topology.MemberTerminatedMessageProcessor} - =C2=A0Member termin= ated: [service] php [cluster] myphp.php.domain [member] b4468fb6-529b-11e4-= acbb-08002794b041
TID: [0] [STRATOS] [2014-10-13 11:15:32,580] DE= BUG {org.apache.stratos.cloud.controller.runtime.FasterLookUpDataHolder} - = =C2=A0MemberContext [id]: b4468fb6-529b-11e4-acbb-08002794b041 removed from= information model.
TID: [0] [STRATOS] [2014-10-13 11:15:32,581] = =C2=A0INFO {org.apache.stratos.cloud.controller.runtime.FasterLookUpDataHol= der} - =C2=A0Scheduled Pod Activation Watcher task canceled.
TID:= [0] [STRATOS] [2014-10-13 11:15:32,581] =C2=A0WARN {org.apache.stratos.aut= oscaler.monitor.KubernetesClusterMonitor} - =C2=A0Member is in the wrong li= st and it is removed from active members list
TID: [0] [STRATOS] = [2014-10-13 11:15:32,582] =C2=A0INFO {org.apache.stratos.autoscaler.monitor= .KubernetesClusterMonitor} - =C2=A0Member stat context has been removed suc= cessfully: [member] b4468fb6-529b-11e4-acbb-08002794b041
TID: [0]= [STRATOS] [2014-10-13 11:15:32,582] =C2=A0WARN {org.apache.stratos.messagi= ng.message.processor.topology.MemberTerminatedMessageProcessor} - =C2=A0Mem= ber already terminated: [service] php [cluster] myphp.php.domain [member] b= 4468fb6-529b-11e4-acbb-08002794b041
TID: [0] [STRATOS] [2014-10-1= 3 11:15:32,583] =C2=A0INFO {org.apache.stratos.cep.extension.CEPTopologyEve= ntReceiver} - =C2=A0Member [member id] b4468fb6-529b-11e4-acbb-08002794b041= was removed from the time stamp map.
TID: [0] [STRATOS] [2014-10= -13 11:15:32,583] =C2=A0WARN {org.apache.stratos.messaging.message.processo= r.topology.MemberTerminatedMessageProcessor} - =C2=A0Member already termina= ted: [service] php [cluster] myphp.php.domain [member] b4468fb6-529b-11e4-a= cbb-08002794b041
TID: [0] [STRATOS] [2014-10-13 11:15:32,593] DEB= UG {org.apache.stratos.cloud.controller.topology.TopologyManager} - =C2=A0W= rite lock acquired
TID: [0] [STRATOS] [2014-10-13 11:15:32,593] D= EBUG {org.apache.stratos.cloud.controller.topology.TopologyManager} - =C2= =A0Updating topology
TID: [0] [STRATOS] [2014-10-13 11:15:32,603]= DEBUG {org.apache.stratos.cloud.controller.topology.TopologyManager} - =C2= =A0Topology updated: {"serviceMap":{"php":{"servic= eName":"php","serviceType":"SingleTenant"= ;,"clusterIdClusterMap":{"myphp.php.domain":{"serv= iceName":"php","clusterId":"myphp.php.domain&= quot;,"autoscalePolicyName":"economy","hostNames&q= uot;:["my= php.apachestratos.org"],"isLbCluster":false,"isKube= rnetesCluster":true,"memberMap":{"3ccf4e4e-529a-11e4-ac= bb-08002794b041":{"serviceName":"php","cluste= rId":"myphp.php.domain","memberId":"3ccf4e4e-= 529a-11e4-acbb-08002794b041","portMap":{"8280":{&q= uot;protocol":"http","value":80,"proxy":= 8280}},"memberPublicIp":"172.17.8.101","status&quo= t;:"Activated","memberIp":"172.17.8.101",&quo= t;properties":{"ALLOCATED_SERVICE_HOST_PORT":"4500"= ;}},"b44650cb-529b-11e4-acbb-08002794b041":{"serviceName&quo= t;:"php","clusterId":"myphp.php.domain","= ;memberId":"b44650cb-529b-11e4-acbb-08002794b041","port= Map":{"8280":{"protocol":"http","va= lue":80,"proxy":8280}},"memberPublicIp":"172.= 17.8.101","status":"Activated","memberIp"= ;:"172.17.8.101","properties":{}},"3ccfb8bd-529a-1= 1e4-acbb-08002794b041":{"serviceName":"php","= clusterId":"myphp.php.domain","memberId":"3cc= fb8bd-529a-11e4-acbb-08002794b041","portMap":{"8280&quo= t;:{"protocol":"http","value":80,"proxy&= quot;:8280}},"memberPublicIp":"172.17.8.101","stat= us":"Activated","memberIp":"172.17.8.101"= ;,"properties":{"ALLOCATED_SERVICE_HOST_PORT":"450= 0"}}},"status":"Created","properties":{&= quot;payload_parameter.APP_PATH":"/var/www","KUBERNETES= _REPLICAS_MAX":"20","KUBERNETES_CLUSTER_ID":"= KubGrp1","payload_parameter.LOG_LEVEL":"DEBUG",&qu= ot;KUBERNETES_REPLICAS_MIN":"2","payload_parameter.MB_I= P":"172.17.42.1","payload_parameter.CEP_IP":"= 172.17.42.1","payload_parameter.MB_PORT":"1883",&q= uot;payload_parameter.CEP_PORT":"7611"}}},"portMap"= ;:{"8280":{"protocol":"http","value"= ;:80,"proxy":8280}},"properties":{}}},"initialized= ":false}
TID: [0] [STRATOS] [2014-10-13 11:15:32,603] DEBUG = {org.apache.stratos.cloud.controller.topology.TopologyManager} - =C2=A0Writ= e lock released
TID: [0] [STRATOS] [2014-10-13 11:15:32,603] =C2= =A0INFO {org.apache.stratos.cloud.controller.topology.TopologyEventPublishe= r} - =C2=A0Publishing member terminated event: [service] php [cluster] myph= p.php.domain [network-partition] null [partition] null [member] b446c0dd-52= 9b-11e4-acbb-08002794b041
TID: [0] [STRATOS] [2014-10-13 11:15:32= ,607] =C2=A0INFO {org.apache.stratos.cep.extension.CEPTopologyEventReceiver= } - =C2=A0Member [member id] b446c0dd-529b-11e4-acbb-08002794b041 was remov= ed from the time stamp map.
TID: [0] [STRATOS] [2014-10-13 11:15:= 32,608] =C2=A0INFO {org.apache.stratos.messaging.message.processor.topology= .MemberTerminatedMessageProcessor} - =C2=A0Member terminated: [service] php= [cluster] myphp.php.domain [member] b446c0dd-529b-11e4-acbb-08002794b041
TID: [0] [STRATOS] [2014-10-13 11:15:32,609] =C2=A0INFO {org.apach= e.stratos.manager.topology.receiver.StratosManagerTopologyEventReceiver} - = =C2=A0[MemberTerminatedEventListener] Received: class org.apache.stratos.me= ssaging.event.topology.MemberTerminatedEvent
TID: [0] [STRATOS] [= 2014-10-13 11:15:32,609] DEBUG {org.apache.stratos.manager.topology.model.T= opologyClusterInformationModel} - =C2=A0 Adding cluster [myphp.php.domain]= =C2=A0
TID: [0] [STRATOS] [2014-10-13 11:15:32,609] =C2=A0WARN {o= rg.apache.stratos.messaging.message.processor.topology.MemberTerminatedMess= ageProcessor} - =C2=A0Member already terminated: [service] php [cluster] my= php.php.domain [member] b446c0dd-529b-11e4-acbb-08002794b041
TID:= [0] [STRATOS] [2014-10-13 11:15:32,609] =C2=A0WARN {org.apache.stratos.aut= oscaler.monitor.KubernetesClusterMonitor} - =C2=A0Member is in the wrong li= st and it is removed from active members list
TID: [0] [STRATOS] = [2014-10-13 11:15:32,609] =C2=A0INFO {org.apache.stratos.autoscaler.monitor= .KubernetesClusterMonitor} - =C2=A0Member stat context has been removed suc= cessfully: [member] b446c0dd-529b-11e4-acbb-08002794b041
TID: [0]= [STRATOS] [2014-10-13 11:15:32,610] DEBUG {org.apache.stratos.cloud.contro= ller.runtime.FasterLookUpDataHolder} - =C2=A0MemberContext [id]: b446c0dd-5= 29b-11e4-acbb-08002794b041 removed from information model.
TID: [= 0] [STRATOS] [2014-10-13 11:15:32,610] =C2=A0WARN {org.apache.stratos.messa= ging.message.processor.topology.MemberTerminatedMessageProcessor} - =C2=A0M= ember already terminated: [service] php [cluster] myphp.php.domain [member]= b446c0dd-529b-11e4-acbb-08002794b041
TID: [0] [STRATOS] [2014-10= -13 11:15:32,610] =C2=A0INFO {org.apache.stratos.cloud.controller.runtime.F= asterLookUpDataHolder} - =C2=A0Scheduled Pod Activation Watcher task cancel= ed.
TID: [0] [STRATOS] [2014-10-13 11:15:32,623] DEBUG {org.apach= e.stratos.cloud.controller.topology.TopologyManager} - =C2=A0Write lock acq= uired
TID: [0] [STRATOS] [2014-10-13 11:15:32,624] DEBUG {org.apa= che.stratos.cloud.controller.topology.TopologyManager} - =C2=A0Updating top= ology
TID: [0] [STRATOS] [2014-10-13 11:15:32,627] ERROR {org.apa= che.stratos.kubernetes.client.KubernetesApiClient} - =C2=A0Pod [3ccf4e4e-52= 9a-11e4-acbb-08002794b041] doesn't exist.
TID: [0] [STRATOS] = [2014-10-13 11:15:32,627] ERROR {org.apache.stratos.cloud.controller.util.P= odActivationWatcher} - =C2=A0Container Activation Watcher Failed.. Cause: P= od [3ccf4e4e-529a-11e4-acbb-08002794b041] doesn't exist.
TID:= [0] [STRATOS] [2014-10-13 11:15:32,628] DEBUG {org.apache.stratos.cloud.co= ntroller.util.PodActivationWatcher} - =C2=A0org.apache.stratos.kubernetes.c= lient.exceptions.KubernetesClientException: Pod [3ccf4e4e-529a-11e4-acbb-08= 002794b041] doesn't exist.
TID: [0] [STRATOS] [2014-10-13 11:= 15:32,633] DEBUG {org.apache.stratos.cloud.controller.util.PodActivationWat= cher} - =C2=A0PodActivationWatcher running : Running
TID: [0] [ST= RATOS] [2014-10-13 11:15:32,642] DEBUG {org.apache.stratos.cloud.controller= .topology.TopologyManager} - =C2=A0Topology updated: {"serviceMap"= ;:{"php":{"serviceName":"php","serviceTy= pe":"SingleTenant","clusterIdClusterMap":{"my= php.php.domain":{"serviceName":"php","cluster= Id":"myphp.php.domain","autoscalePolicyName":"= ;economy","hostNames":["myphp.apachestratos.org"],"isLb= Cluster":false,"isKubernetesCluster":true,"memberMap&qu= ot;:{"b44650cb-529b-11e4-acbb-08002794b041":{"serviceName&qu= ot;:"php","clusterId":"myphp.php.domain",&quo= t;memberId":"b44650cb-529b-11e4-acbb-08002794b041","por= tMap":{"8280":{"protocol":"http","v= alue":80,"proxy":8280}},"memberPublicIp":"172= .17.8.101","status":"Activated","memberIp&quo= t;:"172.17.8.101","properties":{}},"3ccfb8bd-529a-= 11e4-acbb-08002794b041":{"serviceName":"php","= ;clusterId":"myphp.php.domain","memberId":"3c= cfb8bd-529a-11e4-acbb-08002794b041","portMap":{"8280&qu= ot;:{"protocol":"http","value":80,"proxy= ":8280}},"memberPublicIp":"172.17.8.101","sta= tus":"Activated","memberIp":"172.17.8.101&quo= t;,"properties":{"ALLOCATED_SERVICE_HOST_PORT":"45= 00"}}},"status":"Created","properties":{= "payload_parameter.APP_PATH":"/var/www","KUBERNETE= S_REPLICAS_MAX":"20","KUBERNETES_CLUSTER_ID":"= ;KubGrp1","payload_parameter.LOG_LEVEL":"DEBUG",&q= uot;KUBERNETES_REPLICAS_MIN":"2","payload_parameter.MB_= IP":"172.17.42.1","payload_parameter.CEP_IP":"= ;172.17.42.1","payload_parameter.MB_PORT":"1883",&= quot;payload_parameter.CEP_PORT":"7611"}}},"portMap&quo= t;:{"8280":{"protocol":"http","value&quo= t;:80,"proxy":8280}},"properties":{}}},"initialize= d":false}
TID: [0] [STRATOS] [2014-10-13 11:15:32,642] DEBUG= {org.apache.stratos.cloud.controller.topology.TopologyManager} - =C2=A0Wri= te lock released
= TID: [0] [STRATOS] [2014-10-13 11:15:32,642] =C2=A0INFO {org.apache.stratos= .cloud.controller.topology.TopologyEventPublisher} - =C2=A0Publishing membe= r terminated event: [service] php [cluster] myphp.php.domain [network-parti= tion] null [partition] null [member] 3ccf4e4e-529a-11e4-acbb-08002794b041
TID: [0] [S= TRATOS] [2014-10-13 11:15:32,650] =C2=A0INFO {org.apache.stratos.cep.extens= ion.CEPTopologyEventReceiver} - =C2=A0Member [member id] 3ccf4e4e-529a-11e4= -acbb-08002794b041 was removed from the time stamp map.
TI= D: [0] [STRATOS] [2014-10-13 11:15:32,652] =C2=A0INFO {org.apache.stratos.m= essaging.message.processor.topology.MemberTerminatedMessageProcessor} - =C2= =A0Member terminated: [service] php [cluster] myphp.php.domain [member] 3cc= f4e4e-529a-11e4-acbb-08002794b041
TID: [0] [STRATOS] [2014-10-13 = 11:15:32,653] =C2=A0INFO {org.apache.stratos.manager.topology.receiver.Stra= tosManagerTopologyEventReceiver} - =C2=A0[MemberTerminatedEventListener] Re= ceived: class org.apache.stratos.messaging.event.topology.MemberTerminatedE= vent
TID: [0] [STRATOS] [2014-10-13 11:15:32,653] DEBUG {org.apac= he.stratos.manager.topology.model.TopologyClusterInformationModel} - =C2=A0= Adding cluster [myphp.php.domain]=C2=A0
TID: [0] [STRATOS] [2014= -10-13 11:15:32,653] =C2=A0WARN {org.apache.stratos.messaging.message.proce= ssor.topology.MemberTerminatedMessageProcessor} - =C2=A0Member already term= inated: [service] php [cluster] myphp.php.domain [member] 3ccf4e4e-529a-11e= 4-acbb-08002794b041
TID: [0] [STRATOS] [2014-10-13 11:15:32,654] = =C2=A0WARN {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor}= - =C2=A0Member is in the wrong list and it is removed from active members = list
TID: [0] [STRATOS] [2014-10-13 11:15:32,654] DEBUG {org.apac= he.stratos.cloud.controller.runtime.FasterLookUpDataHolder} - =C2=A0MemberC= ontext [id]: 3ccf4e4e-529a-11e4-acbb-08002794b041 removed from information = model.
TID: [0] [STRATOS] [2014-10-13 11:15:32,654] =C2=A0INFO {o= rg.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - =C2=A0Memb= er stat context has been removed successfully: [member] 3ccf4e4e-529a-11e4-= acbb-08002794b041
TID: [0] [STRATOS] [2014-10-13 11:15:32,655] = =C2=A0WARN {org.apache.stratos.messaging.message.processor.topology.MemberT= erminatedMessageProcessor} - =C2=A0Member already terminated: [service] php= [cluster] myphp.php.domain [member] 3ccf4e4e-529a-11e4-acbb-08002794b041
TID: [0] [STRATOS] [2014-10-13 11:15:32,654] =C2=A0INFO {org.apach= e.stratos.cloud.controller.runtime.FasterLookUpDataHolder} - =C2=A0Schedule= d Pod Activation Watcher task canceled.
TID: [0] [STRATOS] [2014-= 10-13 11:15:32,696] =C2=A0INFO {org.apache.stratos.cloud.controller.impl.Cl= oudControllerServiceImpl} - =C2=A0Kubernetes entities are successfully star= ting up. [MemberContext [memberId=3Db4468fb6-529b-11e4-acbb-08002794b041, n= odeId=3Dnull, instanceId=3Dnull, clusterId=3Dmyphp.php.domain, partition=3D= null, cartridgeType=3Dphp, privateIpAddress=3D172.17.8.100, publicIpAddress= =3D172.17.8.100, allocatedIpAddress=3Dnull, initTime=3D1413178952532, lbClu= sterId=3Dnull, networkPartitionId=3Dnull, properties=3DProperties [properti= es=3D[Property [name=3DALLOCATED_SERVICE_HOST_PORT, value=3Dnull]]]], Membe= rContext [memberId=3Db446c0dd-529b-11e4-acbb-08002794b041, nodeId=3Dnull, i= nstanceId=3Dnull, clusterId=3Dmyphp.php.domain, partition=3Dnull, cartridge= Type=3Dphp, privateIpAddress=3D172.17.8.101, publicIpAddress=3D172.17.8.101= , allocatedIpAddress=3Dnull, initTime=3D1413178952532, lbClusterId=3Dnull, = networkPartitionId=3Dnull, properties=3DProperties [properties=3D[Property = [name=3DALLOCATED_SERVICE_HOST_PORT, value=3Dnull]]]], MemberContext [membe= rId=3D3ccf4e4e-529a-11e4-acbb-08002794b041, nodeId=3Dnull, instanceId=3Dnul= l, clusterId=3Dmyphp.php.domain, partition=3Dnull, cartridgeType=3Dphp, pri= vateIpAddress=3D172.17.8.101, publicIpAddress=3D172.17.8.101, allocatedIpAd= dress=3Dnull, initTime=3D1413178322623, lbClusterId=3Dnull, networkPartitio= nId=3Dnull, properties=3DProperties [properties=3D[Property [name=3DALLOCAT= ED_SERVICE_HOST_PORT, value=3D4500]]]]]
TID: [0] [STRATOS] [2014-= 10-13 11:15:32,709] DEBUG {org.apache.stratos.autoscaler.rule.RuleTasksDele= gator} - =C2=A0Scaling down, terminated the member with id b4468fb6-529b-11= e4-acbb-08002794b041 in cluster myphp.php.domain
TID: [0] [STRATO= S] [2014-10-13 11:15:32,709] DEBUG {org.apache.stratos.autoscaler.rule.Rule= TasksDelegator} - =C2=A0Scaling down, terminated the member with id b446c0d= d-529b-11e4-acbb-08002794b041 in cluster myphp.php.domain
TID: [0= ] [STRATOS] [2014-10-13 11:15:32,709] DEBUG {org.apache.stratos.autoscaler.= rule.RuleTasksDelegator} - =C2=A0Scaling down, terminated the member with i= d 3ccf4e4e-529a-11e4-acbb-08002794b041 in cluster myphp.php.domain
TID: [0] [STRATOS] [2014-10-13 11:15:32,709] DEBUG {org.apache.stratos.au= toscaler.rule.AutoscalerRuleEvaluator} - =C2=A0Scale check executed for : o= rg.apache.stratos.autoscaler.KubernetesClusterContext@29d32c84=C2=A0
<= div>TID: [0] [STRATOS] [2014-10-13 11:16:01,002] DEBUG {org.apache.stratos.= manager.publisher.TenantSynzhronizerTask} - =C2=A0Publishing complete tenan= t event
TID: [0] [STRATOS] [2014-10-13 11:16:01,006] DEBUG {org.a= pache.stratos.cloud.controller.topology.TopologySynchronizerTask} - =C2=A0E= xecuting topology synchronization task
TID: [0] [STRATOS] [2014-1= 0-13 11:16:01,006] DEBUG {org.apache.stratos.cloud.controller.topology.Topo= logyEventPublisher} - =C2=A0Publishing complete topology event
TI= D: [0] [STRATOS] [2014-10-13 11:16:22,230] DEBUG {org.apache.stratos.autosc= aler.monitor.KubernetesClusterMonitor} - =C2=A0Grad of load avg event: [clu= ster] myphp.php.domain [value] -0.70397747
TID: [0] [STRATOS] [20= 14-10-13 11:16:22,232] DEBUG {org.apache.stratos.autoscaler.monitor.Kuberne= tesClusterMonitor} - =C2=A0Grad of Memory Consumption event: [cluster] myph= p.php.domain [value] -0.46528494
TID: [0] [STRATOS] [2014-10-13 1= 1:16:22,234] DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesCluster= Monitor} - =C2=A0Avg Memory Consumption event: [cluster] myphp.php.domain [= value] 27.538944
TID: [0] [STRATOS] [2014-10-13 11:16:22,236] DEB= UG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - =C2= =A0Second Derivation of load avg event: [cluster] myphp.php.domain [value] = -0.0012131054
TID: [0] [STRATOS] [2014-10-13 11:16:22,236] DEBUG = {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - =C2=A0Se= cond Derivation of Memory Consumption event: [cluster] myphp.php.domain [va= lue] 0.021650476
TID: [0] [STRATOS] [2014-10-13 11:16:22,236] DEB= UG {org.apache.stratos.autoscaler.KubernetesClusterContext} - =C2=A0Memory = consumption stats are reset, ready to do scale check [kub cluster] KubGrp1<= /div>
TID: [0] [STRATOS] [2014-10-13 11:16:22,237] DEBUG {org.apache.st= ratos.autoscaler.monitor.KubernetesClusterMonitor} - =C2=A0Avg load avg eve= nt: [cluster] myphp.php.domain [value] 53.7
TID: [0] [STRATOS] [2= 014-10-13 11:16:22,237] DEBUG {org.apache.stratos.autoscaler.KubernetesClus= terContext} - =C2=A0Load average stats are reset, ready to do scale check [= kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-13 11:16:22,425]= DEBUG {org.apache.stratos.autoscaler.message.receiver.health.AutoscalerHea= lthStatEventReceiver} - =C2=A0Member not found in the Topology: [member] 3c= cf4e4e-529a-11e4-acbb-08002794b041
TID: [0] [STRATOS] [2014-10-13= 11:16:22,426] DEBUG {org.apache.stratos.autoscaler.message.receiver.health= .AutoscalerHealthStatEventReceiver} - =C2=A0Member not found in the Topolog= y: [member] 3ccf4e4e-529a-11e4-acbb-08002794b041
TID: [0] [STRATO= S] [2014-10-13 11:16:22,426] DEBUG {org.apache.stratos.autoscaler.message.r= eceiver.health.AutoscalerHealthStatEventReceiver} - =C2=A0Member not found = in the Topology: [member] 3ccf4e4e-529a-11e4-acbb-08002794b041
TI= D: [0] [STRATOS] [2014-10-13 11:16:22,427] DEBUG {org.apache.stratos.autosc= aler.message.receiver.health.AutoscalerHealthStatEventReceiver} - =C2=A0Mem= ber not found in the Topology: [member] 3ccf4e4e-529a-11e4-acbb-08002794b04= 1
TID: [0] [STRATOS] [2014-10-13 11:16:22,427] DEBUG {org.apache.= stratos.autoscaler.message.receiver.health.AutoscalerHealthStatEventReceive= r} - =C2=A0Member not found in the Topology: [member] 3ccf4e4e-529a-11e4-ac= bb-08002794b041
TID: [0] [STRATOS] [2014-10-13 11:16:52,406] DEBU= G {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -= =C2=A0KubernetesServiceClusterMonitor is running.. KubernetesServiceCluste= rMonitor [ kubernetesHostClusterId=3DKubGrp1, clusterId=3Dmyphp.php.domain,= serviceId=3Dphp]
TID: [0] [STRATOS] [2014-10-13 11:16:52,407] DE= BUG {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor}= - =C2=A0Running min check for [kub-cluster] : KubGrp1 [cluster] : myphp.ph= p.domain=C2=A0
TID: [0] [STRATOS] [2014-10-13 11:16:52,408] =C2= =A0INFO {org.apache.stratos.autoscaler.rule.RuleLog} - =C2=A0Running minimu= m rule: [kub-cluster] KubGrp1 [cluster] myphp.php.domain
TID: [0]= [STRATOS] [2014-10-13 11:16:52,408] =C2=A0INFO {org.apache.stratos.autosca= ler.rule.RuleLog} - =C2=A0[min-check] =C2=A0[cluster] : myphp.php.domain [R= eplicas] nonTerminated : 2
TID: [0] [STRATOS] [2014-10-13 11:16:5= 2,409] =C2=A0INFO {org.apache.stratos.autoscaler.rule.RuleLog} - =C2=A0[min= -check] =C2=A0[cluster] : myphp.php.domain [Replicas] minReplicas : 2
=
TID: [0] [STRATOS] [2014-10-13 11:16:52,409] =C2=A0INFO {org.apache.st= ratos.autoscaler.rule.RuleLog} - =C2=A0Running obsolete containers rule [ku= b-cluster] : KubGrp1 [cluster] : myphp.php.domain
TID: [0] [STRAT= OS] [2014-10-13 11:16:52,410] =C2=A0INFO {org.apache.stratos.autoscaler.rul= e.RuleLog} - =C2=A0[obsolete-check] [cluster] : myphp.php.domain [Replicas]= obsoleteReplicas : 0
TID: [0] [STRATOS] [2014-10-13 11:16:52,411= ] DEBUG {org.apache.stratos.autoscaler.rule.AutoscalerRuleEvaluator} - =C2= =A0Minimum check executed for : org.apache.stratos.autoscaler.KubernetesClu= sterContext@29d32c84=C2=A0
TID: [0] [STRATOS] [2014-10-13 11:16:5= 2,411] DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesServiceCluste= rMonitor} - =C2=A0flag of rifReset : false flag of memoryConsumptionReset := true flag of loadAverageReset : true
TID: [0] [STRATOS] [2014-10= -13 11:16:52,412] DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesSe= rviceClusterMonitor} - =C2=A0Running scale check for [kub-cluster] : KubGrp= 1 [cluster] : myphp.php.domain=C2=A0
TID: [0] [STRATOS] [2014-10-= 13 11:16:52,412] =C2=A0INFO {org.apache.stratos.autoscaler.rule.RuleLog} - = =C2=A0Running scaling rule [kub-cluster] : KubGrp1 [cluster] : myphp.php.do= main
TID: [0] [STRATOS] [2014-10-13 11:16:52,414] DEBUG {org.apac= he.stratos.autoscaler.rule.RuleTasksDelegator} - =C2=A0Predicting the value= , [average]: 0.0 , [gradient]: 0.0 , [second derivative]: 0.0 , [time inter= vals]: 1=C2=A0
TID: [0] [STRATOS] [2014-10-13 11:16:52,415] DEBUG= {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - =C2=A0Predicting= the value, [average]: 27.538944 , [gradient]: -0.46528494 , [second deriva= tive]: 0.021650476 , [time intervals]: 1=C2=A0
TID: [0] [STRATOS]= [2014-10-13 11:16:52,416] DEBUG {org.apache.stratos.autoscaler.rule.RuleTa= sksDelegator} - =C2=A0Predicting the value, [average]: 53.7 , [gradient]: -= 0.70397747 , [second derivative]: -0.0012131054 , [time intervals]: 1=C2=A0=
TID: [0] [STRATOS] [2014-10-13 11:16:52,417] =C2=A0INFO {org.apa= che.stratos.autoscaler.rule.RuleLog} - =C2=A0[scaling] =C2=A0[cluster] : my= php.php.domain [Replicas] minReplicas : 2
TID: [0] [STRATOS] [201= 4-10-13 11:16:52,417] =C2=A0INFO {org.apache.stratos.autoscaler.rule.RuleLo= g} - =C2=A0[scaling] =C2=A0[cluster] : myphp.php.domain [Replicas] maxRepli= cas : 20
TID: [0] [STRATOS] [2014-10-13 11:16:52,417] =C2=A0INFO = {org.apache.stratos.autoscaler.rule.RuleLog} - =C2=A0[scaling] =C2=A0[clust= er] : myphp.php.domain [Replicas] nonTerminated : 2
TID: [0] [STR= ATOS] [2014-10-13 11:16:52,417] =C2=A0INFO {org.apache.stratos.autoscaler.r= ule.RuleLog} - =C2=A0[scaling] =C2=A0[cluster] : myphp.php.domain [Replicas= ] activeReplicas : 2
TID: [0] [STRATOS] [2014-10-13 11:16:52,417]= DEBUG {org.apache.stratos.autoscaler.rule.RuleLog} - =C2=A0[scaling] =C2= =A0[cluster] : myphp.php.domain [RequestInFlight] resetted ? : false
<= div>TID: [0] [STRATOS] [2014-10-13 11:16:52,417] DEBUG {org.apache.stratos.= autoscaler.rule.RuleLog} - =C2=A0[scaling] =C2=A0[cluster] : myphp.php.doma= in [RequestInFlight] predicted value : 0.0
TID: [0] [STRATOS] [20= 14-10-13 11:16:52,418] DEBUG {org.apache.stratos.autoscaler.rule.RuleLog} -= =C2=A0[scaling] =C2=A0[cluster] : myphp.php.domain [RequestInFlight] upper= limit : 80.0
TID: [0] [STRATOS] [2014-10-13 11:16:52,418] DEBUG = {org.apache.stratos.autoscaler.rule.RuleLog} - =C2=A0[scaling] =C2=A0[clust= er] : myphp.php.domain [RequestInFlight] lower limit : 5.0
TID: [= 0] [STRATOS] [2014-10-13 11:16:52,418] DEBUG {org.apache.stratos.autoscaler= .rule.RuleLog} - =C2=A0[scaling] =C2=A0[cluster] : myphp.php.domain [Memory= Consumption] resetted ? : true
TID: [0] [STRATOS] [2014-10-13 11:= 16:52,418] =C2=A0INFO {org.apache.stratos.autoscaler.rule.RuleLog} - =C2=A0= [scaling] =C2=A0[cluster] : myphp.php.domain [MemoryConsumption] predicted = value : 27.084485135041177
TID: [0] [STRATOS] [2014-10-13 11:16:5= 2,418] =C2=A0INFO {org.apache.stratos.autoscaler.rule.RuleLog} - =C2=A0[sca= ling] =C2=A0[cluster] : myphp.php.domain [MemoryConsumption] upper limit : = 80.0
TID: [0] [STRATOS] [2014-10-13 11:16:52,419] =C2=A0INFO {org= .apache.stratos.autoscaler.rule.RuleLog} - =C2=A0[scaling] =C2=A0[cluster] = : myphp.php.domain [MemoryConsumption] lower limit : 15.0
TID: [0= ] [STRATOS] [2014-10-13 11:16:52,419] DEBUG {org.apache.stratos.autoscaler.= rule.RuleLog} - =C2=A0[scaling] =C2=A0[cluster] : myphp.php.domain [LoadAve= rage] resetted ? : true
TID: [0] [STRATOS] [2014-10-13 11:16:52,4= 19] =C2=A0INFO {org.apache.stratos.autoscaler.rule.RuleLog} - =C2=A0[scalin= g] =C2=A0[cluster] : myphp.php.domain [LoadAverage] predicted value : 52.99= 54185327515
TID: [0] [STRATOS] [2014-10-13 11:16:52,419] =C2=A0IN= FO {org.apache.stratos.autoscaler.rule.RuleLog} - =C2=A0[scaling] =C2=A0[cl= uster] : myphp.php.domain [LoadAverage] upper limit : 180.0
TID: = [0] [STRATOS] [2014-10-13 11:16:52,420] =C2=A0INFO {org.apache.stratos.auto= scaler.rule.RuleLog} - =C2=A0[scaling] =C2=A0[cluster] : myphp.php.domain [= LoadAverage] lower limit : 20.0
TID: [0] [STRATOS] [2014-10-13 11= :16:52,420] =C2=A0INFO {org.apache.stratos.autoscaler.rule.RuleLog} - =C2= =A0[scaling] =C2=A0[cluster] : myphp.php.domain scale-up action : false
TID: [0] [STRATOS] [2014-10-13 11:16:52,420] =C2=A0INFO {org.apache.= stratos.autoscaler.rule.RuleLog} - =C2=A0[scaling] =C2=A0[cluster] : myphp.= php.domain scale-down action : true
TID: [0] [STRATOS] [2014-10-1= 3 11:16:52,421] =C2=A0INFO {org.apache.stratos.autoscaler.rule.RuleLog} - = =C2=A0[scaling] =C2=A0[cluster] : myphp.php.domain Even if all predicted st= ats < upperLimit, still activeReplicas <=3D minReplicas ; hence not s= caling down=C2=A0
TID: [0] [STRATOS] [2014-10-13 11:16:52,421] DE= BUG {org.apache.stratos.autoscaler.rule.AutoscalerRuleEvaluator} - =C2=A0Sc= ale check executed for : org.apache.stratos.autoscaler.KubernetesClusterCon= text@29d32c84=C2=A0
TID: [0] [STRATOS] [2014-10-13 11:17:01,001] = DEBUG {org.apache.stratos.manager.publisher.TenantSynzhronizerTask} - =C2= =A0Publishing complete tenant event
TID: [0] [STRATOS] [2014-10-1= 3 11:17:01,002] DEBUG {org.apache.stratos.cloud.controller.topology.Topolog= ySynchronizerTask} - =C2=A0Executing topology synchronization task
TID: [0] [STRATOS] [2014-10-13 11:17:01,002] DEBUG {org.apache.stratos.cl= oud.controller.topology.TopologyEventPublisher} - =C2=A0Publishing complete= topology event
T= ID: [0] [STRATOS] [2014-10-13 11:17:16,141] =C2=A0INFO {org.apache.stratos.= cep.extension.FaultHandlingWindowProcessor} - =C2=A0Faulty member detected = [member-id] 3ccf4e4e-529a-11e4-acbb-08002794b041 with [last time-stamp] 141= 3179137678 [time-out] 60000 milliseconds
TID: [0] [STRATOS= ] [2014-10-13 11:17:16,141] ERROR {org.apache.stratos.cep.extension.FaultHa= ndlingWindowProcessor} - =C2=A0Failed to publish member fault event. Member= having [member-id] 3ccf4e4e-529a-11e4-acbb-08002794b041 does not exist in = topology
TID: [0] [STRATOS] [2014-10-13 11:17:22,231] DEBUG {org.= apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - =C2=A0Avg loa= d avg event: [cluster] myphp.php.domain [value] 23.75
TID: [0] [S= TRATOS] [2014-10-13 11:17:22,232] DEBUG {org.apache.stratos.autoscaler.moni= tor.KubernetesClusterMonitor} - =C2=A0Grad of load avg event: [cluster] myp= hp.php.domain [value] -0.2682943
TID: [0] [STRATOS] [2014-10-13 1= 1:17:22,236] DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesCluster= Monitor} - =C2=A0Avg Memory Consumption event: [cluster] myphp.php.domain [= value] 17.602432
TID: [0] [STRATOS] [2014-10-13 11:17:22,237] DEB= UG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} - =C2= =A0Second Derivation of load avg event: [cluster] myphp.php.domain [value] = -0.0045045386
TID: [0] [STRATOS] [2014-10-13 11:17:22,237] DEBUG = {org.apache.stratos.autoscaler.KubernetesClusterContext} - =C2=A0Load avera= ge stats are reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-13 11:17:22,237] DEBUG {org.apache.stratos.aut= oscaler.monitor.KubernetesClusterMonitor} - =C2=A0Grad of Memory Consumptio= n event: [cluster] myphp.php.domain [value] 0.0016795773
TID: [0]= [STRATOS] [2014-10-13 11:17:22,238] DEBUG {org.apache.stratos.autoscaler.m= onitor.KubernetesClusterMonitor} - =C2=A0Second Derivation of Memory Consum= ption event: [cluster] myphp.php.domain [value] 2.3589536E-4
TID:= [0] [STRATOS] [2014-10-13 11:17:22,238] DEBUG {org.apache.stratos.autoscal= er.KubernetesClusterContext} - =C2=A0Memory consumption stats are reset, re= ady to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2= 014-10-13 11:18:01,000] DEBUG {org.apache.stratos.manager.publisher.TenantS= ynzhronizerTask} - =C2=A0Publishing complete tenant event
TID: [0= ] [STRATOS] [2014-10-13 11:18:01,001] DEBUG {org.apache.stratos.cloud.contr= oller.topology.TopologySynchronizerTask} - =C2=A0Executing topology synchro= nization task
TID: [0] [STRATOS] [2014-10-13 11:18:01,001] DEBUG = {org.apache.stratos.cloud.controller.topology.TopologyEventPublisher} - =C2= =A0Publishing complete topology event
TID: [0] [STRATOS] [2014-10-13 11:18:16,142] =C2=A0INF= O {org.apache.stratos.cep.extension.FaultHandlingWindowProcessor} - =C2=A0F= aulty member detected [member-id] 3ccf4e4e-529a-11e4-acbb-08002794b041 with= [last time-stamp] 1413179137678 [time-out] 60000 milliseconds
=
TID: [0] [STRATOS] [2014-10-13 11:18:16,144] ERROR {org.apache.stratos= .cep.extension.FaultHandlingWindowProcessor} - =C2=A0Failed to publish memb= er fault event. Member having [member-id] 3ccf4e4e-529a-11e4-acbb-08002794b= 041 does not exist in topology

Thanks.

--
Rajkumar Rajaratnam
Software Engineer | WSO2, Inc.



--
Regards,
Manula Chathurika Thantriwatte
Soft= ware Engineer
WSO2 Inc. : http://wso2.com
lean . enterprise . middle= ware

<= div>email : manula= c@wso2.com / man= ula@apache.org
<= font color=3D"#888888">phone : +94 772492511



--089e0112ce22d0836c05054a6d74--