In the same log, you can see member consumption also has too large value

TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -  Second Derivation of Memory Consumption event: [cluster] php.php.domain [value] 200000.0

On Sat, Oct 25, 2014 at 10:39 PM, Rajkumar Rajaratnam <rajkumarr@wso2.com> wrote:
Hi,

I am doing a test round with M3 with python agent. And faced this very strange scenario, $Subject.


TID: [0] [STRATOS] [2014-10-25 22:10:38,693] DEBUG {org.apache.stratos.autoscaler.KubernetesClusterContext} -  Load average stats are reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,693] DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -  Grad of load avg event: [cluster] php.php.domain [value] -2000.0
TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG {org.apache.stratos.autoscaler.KubernetesClusterContext} -  Load average stats are reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -  Grad of load avg event: [cluster] php.php.domain [value] 0.0
TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG {org.apache.stratos.autoscaler.KubernetesClusterContext} -  Load average stats are reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -  Avg load avg event: [cluster] php.php.domain [value] 1.9090909
TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG {org.apache.stratos.autoscaler.KubernetesClusterContext} -  Load average stats are reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,694] DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -  Avg Memory Consumption event: [cluster] php.php.domain [value] 28.718182
TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -  Second Derivation of Memory Consumption event: [cluster] php.php.domain [value] 200000.0
TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -  Grad of load avg event: [cluster] php.php.domain [value] 0.0
TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG {org.apache.stratos.autoscaler.KubernetesClusterContext} -  Load average stats are reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -  Second Derivation of load avg event: [cluster] php.php.domain [value] 0.0
TID: [0] [STRATOS] [2014-10-25 22:10:38,695] DEBUG {org.apache.stratos.autoscaler.KubernetesClusterContext} -  Load average stats are reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -  Second Derivation of load avg event: [cluster] php.php.domain [value] 0.0
TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG {org.apache.stratos.autoscaler.KubernetesClusterContext} -  Load average stats are reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -  Avg load avg event: [cluster] php.php.domain [value] 3.3636363
TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG {org.apache.stratos.autoscaler.KubernetesClusterContext} -  Load average stats are reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -  Grad of Memory Consumption event: [cluster] php.php.domain [value] 0.11670045
TID: [0] [STRATOS] [2014-10-25 22:10:38,696] DEBUG {org.apache.stratos.autoscaler.KubernetesClusterContext} -  Memory consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -  Avg Memory Consumption event: [cluster] php.php.domain [value] 28.7
TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG {org.apache.stratos.autoscaler.KubernetesClusterContext} -  Memory consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -  Second Derivation of Memory Consumption event: [cluster] php.php.domain [value] 0.0
TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG {org.apache.stratos.autoscaler.KubernetesClusterContext} -  Memory consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -  Avg Memory Consumption event: [cluster] php.php.domain [value] 28.7
TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG {org.apache.stratos.autoscaler.KubernetesClusterContext} -  Memory consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -  Avg load avg event: [cluster] php.php.domain [value] 5.0
TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG {org.apache.stratos.autoscaler.KubernetesClusterContext} -  Load average stats are reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,697] DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -  Grad of load avg event: [cluster] php.php.domain [value] 2000.0
TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG {org.apache.stratos.autoscaler.KubernetesClusterContext} -  Load average stats are reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -  Avg Memory Consumption event: [cluster] php.php.domain [value] 28.7
TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG {org.apache.stratos.autoscaler.KubernetesClusterContext} -  Memory consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -  Grad of Memory Consumption event: [cluster] php.php.domain [value] 0.0
TID: [0] [STRATOS] [2014-10-25 22:10:38,698] DEBUG {org.apache.stratos.autoscaler.KubernetesClusterContext} -  Memory consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,699] DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -  Second Derivation of Memory Consumption event: [cluster] php.php.domain [value] 0.0
TID: [0] [STRATOS] [2014-10-25 22:10:38,699] DEBUG {org.apache.stratos.autoscaler.KubernetesClusterContext} -  Memory consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -  Second Derivation of load avg event: [cluster] php.php.domain [value] 2000000.0
TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG {org.apache.stratos.autoscaler.KubernetesClusterContext} -  Load average stats are reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -  Grad of load avg event: [cluster] php.php.domain [value] 13500.0
TID: [0] [STRATOS] [2014-10-25 22:10:38,700] DEBUG {org.apache.stratos.autoscaler.KubernetesClusterContext} -  Load average stats are reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -  Grad of Memory Consumption event: [cluster] php.php.domain [value] 50.0
TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG {org.apache.stratos.autoscaler.KubernetesClusterContext} -  Memory consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -  Avg Memory Consumption event: [cluster] php.php.domain [value] 28.738462
TID: [0] [STRATOS] [2014-10-25 22:10:38,701] DEBUG {org.apache.stratos.autoscaler.KubernetesClusterContext} -  Memory consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -  Avg load avg event: [cluster] php.php.domain [value] 2.3333333
TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG {org.apache.stratos.autoscaler.KubernetesClusterContext} -  Load average stats are reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -  Grad of load avg event: [cluster] php.php.domain [value] 0.0
TID: [0] [STRATOS] [2014-10-25 22:10:38,702] DEBUG {org.apache.stratos.autoscaler.KubernetesClusterContext} -  Load average stats are reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -  Grad of Memory Consumption event: [cluster] php.php.domain [value] 0.0
TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG {org.apache.stratos.autoscaler.KubernetesClusterContext} -  Memory consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -  Avg load avg event: [cluster] php.php.domain [value] 31.0
TID: [0] [STRATOS] [2014-10-25 22:10:38,703] DEBUG {org.apache.stratos.autoscaler.KubernetesClusterContext} -  Load average stats are reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -  Grad of Memory Consumption event: [cluster] php.php.domain [value] 0.0
TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG {org.apache.stratos.autoscaler.KubernetesClusterContext} -  Memory consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -  Avg load avg event: [cluster] php.php.domain [value] 39.0
TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG {org.apache.stratos.autoscaler.KubernetesClusterContext} -  Load average stats are reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,704] DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -  Avg load avg event: [cluster] php.php.domain [value] 39.0
TID: [0] [STRATOS] [2014-10-25 22:10:38,705] DEBUG {org.apache.stratos.autoscaler.KubernetesClusterContext} -  Load average stats are reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,706] DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -  Grad of load avg event: [cluster] php.php.domain [value] -11000.0
TID: [0] [STRATOS] [2014-10-25 22:10:38,706] DEBUG {org.apache.stratos.autoscaler.KubernetesClusterContext} -  Load average stats are reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,708] DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -  Grad of Memory Consumption event: [cluster] php.php.domain [value] 0.0
TID: [0] [STRATOS] [2014-10-25 22:10:38,708] DEBUG {org.apache.stratos.autoscaler.KubernetesClusterContext} -  Memory consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,709] DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesClusterMonitor} -  Avg Memory Consumption event: [cluster] php.php.domain [value] 28.8
TID: [0] [STRATOS] [2014-10-25 22:10:38,709] DEBUG {org.apache.stratos.autoscaler.KubernetesClusterContext} -  Memory consumption stats are reset, ready to do scale check [kub cluster] KubGrp1
TID: [0] [STRATOS] [2014-10-25 22:10:38,811] DEBUG {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -  PodActivationWatcher running : Running
TID: [0] [STRATOS] [2014-10-25 22:10:38,942] DEBUG {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -  PodActivationWatcher running : Running
TID: [0] [STRATOS] [2014-10-25 22:10:38,945] DEBUG {org.apache.stratos.cloud.controller.util.PodActivationWatcher} -  PodActivationWatcher running : Running
TID: [0] [STRATOS] [2014-10-25 22:10:43,712] DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -  KubernetesServiceClusterMonitor is running.. KubernetesServiceClusterMonitor [ kubernetesHostClusterId=KubGrp1, clusterId=php.php.domain, serviceId=php]
TID: [0] [STRATOS] [2014-10-25 22:10:43,713] DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -  Running min check for [kub-cluster] : KubGrp1 [cluster] : php.php.domain 
TID: [0] [STRATOS] [2014-10-25 22:10:43,713]  INFO {org.apache.stratos.autoscaler.rule.RuleLog} -  Running minimum rule: [kub-cluster] KubGrp1 [cluster] php.php.domain
TID: [0] [STRATOS] [2014-10-25 22:10:43,714]  INFO {org.apache.stratos.autoscaler.rule.RuleLog} -  [min-check]  [cluster] : php.php.domain [Replicas] nonTerminated : 3
TID: [0] [STRATOS] [2014-10-25 22:10:43,714]  INFO {org.apache.stratos.autoscaler.rule.RuleLog} -  [min-check]  [cluster] : php.php.domain [Replicas] minReplicas : 3
TID: [0] [STRATOS] [2014-10-25 22:10:43,715]  INFO {org.apache.stratos.autoscaler.rule.RuleLog} -  Running obsolete containers rule [kub-cluster] : KubGrp1 [cluster] : php.php.domain
TID: [0] [STRATOS] [2014-10-25 22:10:43,715]  INFO {org.apache.stratos.autoscaler.rule.RuleLog} -  [obsolete-check] [cluster] : php.php.domain [Replicas] obsoleteReplicas : 0
TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG {org.apache.stratos.autoscaler.rule.AutoscalerRuleEvaluator} -  Minimum check executed for : org.apache.stratos.autoscaler.KubernetesClusterContext@21d906d0 
TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -  flag of rifReset : true flag of memoryConsumptionReset : true flag of loadAverageReset : true
TID: [0] [STRATOS] [2014-10-25 22:10:43,715] DEBUG {org.apache.stratos.autoscaler.monitor.KubernetesServiceClusterMonitor} -  Running scale check for [kub-cluster] : KubGrp1 [cluster] : php.php.domain 
TID: [0] [STRATOS] [2014-10-25 22:10:43,716]  INFO {org.apache.stratos.autoscaler.rule.RuleLog} -  Running scaling rule [kub-cluster] : KubGrp1 [cluster] : php.php.domain
TID: [0] [STRATOS] [2014-10-25 22:10:43,716] DEBUG {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} -  Predicting the value, [average]: 0.0 , [gradient]: 0.0 , [second derivative]: 0.0 , [time intervals]: 1 
TID: [0] [STRATOS] [2014-10-25 22:10:43,716] DEBUG {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} -  Predicting the value, [average]: 28.8 , [gradient]: 0.0 , [second derivative]: 0.0 , [time intervals]: 1 
TID: [0] [STRATOS] [2014-10-25 22:10:43,717] DEBUG {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} -  Predicting the value, [average]: 39.0 , [gradient]: -11000.0 , [second derivative]: 2000000.0 , [time intervals]: 1 
TID: [0] [STRATOS] [2014-10-25 22:10:43,717]  INFO {org.apache.stratos.autoscaler.rule.RuleLog} -  [scaling]  [cluster] : php.php.domain [Replicas] minReplicas : 3
TID: [0] [STRATOS] [2014-10-25 22:10:43,717]  INFO {org.apache.stratos.autoscaler.rule.RuleLog} -  [scaling]  [cluster] : php.php.domain [Replicas] maxReplicas : 10
TID: [0] [STRATOS] [2014-10-25 22:10:43,718]  INFO {org.apache.stratos.autoscaler.rule.RuleLog} -  [scaling]  [cluster] : php.php.domain [Replicas] nonTerminated : 3
TID: [0] [STRATOS] [2014-10-25 22:10:43,718]  INFO {org.apache.stratos.autoscaler.rule.RuleLog} -  [scaling]  [cluster] : php.php.domain [Replicas] activeReplicas : 3
TID: [0] [STRATOS] [2014-10-25 22:10:43,718]  INFO {org.apache.stratos.autoscaler.rule.RuleLog} -  [scaling]  [cluster] : php.php.domain [RequestInFlight] predicted value : 0.0
TID: [0] [STRATOS] [2014-10-25 22:10:43,718]  INFO {org.apache.stratos.autoscaler.rule.RuleLog} -  [scaling]  [cluster] : php.php.domain [RequestInFlight] upper limit : 80.0
TID: [0] [STRATOS] [2014-10-25 22:10:43,718]  INFO {org.apache.stratos.autoscaler.rule.RuleLog} -  [scaling]  [cluster] : php.php.domain [RequestInFlight] lower limit : 5.0
TID: [0] [STRATOS] [2014-10-25 22:10:43,718]  INFO {org.apache.stratos.autoscaler.rule.RuleLog} -  [scaling]  [cluster] : php.php.domain [MemoryConsumption] predicted value : 28.799999237060547
TID: [0] [STRATOS] [2014-10-25 22:10:43,718]  INFO {org.apache.stratos.autoscaler.rule.RuleLog} -  [scaling]  [cluster] : php.php.domain [MemoryConsumption] upper limit : 80.0
TID: [0] [STRATOS] [2014-10-25 22:10:43,718]  INFO {org.apache.stratos.autoscaler.rule.RuleLog} -  [scaling]  [cluster] : php.php.domain [MemoryConsumption] lower limit : 15.0
TID: [0] [STRATOS] [2014-10-25 22:10:43,718]  INFO {org.apache.stratos.autoscaler.rule.RuleLog} -  [scaling]  [cluster] : php.php.domain [LoadAverage] predicted value : 989039.0
TID: [0] [STRATOS] [2014-10-25 22:10:43,719]  INFO {org.apache.stratos.autoscaler.rule.RuleLog} -  [scaling]  [cluster] : php.php.domain [LoadAverage] upper limit : 180.0
TID: [0] [STRATOS] [2014-10-25 22:10:43,719]  INFO {org.apache.stratos.autoscaler.rule.RuleLog} -  [scaling]  [cluster] : php.php.domain [LoadAverage] lower limit : 20.0
TID: [0] [STRATOS] [2014-10-25 22:10:43,719]  INFO {org.apache.stratos.autoscaler.rule.RuleLog} -  [scaling]  [cluster] : php.php.domain scale-up action : true
TID: [0] [STRATOS] [2014-10-25 22:10:43,719]  INFO {org.apache.stratos.autoscaler.rule.RuleLog} -  [scaling]  [cluster] : php.php.domain scale-down action : false
TID: [0] [STRATOS] [2014-10-25 22:10:43,719]  INFO {org.apache.stratos.autoscaler.rule.RuleLog} -  [scaling]  [cluster] : php.php.domain [LoadAverage] predicted replicas : 16484
TID: [0] [STRATOS] [2014-10-25 22:10:43,720]  INFO {org.apache.stratos.autoscaler.rule.RuleLog} -  [scaling]  [cluster] : php.php.domain predicted replicas > max replicas : 
TID: [0] [STRATOS] [2014-10-25 22:10:43,720]  INFO {org.apache.stratos.autoscaler.rule.RuleLog} -  [scaling] Decided to scale-up : [cluster] : php.php.domain
TID: [0] [STRATOS] [2014-10-25 22:10:43,720]  INFO {org.apache.stratos.autoscaler.rule.RuleLog} -  [scaling-up]  [cluster] : php.php.domain valid number of replicas to expand : 10
TID: [0] [STRATOS] [2014-10-25 22:10:43,720]  INFO {org.apache.stratos.autoscaler.client.cloud.controller.CloudControllerClient} -  Updating kubernetes replication controller via cloud controller: [cluster] php.php.domain [replicas] 10
TID: [0] [STRATOS] [2014-10-25 22:10:43,734] DEBUG {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} -  CloudControllerServiceImpl:updateContainers for cluster : php.php.domain
TID: [0] [STRATOS] [2014-10-25 22:10:43,758] DEBUG {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} -  Cloud Controller is delegating request to update a replication controller php.php.domain to Kubernetes layer.


I couldn't investigate the agent side, since containers are destroyed.

Any idea?

Thanks.

--
Rajkumar Rajaratnam
Software Engineer | WSO2, Inc.



--
Rajkumar Rajaratnam
Software Engineer | WSO2, Inc.
Mobile +94777568639 | +94783498120