Return-Path: X-Original-To: apmail-stratos-dev-archive@minotaur.apache.org Delivered-To: apmail-stratos-dev-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 7166217ED5 for ; Sun, 12 Jul 2015 03:54:06 +0000 (UTC) Received: (qmail 17724 invoked by uid 500); 12 Jul 2015 03:54:06 -0000 Delivered-To: apmail-stratos-dev-archive@stratos.apache.org Received: (qmail 17662 invoked by uid 500); 12 Jul 2015 03:54:06 -0000 Mailing-List: contact dev-help@stratos.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@stratos.apache.org Delivered-To: mailing list dev@stratos.apache.org Received: (qmail 17648 invoked by uid 99); 12 Jul 2015 03:54:05 -0000 Received: from Unknown (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 12 Jul 2015 03:54:05 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 3A1DE18285C for ; Sun, 12 Jul 2015 03:54:05 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 4.15 X-Spam-Level: **** X-Spam-Status: No, score=4.15 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, HTML_MESSAGE=3, KAM_LIVE=1, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-us-east.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id 1fT0YoZMzXEh for ; Sun, 12 Jul 2015 03:53:52 +0000 (UTC) Received: from mail-ig0-f170.google.com (mail-ig0-f170.google.com [209.85.213.170]) by mx1-us-east.apache.org (ASF Mail Server at mx1-us-east.apache.org) with ESMTPS id D947243DF1 for ; Sun, 12 Jul 2015 03:53:51 +0000 (UTC) Received: by iggp10 with SMTP id p10so57485936igg.0 for ; Sat, 11 Jul 2015 20:53:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type; bh=GN7j7a2hGE0BuQECnszRlTKaPWhzVURWEV7trzBoMNY=; b=mqXjDAtEUAufz8fuT6uQ+wsfLFu/KXcU4dhgYxaHRAZHLRJ+L2h1/T9fVdy5jWfNPy rYfxeFQ0b+Dgnv7o+iwA+S8Doc81x5cFPAY2pniurW8NqadcSuLLh/O3MD/91riQp1E6 iaYTaJZB7MBqoBZX/GaTwCQ/tEmwD+VDCKrGAt9kisk8FAKON2TD/aiEoipdP4hiXnW9 ja/eIWCqisBbh4xFtRDzYAmU2YsvlDcGKmIUqDEWC2tYot7XilOKxsCwbBfBxplGY9WT XxrlmFY5+rz3z1y4JaQCEpWSfiHDX3es4es/5x474StKiKxAzj9s9ASyNIOWBT3s5Htf g/aw== X-Received: by 10.50.171.228 with SMTP id ax4mr354026igc.32.1436673231307; Sat, 11 Jul 2015 20:53:51 -0700 (PDT) MIME-Version: 1.0 Received: by 10.107.4.195 with HTTP; Sat, 11 Jul 2015 20:53:36 -0700 (PDT) In-Reply-To: References: From: Kishanthan Thangarajah Date: Sun, 12 Jul 2015 09:23:36 +0530 Message-ID: Subject: Re: Stratos on Kubernetes To: dev@stratos.apache.org Content-Type: multipart/alternative; boundary=089e0111d4109c4ad1051aa58d3a --089e0111d4109c4ad1051aa58d3a Content-Type: text/plain; charset=UTF-8 Thanks Akila and Imesh. After updating, I actually redid the kubernetes setup again. Now I could see that all three nodes are started properly in the vbox. I also could deploy the php cartridge app using startos on kubernetes successfully. Few things I noticed while trying out this. 1. When the sample app was deployed on stratos, some times the deployment ended up with the following error. Looks like a similar issue was already reported in : http://markmail.org/message/geuinnqjameg7c5f. But I had the docker image downloaded on both minions. I'm not quite sure on why this was popping up a few times. Could this be due to network issue or delay is starting the container? TID: [0] [STRATOS] [2015-07-11 23:15:18,006] INFO {org.apache.stratos.cloud.controller.iaases.kubernetes.KubernetesIaas} - Waiting pod status to be changed to running: [application] single-cartridge-app [cartridge] php [member] single-cartridge-app.my-php.php.domain91f5d7e0-6ce0-45eb-9e83-ab40d32d63eb [pod] pod-1 TID: [0] [STRATOS] [2015-07-11 23:15:18,006] ERROR {org.apache.stratos.cloud.controller.iaases.kubernetes.KubernetesIaas} - Pod status did not change to running within 60 sec: [application] single-cartridge-app [cartridge] php [member] single-cartridge-app.my-php.php.domain91f5d7e0-6ce0-45eb-9e83-ab40d32d63eb [pod] pod-1 TID: [0] [STRATOS] [2015-07-11 23:15:18,007] ERROR {org.apache.stratos.cloud.controller.iaases.kubernetes.KubernetesIaas} - Could not start container: [application] single-cartridge-app [cartridge] php [member] single-cartridge-app.my-php.php.domain91f5d7e0-6ce0-45eb-9e83-ab40d32d63eb java.lang.RuntimeException: Pod status did not change to running within 60 sec: [application] single-cartridge-app [cartridge] php [member] single-cartridge-app.my-php.php.domain91f5d7e0-6ce0-45eb-9e83-ab40d32d63eb [pod] pod-1 at org.apache.stratos.cloud.controller.iaases.kubernetes.KubernetesIaas.waitForPodToBeActivated(KubernetesIaas.java:341) at org.apache.stratos.cloud.controller.iaases.kubernetes.KubernetesIaas.startContainer(KubernetesIaas.java:226) at org.apache.stratos.cloud.controller.iaases.kubernetes.KubernetesIaas.startInstance(KubernetesIaas.java:125) at org.apache.stratos.cloud.controller.services.impl.InstanceCreator.startInstance(InstanceCreator.java:109) at org.apache.stratos.cloud.controller.services.impl.InstanceCreator.run(InstanceCreator.java:68) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 2. If the app is undeployed, shouldn't the containers ("php" service in this case) also be stopped / removed from the minions? Noticed that the containers were still up and running when app was undeployed. 3. Observed the following error sometimes during app undeployment using the script file. Due to this, the app could not be deployed again. This was only the app deployed on stratos at this time. Saw a similar issue reported with STRATOS-1281. TID: [0] [STRATOS] [2015-07-12 01:53:18,956] ERROR {org.apache.stratos.rest.endpoint.api.StratosApiV41Utils} - Cannot remove cartridge : [cartridge-type] php since it is used in another cartridge group or an application TID: [0] [STRATOS] [2015-07-12 01:53:18,958] ERROR {org.apache.stratos.rest.endpoint.handlers.CustomExceptionMapper} - Cannot remove cartridge : [cartridge-type] php since it is used in another cartridge group or an application org.apache.stratos.rest.endpoint.exception.RestAPIException: Cannot remove cartridge : [cartridge-type] php since it is used in another cartridge group or an application at org.apache.stratos.rest.endpoint.api.StratosApiV41Utils.removeCartridge(StratosApiV41Utils.java:248) at org.apache.stratos.rest.endpoint.api.StratosApiV41.removeCartridge(StratosApiV41.java:431) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.cxf.service.invoker.AbstractInvoker.performInvocation(AbstractInvoker.java:180) On Fri, Jul 10, 2015 at 8:40 PM, Akila Ravihansa Perera wrote: > Adding one more point to it. > > There was a problem in one of the 'system' commands in Dockerfile causing > connection timeouts. > > "The problem is that each of the system statements in the Vagrantfile is > executed in a separate subshell. This means that the environment variables > are not being exported between calls, hence kubectl does not know the > $KUBERNETES_MASTER address." > > This has been fixed in upstream repo with [1]. > > [1] > https://github.com/pires/kubernetes-vagrant-coreos-cluster/pull/122/files > > On Fri, Jul 10, 2015 at 6:06 PM, Imesh Gunaratne wrote: > >> Thanks Akila! I have now merged it. >> >> For others information, the problem was that when the internet connection >> is slow, the Kubernetes installation process get restarted after reaching >> the timeout. Now Akila has increased it to 400 seconds. >> >> On Fri, Jul 10, 2015 at 10:32 AM, Akila Ravihansa Perera < >> ravihansa@wso2.com> wrote: >> >>> Hi Kishanthan, >>> >>> Actually your master node has not been configured properly. See the logs >>> at >>> >>> *==> master: Waiting for Kubernetes master to become ready...* >>> >>> The connection to the server localhost:8080 was refused - did you >>> specify the right host or port? >>> >>> >>> Also please check whether you are running on latest Vagrant version. >>> >>> @Imesh: this is a known issue and has been fixed in upstream repo. I've >>> sent you a PR with the fix at [1]. >>> >>> [1] https://github.com/imesh/kubernetes-vagrant-setup/pull/1 >>> >>> Thanks. >>> >>> >>> On Fri, Jul 10, 2015 at 9:29 AM, Imesh Gunaratne >>> wrote: >>> >>>> Hi Kishanthan, >>>> >>>> You could try to ssh into master and node-01 and execute journalctl -f >>>> to view the log. Kubernetes installation may have failed in one of the >>>> hosts. >>>> >>>> On Fri, Jul 10, 2015 at 9:07 AM, Kishanthan Thangarajah < >>>> kshanth2101@gmail.com> wrote: >>>> >>>>> Hi Devs, >>>>> >>>>> I was trying out the steps provided in : >>>>> https://gist.github.com/imesh/b8f81fac8de39183a504 >>>>> >>>>> I couldn't get the vagrant up and running with minions which fails >>>>> with the error at the bottom. >>>>> >>>>> I assume node-01 here is one of the minion. From the logs, I think the >>>>> master node is correctly configured, even with the version constraint. But >>>>> the node-01 had failed on the version constraint while adding the core-os. >>>>> Is this a known issue? Do I have to change/remove the version constraint in >>>>> some config file? >>>>> >>>>> kubernetes-vagrant-setup kishanthan$ vagrant up >>>>> >>>>> Bringing machine 'master' up with 'virtualbox' provider... >>>>> >>>>> Bringing machine 'node-01' up with 'virtualbox' provider... >>>>> >>>>> Bringing machine 'node-02' up with 'virtualbox' provider... >>>>> >>>>> *==> master: Running triggers before up...* >>>>> >>>>> *==> master: Setting Kubernetes version 0.18.0* >>>>> >>>>> *==> master: Configuring Kubernetes cluster DNS...* >>>>> >>>>> *==> master: Box 'coreos-alpha' could not be found. Attempting to find >>>>> and install...* >>>>> >>>>> master: Box Provider: virtualbox >>>>> >>>>> master: Box Version: >= 738.1.0 >>>>> >>>>> *==> master: Loading metadata for box >>>>> 'http://alpha.release.core-os.net/amd64-usr/current/coreos_production_vagrant.json >>>>> '* >>>>> >>>>> master: URL: >>>>> http://alpha.release.core-os.net/amd64-usr/current/coreos_production_vagrant.json >>>>> >>>>> *==> master: Adding box 'coreos-alpha' (v738.1.0) for provider: >>>>> virtualbox* >>>>> >>>>> master: Downloading: >>>>> http://alpha.release.core-os.net/amd64-usr/738.1.0/coreos_production_vagrant.box >>>>> >>>>> master: Calculating and comparing box checksum... >>>>> >>>>> *==> master: Successfully added box 'coreos-alpha' (v738.1.0) for >>>>> 'virtualbox'!* >>>>> >>>>> >>>>> *-----------------------------------------------------------------------------------------------------------------------* >>>>> >>>>> *==> master: Waiting for Kubernetes master to become ready...* >>>>> >>>>> The connection to the server localhost:8080 was refused - did you >>>>> specify the right host or port? >>>>> >>>>> The connection to the server localhost:8080 was refused - did you >>>>> specify the right host or port? >>>>> >>>>> *==> node-01: Box 'coreos-alpha' could not be found. Attempting to >>>>> find and install...* >>>>> >>>>> node-01: Box Provider: virtualbox >>>>> >>>>> node-01: Box Version: >= 738.1.0 >>>>> >>>>> You specified a box version constraint with a direct box file >>>>> >>>>> path. Box version constraints only work with boxes from Vagrant >>>>> >>>>> Cloud or a custom box host. Please remove the version constraint >>>>> >>>>> and try again. >>>>> >>>> >>>> >>>> >>>> -- >>>> Imesh Gunaratne >>>> >>>> Senior Technical Lead, WSO2 >>>> Committer & PMC Member, Apache Stratos >>>> >>> >>> >>> >>> -- >>> Akila Ravihansa Perera >>> Software Engineer, WSO2 >>> >>> Blog: http://ravihansa3000.blogspot.com >>> >> >> >> >> -- >> Imesh Gunaratne >> >> Senior Technical Lead, WSO2 >> Committer & PMC Member, Apache Stratos >> > > > > -- > Akila Ravihansa Perera > Software Engineer, WSO2 > > Blog: http://ravihansa3000.blogspot.com > --089e0111d4109c4ad1051aa58d3a Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Thanks Akila and Imesh.

After updating,= I actually redid the kubernetes setup again. Now I could see that all thre= e nodes are started properly in the vbox. I also could deploy the php cartr= idge app using startos on kubernetes successfully.

Few things I noticed while trying out this.

1. Wh= en the sample app was deployed on stratos, some times the deployment ended = up with the following error. Looks like a similar issue was already reporte= d in :=C2=A0http://markmail.org/message/geuinnqjameg7c5f. But I had the= docker image downloaded on both minions. I'm not quite sure on why thi= s was popping up a few times. Could this be due to network issue or delay i= s starting the container?

TID: [0] [STRATOS] [2015-07-11 23:15:18,006]=C2= =A0 INFO {org.apache.stratos.cloud.controller.iaases.kubernetes.KubernetesI= aas} -=C2=A0 Waiting pod status to be changed to running: [application] sin= gle-cartridge-app [cartridge] php [member] single-cartridge-app.my-php.php.= domain91f5d7e0-6ce0-45eb-9e83-ab40d32d63eb [pod] pod-1

TID: [0] [STRATOS] [2015= -07-11 23:15:18,006] ERROR {org.apache.stratos.cloud.controller.iaases.kube= rnetes.KubernetesIaas} -=C2=A0 Pod status did not change to running within = 60 sec: [application] single-cartridge-app [cartridge] php [member] single-= cartridge-app.my-php.php.domain91f5d7e0-6ce0-45eb-9e83-ab40d32d63eb [pod] p= od-1

TID: [0] [STRATOS] [2015-07-11 23:15:18,007] ERROR {org.apache.stratos.clo= ud.controller.iaases.kubernetes.KubernetesIaas} -=C2=A0 Could not start con= tainer: [application] single-cartridge-app [cartridge] php [member] single-= cartridge-app.my-php.php.domain91f5d7e0-6ce0-45eb-9e83-ab40d32d63eb

java.lang.R= untimeException: Pod status did not change to running within 60 sec: [appli= cation] single-cartridge-app [cartridge] php [member] single-cartridge-app.= my-php.php.domain91f5d7e0-6ce0-45eb-9e83-ab40d32d63eb [pod] pod-1

=

at org.apache.stratos.cloud.controller.ia= ases.kubernetes.KubernetesIaas.waitForPodToBeActivated(KubernetesIaas.java:= 341)

at org.apache.stratos.cloud.c= ontroller.iaases.kubernetes.KubernetesIaas.startContainer(KubernetesIaas.ja= va:226)

at org.apache.stratos.clou= d.controller.iaases.kubernetes.KubernetesIaas.startInstance(KubernetesIaas.= java:125)

at org.apache.stratos.cl= oud.controller.services.impl.InstanceCreator.startInstance(InstanceCreator.= java:109)

at org.apache.stratos.cl= oud.controller.services.impl.InstanceCreator.run(InstanceCreator.java:68)

at java.util.concurrent.ThreadPoolE= xecutor.runWorker(ThreadPoolExecutor.java:1145)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadP= oolExecutor.java:615)

at java.lang= .Thread.run(Thread.java:745)


2. I= f the app is undeployed, shouldn't the containers ("php" serv= ice in this case) also be stopped / removed from the minions? Noticed that = the containers were still up and running when app was undeployed.

3. Observed the following error sometimes during app undepl= oyment using the script file. Due to this, the app could not be deployed ag= ain. This was only the app deployed on stratos at this time. Saw a similar = issue reported with STRATOS-1281.

TID: [0] [STRATOS] [2015-07-12 01:53:18,956] = ERROR {org.apache.stratos.rest.endpoint.api.StratosApiV41Utils} -=C2=A0 Can= not remove cartridge : [cartridge-type] php since it is used in another car= tridge group or an application

TID: [0] [STRATOS] [2015-07-12 01:53:18,958] ERR= OR {org.apache.stratos.rest.endpoint.handlers.CustomExceptionMapper} -=C2= =A0 Cannot remove cartridge : [cartridge-type] php since it is used in anot= her cartridge group or an application

org.apache.stratos.rest.endpoint.exceptio= n.RestAPIException: Cannot remove cartridge : [cartridge-type] php since it= is used in another cartridge group or an application

at org.apache.stratos.rest.endpoint.api.StratosApiV41Ut= ils.removeCartridge(StratosApiV41Utils.java:248)

at org.apache.stratos.rest.endpoint.api.StratosApiV41.remove= Cartridge(StratosApiV41.java:431)

= at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke= (NativeMethodAccessorImpl.java:57)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccesso= rImpl.java:43)

at java.lang.reflec= t.Method.invoke(Method.java:606)

a= t org.apache.cxf.service.invoker.AbstractInvoker.performInvocation(Abstract= Invoker.java:180)




On Fri, Jul= 10, 2015 at 8:40 PM, Akila Ravihansa Perera <ravihansa@wso2.com>= wrote:
Adding on= e more point to it.

There was a problem in one of the &#= 39;system' commands in Dockerfile causing connection timeouts.

"The problem is that each of the system statements in the = Vagrantfile is executed in a separate subshell. This means that the environ= ment variables are not being exported between calls, hence kubectl does not= know the $KUBERNETES_MASTER address."

This has bee= n fixed in upstream repo with [1].=C2=A0


On Fri, J= ul 10, 2015 at 6:06 PM, Imesh Gunaratne <imesh@apache.org> wr= ote:
Thanks Akila! I hav= e now merged it.=C2=A0

For others information, the probl= em was that when the internet connection is slow, the Kubernetes installati= on process get restarted after reaching the timeout. Now Akila has increase= d it to 400 seconds.

On Fri, Jul 10, 2015 at 10:32 AM, Akila Ravihansa Perera= <ravihansa@wso2.com> wrote:
Hi=C2=A0Kishanthan,

Actually y= our master node has not been configured properly. See the logs at

=3D=3D> master: Waiting for Kubernetes master to= become ready...

The connection to the server localhost:8080 was = refused - did you specify the right host or port?



=
Also please check whether you are running on latest Vagra= nt version.

@Imesh: this is a known issue and has been = fixed in upstream repo. I've sent you a PR with the fix at [1].

Thanks.


= On Fri, Jul 10, 2015 at 9:29 AM, Imesh Gunaratne <imesh@apache.org><= /span> wrote:
Hi Kishant= han,

You could try to ssh into master and node-01 and ex= ecute journalctl -f to view the log. Kubernetes installation may have faile= d in one of the hosts.

=
On Fri, Jul 10, 2015 at 9:07 AM, Kishanthan Than= garajah <kshanth2101@gmail.com> wrote:
Hi Devs,

I was trying out the step= s provided in : https://gist.github.com/imesh/b8f81fac8de39183a504

I couldn't get the vagrant up and running with minions which f= ails with the error at the bottom.=C2=A0

I assume node-0= 1 here is one of the minion. From the logs, I think the master node is corr= ectly configured, even with the version constraint. But the node-01 had fai= led on the version constraint while adding the core-os. Is this a known iss= ue? Do I have to change/remove the version constraint in some config file?<= /div>

kubernetes-vagrant-setup kishanthan$ vagrant up

Bringing machine &= #39;master' up with 'virtualbox' provider...

Bringing machine &= #39;node-01' up with 'virtualbox' provider...

Bringing machine &= #39;node-02' up with 'virtualbox' provider...

=3D=3D> mast= er: Running triggers before up...

=3D=3D> mast= er: Setting Kubernetes version 0.18.0

=3D=3D> mast= er: Configuring Kubernetes cluster DNS...

=3D=3D> mast= er: Box 'coreos-alpha' could not be found. Attempting to find and i= nstall...

=C2=A0 =C2=A0 mast= er: Box Provider: virtualbox

=C2=A0 =C2=A0 mast= er: Box Version: >=3D 738.1.0

=3D=3D> mast= er: Loading metadata for box 'http= ://alpha.release.core-os.net/amd64-usr/current/coreos_production_vagrant.js= on'

=C2=A0 =C2=A0 mast= er: URL: http://alpha.release.core-os.= net/amd64-usr/current/coreos_production_vagrant.json

=3D=3D> mast= er: Adding box 'coreos-alpha' (v738.1.0) for provider: virtualbox

=C2=A0 =C2=A0 mast= er: Downloading: http://alpha.release.c= ore-os.net/amd64-usr/738.1.0/coreos_production_vagrant.box

=C2=A0 =C2=A0 mast= er: Calculating and comparing box checksum...

=3D=3D> master: Successfully added box 'coreos-alpha' (v73= 8.1.0) for 'virtualbox'!


-----= ---------------------------------------------------------------------------= ---------------------------------------

=3D=3D> master: Waiting for K= ubernetes master to become ready...

The connection to = the server localhost:8080 was refused - did you specify the right host or p= ort?

The connection to = the server localhost:8080 was refused - did you specify the right host or p= ort?

=3D=3D> node= -01: Box 'coreos-alpha' could not be found. Attempting to find and = install...

=C2=A0 =C2=A0 node= -01: Box Provider: virtualbox

=C2=A0 =C2=A0 node= -01: Box Version: >=3D 738.1.0

You specified a box version constraint with a direct box file

path. Box version constraints only work with boxes from Vagrant

Cloud or a custom box host. Please remove the version constraint

and try again.




<= font color=3D"#888888">--
I= mesh Gunaratne

Senior Technical Lead, WSO2
Committer & PMC Member, Apache Stratos<= /font>



<= font color=3D"#888888">--
Akila Ravihansa Perera
Software Engin= eer, WSO2

Blog: http://ravihansa3000.blogspot.com



--
Imesh Gunaratne

Senior Technical Lead, WSO2
Committer = & PMC Member, Apache Stratos



--
Akila R= avihansa Perera
Software Engineer, WSO2

Blog: http://ravihansa3000.blogspot.= com

--089e0111d4109c4ad1051aa58d3a--