Chamila,

Here is a container agent log. Any idea what event processing fails?

[2014-11-08 04:20:46,152] DEBUG {eventsubscriber.py:on_message} - Message received: 'topology/CompleteTopologyEvent':
'{"topology":{"serviceMap":{"php":{"serviceName":"php","serviceType":"SingleTenant","clusterIdClusterMap":{"myphp.php.domain":{"serviceName":"php","clusterId":"myphp.php.domain","autoscalePolicyName":"economy","hostNames":["myphp.apachestratos.org"],"tenantRange":"*","isLbCluster":false,"isKubernetesCluster":true,"memberMap":{"bc4ff2cb-66fd-11e4-847b-080027fb95c5":{"serviceName":"php","clusterId":"myphp.php.domain","memberId":"bc4ff2cb-66fd-11e4-847b-080027fb95c5","initTime":1415420070103,"portMap":{"8281":{"protocol":"http","value":4500,"proxy":8281}},"memberPublicIp":"172.17.8.101","memberIp":"172.17.8.101","properties":{"ALLOCATED_SERVICE_HOST_PORT":"4500"},"memberStateManager":{"stateStack":["Created"],"identifier":"bc4ff2cb-66fd-11e4-847b-080027fb95c5"}}},"appId":"test_app_2","properties":{"payload_parameter.APP_PATH":"/var/www/www","KUBERNETES_REPLICAS_MAX":"20","KUBERNETES_CLUSTER_ID":"KubGrp1","payload_parameter.LOG_LEVEL":"DEBUG","KUBERNETES_REPLICAS_MIN":"3","payload_parameter.MB_IP":"172.17.42.1","payload_parameter.CEP_IP":"172.17.42.1","payload_parameter.MB_PORT":"1883","payload_parameter.CEP_PORT":"7711"},"clusterStateManager":{"stateStack":["Created"],"identifier":"myphp.php.domain"}}},"portMap":{"8281":{"protocol":"http","value":80,"proxy":8281}},"properties":{}}},"initialized":false}}'
[2014-11-08 04:20:46,152] DEBUG {eventsubscriber.py:on_message} - Executing handler for event 'CompleteTopologyEvent'
[2014-11-08 04:20:46,153] DEBUG {agent.py:on_complete_topology} - Complete topology event received
[2014-11-08 04:20:46,153] ERROR {eventsubscriber.py:on_message} - Error processing 'CompleteTopologyEvent' event
Traceback (most recent call last):
  File "/mnt/cartridgeagent/modules/subscriber/eventsubscriber.py", line 81, in on_message
    handler(msg)
  File "agent.py", line 252, in on_complete_topology
    event_obj = CompleteTopologyEvent.create_from_json(msg.payload)
  File "/mnt/cartridgeagent/modules/event/topology/events.py", line 173, in create_from_json
    cluster_obj.status = cluster_str["status"]
KeyError: 'status'
[2014-11-08 04:20:46,159] DEBUG {eventsubscriber.py:on_message} - Message received: 'tenant/CompleteTenantEvent':

Thanks.

On Fri, Nov 7, 2014 at 11:10 PM, Imesh Gunaratne <imesh@apache.org> wrote:
As I found this problem this has been caused by the duplicated client id values which we have set before. Now I have done a fix to set unique client id values for each topic subscriber and publisher.

I did several verification tests on this fix using a sample event publisher (sending 1000 events continuously) and with 30 subscribers. Alll subscribers were able to receive 1000 events without a problem.

This is now committed to docker-grouping-merge branch.

On Fri, Nov 7, 2014 at 2:48 PM, Imesh Gunaratne <imesh@apache.org> wrote:
I guess we need to set clean session to false and use reliable delivery. WDYT?

MqttConnectOptions connOpts = new MqttConnectOptions();
connOpts.setCleanSession(false);

* <ul>
* <li>If set to false both the client and server will maintain state across
* restarts of the client, the server and the connection. As state is maintained:
* <ul>
* <li>Message delivery will be reliable meeting
* the specified QOS even if the client, server or connection are restarted.
* <li> The server will treat a subscription as durable.
* </ul>

On Fri, Nov 7, 2014 at 2:45 PM, Imesh Gunaratne <imesh@apache.org> wrote:
Gayan: I see another issue here:

We have set QOS to 2 in publisher:

MqttMessage mqttMSG = new MqttMessage(message.getBytes());
mqttMSG.setQos(QOS); // QOS => 2
                
* <li>Quality of Service 2 - indicates that a message should
* be delivered once.  The message will be persisted to disk, and will
* be subject to a two-phase acknowledgement across the network.
* The message can only be delivered safely if
* it can be persisted, so the application must supply a means of
* persistence using <code>MqttConnectOptions</code>.
* If a persistence mechanism is not specified, the message will not be
* delivered in the event of a client failure.</li>

Here we have set clean session option to true in connect options:

MqttConnectOptions connOpts = new MqttConnectOptions();
connOpts.setCleanSession(true);

* <lI>If set to true the client and server will not maintain state across
* restarts of the client, the server or the connection. This means
* <ul>
* <li>Message delivery to the specified QOS cannot be maintained if the
* client, server or connection are restarted
* <lI>The server will treat a subscription as non-durable
* </ul>

As I see these two configurations contradicts. May be this is why we see message delivery fails intermittently. WDYT?

On Fri, Nov 7, 2014 at 2:29 PM, Gayan Gunarathne <gayang@wso2.com> wrote:

At a particular time we do have multiple listeners and only one topic publisher for a component. Because of that generating a client id for subscribers. 

+1 for have client id unique according to this



Thanks,
Gayan

On Fri, Nov 7, 2014 at 2:14 PM, Imesh Gunaratne <imesh@apache.org> wrote:
Gayan: Was there any reason for generating a client id for subscribers and setting a fixed value for publishers?

TopicSubscriber.doSubscribe():
MqttClient mqttClient = MQTTConnector.getMQTTSubClient(Util.getRandomString(5));





On Fri, Nov 7, 2014 at 1:17 PM, Isuru Haththotuwa <isuruh@apache.org> wrote:


On Fri, Nov 7, 2014 at 12:59 PM, Isuru Haththotuwa <isuruh@apache.org> wrote:

On Fri, Nov 7, 2014 at 12:52 PM, Rajkumar Rajaratnam <rajkumarr@wso2.com> wrote:
I got containers to be created.
But didn't get the member started event. Either agent is not sending or CC is not receiving. Looking into it currently.
I'm seeing this issue as well. The agent is sending the events, but Stratos is not receiving them. Trying with a simple mqtt client now.
In [1], we are using the client id in mqtttopic.properties file as it is and using it to start the MQTTClient. However, in the wiki it says that client id should be unique across all publishers [2]. I might be missing something here, but is this case handled?

[1]. org.apache.stratos.messaging.broker.connect.MQTTConnector
 

On Fri, Nov 7, 2014 at 12:28 PM, Imesh Gunaratne <imesh@apache.org> wrote:
Hi Isuru,

On Fri, Nov 7, 2014 at 11:19 AM, Isuru Haththotuwa <isuruh@apache.org> wrote:

IMHO we should have the context name 'stratos' in the url. Then it is more intuitive. having just /api doesn't say that this API belongs to Stratos. 
Therefore, https://{hostname}:{port}/stratos/api/{resource-path} is better IMHO. If we have a proper hostname (stratos.apache.org) this might do the same, but that might not be the case always. Just my two cents. 


A good point, I reviewed your concern before doing this modification, most of the well known APIs do not include the product name in the context. Rather as you have mentioned they use the hostname to identify the product, please see below examples. This is why I removed "stratos" from the context, WDYT?


Thanks

On Fri, Nov 7, 2014 at 11:19 AM, Isuru Haththotuwa <isuruh@apache.org> wrote:
Hi Imesh,

On Fri, Nov 7, 2014 at 1:35 AM, Imesh Gunaratne <imesh@apache.org> wrote:
Please note that we have now introduced API versioning in docker-grouping-merge branch. Now the REST API context has been changed as follows, no changes were done to existing resource paths:

Previous API URL:
https://{hostname}:{port}/stratos/admin/{resource-path}

New API URL:
https://{hostname}:{port}/api/{resource-path}
IMHO we should have the context name 'stratos' in the url. Then it is more intuitive. having just /api doesn't say that this API belongs to Stratos.
Therefore, https://{hostname}:{port}/stratos/api/{resource-path} is better IMHO. If we have a proper hostname (stratos.apache.org) this might do the same, but that might not be the case always. Just my two cents.

Version specific API URLs:
https://{hostname}:{port}/api/v4.0/{resource-path}
https://{hostname}:{port}/api/v4.1/{resource-path}

Following changes to be done:
- Update CLI and Dashboard
- Allow API version to be specified in an HTTP header

Thanks

On Tue, Nov 4, 2014 at 5:22 PM, Imesh Gunaratne <imesh@apache.org> wrote:
We now have the docker-grouping-branch in a compilable state with latests modifications. Please test all workflows related to service grouping and docker, once things are stable will merge this branch to master.

Thanks

On Mon, Nov 3, 2014 at 7:48 PM, Manula Chathurika Thantriwatte <manulac@wso2.com> wrote:
Hi,

I have completed deploy-application, undeploy-application, deploy-service-group and undeploy-service-group commands in CLI and pushed the changes to docker-grouping-merge branch. I'll compete the other commands as soon as possible and looks for the UI as well.

Thanks !

On Mon, Nov 3, 2014 at 7:40 PM, Imesh Gunaratne <imesh@apache.org> wrote:
I have now fixed autoscaler service stub bundle exports and stratos manager import statements. Currently working on Stratos Manager, seems like there are duplicated domain classes:

org.apache.stratos.common/src/main/java/org/apache/stratos/common/Properties.java
org.apache.stratos.messaging/src/main/java/org/apache/stratos/messaging/util/Properties.java
org.apache.stratos.rest.endpoint/src/main/java/org/apache/stratos/rest/endpoint/bean/kubernetes/Properties.java

On Mon, Nov 3, 2014 at 3:24 PM, Rajkumar Rajaratnam <rajkumarr@wso2.com> wrote:
Fixed AS issues

On Mon, Nov 3, 2014 at 2:44 PM, Lasindu Charith <lasindu@wso2.com> wrote:
I have fixed some compilation errors in Cloud Controller.

On Mon, Nov 3, 2014 at 2:07 PM, Manula Chathurika Thantriwatte <manulac@wso2.com> wrote:
Hi,

I have started on grouping CLI commands.

Thanks !

On Mon, Nov 3, 2014 at 1:53 PM, Rajkumar Rajaratnam <rajkumarr@wso2.com> wrote:
Fixed messaging component issues.

On Mon, Nov 3, 2014 at 1:14 PM, Imesh Gunaratne <imesh@apache.org> wrote:
I have now merged latest changes from 4.0.0-grouping branch to docker-grouping-merge branch. We can now fix merge issues in docker-grouping-merge branch.

Once features are stable will move changes to master branch.

Thanks

On Sun, Nov 2, 2014 at 9:15 AM, Imesh Gunaratne <imesh@apache.org> wrote:
Isuru is working on a major refactoring task ("[Grouping] Branch may be Unstable ATM"). Will hold this effort until that is completed.

On Sun, Nov 2, 2014 at 12:19 AM, Lahiru Sandaruwan <lahirus@wso2.com> wrote:
I have tested grouping feature in EC2 up to instance are spawning. Will be testing further.

Thanks.

On Sat, Nov 1, 2014 at 9:57 PM, Rajkumar Rajaratnam <rajkumarr@wso2.com> wrote:
Hi,

When and where are we creating and adding cluster monitors (not app monitors) to the AutoscalerContext?
I couldn't find any places.

Thanks.

On Sat, Nov 1, 2014 at 9:37 PM, Lahiru Sandaruwan <lahirus@wso2.com> wrote:


On Sat, Nov 1, 2014 at 8:21 PM, Lahiru Sandaruwan <lahirus@wso2.com> wrote:
I get a NPE while trying to deploy the app definition without properties, looking into it.


I have added a null check. Will commit the fix. 

[2014-11-01 20:17:18,922] ERROR {org.apache.stratos.rest.endpoint.handlers.CustomThrowableExceptionMapper} -  Internal server error

java.lang.NullPointerException

at org.apache.stratos.rest.endpoint.services.ServiceUtils.deployApplicationDefinition(ServiceUtils.java:139)

at org.apache.stratos.rest.endpoint.services.StratosAdmin.deployApplicationDefinition(StratosAdmin.java:139)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:606)

at org.apache.cxf.service.invoker.AbstractInvoker.performInvocation(AbstractInvoker.java:180)

at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:96)

at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:194)

at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:100)

at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:57)

at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:93)

at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:271)

at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121)

at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:239)

at org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:223)

at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:203)

at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:137)

at org.apache.cxf.transport.servlet.CXFNonSpringServlet.invoke(CXFNonSpringServlet.java:159)

at org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:286)

at org.apache.cxf.transport.servlet.AbstractHTTPServlet.doPost(AbstractHTTPServlet.java:206)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:755)


On Sat, Nov 1, 2014 at 3:14 PM, Imesh Gunaratne <imesh@apache.org> wrote:
I have now fixed few more issues:
- Removing subscription resource from REST API and CLI.
- Adding isKubernetesCluster property to ApplicationClusterContext.
- Passing properties from Application definition to the cluster objects. This is needed for Kubernetes.
- Packaging drools files in stratos distribution (some of the files were placed in repository/conf while the rest were at repository/conf/drools).

Please find the sample application definition file which I have been using for testing attached with this mail.

On Sat, Nov 1, 2014 at 3:09 PM, Imesh Gunaratne <imesh@apache.org> wrote:
Can a service cluster(say php cluster) span across two partitions(in other words two kub cluster)? I guess kubernetes can manage their containers within a kub cluster, not between kub clusters? For example, if we say we need 3 replicas, it can ensure there are 3 replicas in one kub cluster. It can't manage 3 replicas within two kub cluster. Am I missing something here?

Raj: A good point, we can handle this in Stratos, shouldn't be a problem.



--
Imesh Gunaratne

Technical Lead, WSO2
Committer & PMC Member, Apache Stratos



--
--
Lahiru Sandaruwan
Committer and PMC member, Apache Stratos,
Senior Software Engineer,
WSO2 Inc., http://wso2.com



--
--
Lahiru Sandaruwan
Committer and PMC member, Apache Stratos,
Senior Software Engineer,
WSO2 Inc., http://wso2.com



--
Raj



--
--
Lahiru Sandaruwan
Committer and PMC member, Apache Stratos,
Senior Software Engineer,
WSO2 Inc., http://wso2.com



--
Imesh Gunaratne

Technical Lead, WSO2
Committer & PMC Member, Apache Stratos



--
Imesh Gunaratne

Technical Lead, WSO2
Committer & PMC Member, Apache Stratos



--
Raj



--
Regards,
Manula Chathurika Thantriwatte
Software Engineer
WSO2 Inc. : http://wso2.com
lean . enterprise . middleware







--
Lasindu Charith
Software Engineer, WSO2 Inc.
Mobile: +94714427192



--
Raj



--
Imesh Gunaratne

Technical Lead, WSO2
Committer & PMC Member, Apache Stratos



--
Regards,
Manula Chathurika Thantriwatte
Software Engineer
WSO2 Inc. : http://wso2.com
lean . enterprise . middleware







--
Imesh Gunaratne

Technical Lead, WSO2
Committer & PMC Member, Apache Stratos



--
Imesh Gunaratne

Technical Lead, WSO2
Committer & PMC Member, Apache Stratos

--
Thanks and Regards,

Isuru H.






--
Imesh Gunaratne

Technical Lead, WSO2
Committer & PMC Member, Apache Stratos



--



--
Imesh Gunaratne

Technical Lead, WSO2
Committer & PMC Member, Apache Stratos



--

Gayan Gunarathne
Technical Lead
WSO2 Inc. (http://wso2.com)
email  : gayang@wso2.com  | mobile : +94 766819985
 




--
Imesh Gunaratne

Technical Lead, WSO2
Committer & PMC Member, Apache Stratos



--
Imesh Gunaratne

Technical Lead, WSO2
Committer & PMC Member, Apache Stratos



--
Imesh Gunaratne

Technical Lead, WSO2
Committer & PMC Member, Apache Stratos



--
Raj