stratos-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Imesh Gunaratne <im...@apache.org>
Subject Re: [Discuss] Kubernetes workflow is now working!
Date Fri, 26 Dec 2014 11:40:56 GMT
I fixed the above issue by setting the private and public IPs to Kubernetes
Master IP, we will need to review few things here:
- Private IP address of a member needs to be the service proxy IP address
- However each service proxy will have a unique IP address (each transport
of the server)
- Which means there will be a private IP for each transport

Will discuss this and see how we can resolve this problem later on. For the
moment setting the Kubernetes Master IP is sufficient IMO.

After fixing the above issue, now I see the below error:

[2014-12-26 17:05:30,547] ERROR
{org.apache.stratos.autoscaler.event.receiver.topology.AutoscalerTopologyEventReceiver}
-  Error processing event null
java.lang.NullPointerException
at
org.apache.stratos.autoscaler.monitor.cluster.ClusterMonitor.getClusterInstanceContext(ClusterMonitor.java:1116)
at
org.apache.stratos.autoscaler.monitor.cluster.ClusterMonitor.handleMemberActivatedEvent(ClusterMonitor.java:875)
at
org.apache.stratos.autoscaler.event.receiver.topology.AutoscalerTopologyEventReceiver$12.onEvent(AutoscalerTopologyEventReceiver.java:393)
at
org.apache.stratos.messaging.listener.EventListener.update(EventListener.java:42)
at java.util.Observable.notifyObservers(Observable.java:159)
Thanks


On Fri, Dec 26, 2014 at 4:56 PM, Rajkumar Rajaratnam <rajkumarr@wso2.com>
wrote:

>
>
> On Fri, Dec 26, 2014 at 4:34 PM, Imesh Gunaratne <imesh@apache.org> wrote:
>
>> Hi Lakmal,
>>
>> I added these to Kubernetes Cluster definition for the time being since
>> I'm finding/fixing many issues in agent and docker base image, will discuss
>> and move them to a proper location later on. WDYT?
>>
>> {
>>       "clusterId": "kubernetes-cluster-1",
>>       "description": "Kubernetes CoreOS cluster",
>>       ...
>>     "property":[
>>        {
>>           "name":"payload_parameter.MB_IP",
>>           "value":"192.168.59.3"
>>        },
>>        {
>>           "name":"payload_parameter.MB_PORT",
>>           "value":"1883"
>>        },
>>        {
>>           "name":"payload_parameter.CEP_IP",
>>           "value":"192.168.59.3"
>>        },
>>        {
>>           "name":"payload_parameter.CEP_PORT",
>>           "value":"7711"
>>        },
>>        {
>>           "name":"payload_parameter.LOG_LEVEL",
>>           "value":"DEBUG"
>>        }
>>     ]
>> }
>>
>> After the above fix I'm now seeing the below error in cloud controller.
>> Seems like we do not send the ip address in the member activated event.
>> Need to check agent logic again:
>>
>> [2014-12-26 16:28:06,218]  INFO
>> {org.apache.stratos.manager.listener.InstanceStatusListener} -  Instance
>> status message received
>> [2014-12-26 16:28:06,218]  INFO
>> {org.apache.stratos.manager.listener.InstanceStatusListener} -  Event class
>> name:
>> org.apache.stratos.messaging.event.instance.status.InstanceActivatedEvent
>> [2014-12-26 16:28:06,219]  INFO
>> {org.apache.stratos.cloud.controller.messaging.topology.TopologyBuilder} -
>>  member started event adding status activated
>> [2014-12-26 16:28:06,225]  INFO
>> {org.apache.stratos.cloud.controller.messaging.publisher.TopologyEventPublisher}
>> -  Publishing member activated event: [service-name] php [cluster-id]
>> php.php.domain [cluster-instance-id] single-cartridge-app-1 [member-id]
>> php.php.domaind7655e6f-f962-425b-b93b-2bf7333dc37d [network-partition-id]
>>  [partition-id]
>> [2014-12-26 16:28:06,244] ERROR
>> {org.apache.stratos.messaging.message.receiver.topology.TopologyEventMessageDelegator}
>> -  Failed to retrieve topology event message
>> java.lang.RuntimeException: *No ip address found in member activated
>> event: *[service] php [cluster] php.php.domain [member]
>> php.php.domaind7655e6f-f962-425b-b93b-2bf7333dc37d
>> at
>> org.apache.stratos.messaging.message.processor.topology.MemberActivatedMessageProcessor.doProcess(MemberActivatedMessageProcessor.java:102)
>>
>
> I guess spawned containers doesn't have IP addresses. How we are getting
> IP address for a container is like this.
>
> // PodActivationWatcher.java
> String hostIP = pod.getCurrentState().getHost();
>
> This is returning null or empty, so member activated event is not having
> an IP.
>
> Thanks.
>
> Thanks.
>
>>
>> Thanks
>>
>> On Fri, Dec 26, 2014 at 3:49 PM, Lakmal Warusawithana <lakmal@wso2.com>
>> wrote:
>>
>>> Shall we used metadata service to publish all these generic information?
>>>
>>> On Fri, Dec 26, 2014 at 3:40 PM, Imesh Gunaratne <imesh@apache.org>
>>> wrote:
>>>
>>>> As I can see in the agent.conf some configuration parameters are
>>>> missing:
>>>>
>>>>
>>>> These parameters are not sent in Kuberbenetes workflow, we might need
>>>> to think how to send them. ASFAIK in VM workflow most of these missing
>>>> parameters are sent by puppet master.
>>>> ‚Äč
>>>>
>>>> On Fri, Dec 26, 2014 at 3:18 PM, Imesh Gunaratne <imesh@apache.org>
>>>> wrote:
>>>>
>>>>> Seems like we have a string formatting issue in agent:
>>>>>
>>>>>
>>>>> http://stackoverflow.com/questions/18053500/typeerror-not-all-arguments-converted-during-string-formatting-python
>>>>>
>>>>> On Fri, Dec 26, 2014 at 3:11 PM, Imesh Gunaratne <imesh@apache.org>
>>>>> wrote:
>>>>>
>>>>>> I fixed the above issue with INTERNAL parameter and now I'm seeing
>>>>>> the following error:
>>>>>>
>>>>>> [2014-12-26 09:37:27,321] INFO
>>>>>> {cartridgeagentconfiguration.py:__init__} - Cartridge agent configuration
>>>>>> initialized
>>>>>> Traceback (most recent call last):
>>>>>>   File "/usr/lib/python2.7/logging/__init__.py", line 850, in emit
>>>>>>     msg = self.format(record)
>>>>>>   File "/usr/lib/python2.7/logging/__init__.py", line 723, in format
>>>>>>     return fmt.format(record)
>>>>>>   File "/usr/lib/python2.7/logging/__init__.py", line 464, in format
>>>>>>     record.message = record.getMessage()
>>>>>>   File "/usr/lib/python2.7/logging/__init__.py", line 328, in
>>>>>> getMessage
>>>>>>     msg = msg % self.args
>>>>>> TypeError: not all arguments converted during string formatting
>>>>>> Logged from file agent.py, line 331
>>>>>> Traceback (most recent call last):
>>>>>>   File "/usr/lib/python2.7/logging/__init__.py", line 850, in emit
>>>>>>     msg = self.format(record)
>>>>>>   File "/usr/lib/python2.7/logging/__init__.py", line 723, in format
>>>>>>     return fmt.format(record)
>>>>>>   File "/usr/lib/python2.7/logging/__init__.py", line 464, in format
>>>>>>     record.message = record.getMessage()
>>>>>>   File "/usr/lib/python2.7/logging/__init__.py", line 328, in
>>>>>> getMessage
>>>>>>     msg = msg % self.args
>>>>>> TypeError: not all arguments converted during string formatting
>>>>>> Logged from file agent.py, line 331
>>>>>> Traceback (most recent call last):
>>>>>>   File "/usr/lib/python2.7/logging/__init__.py", line 850, in emit
>>>>>>     msg = self.format(record)
>>>>>>   File "/usr/lib/python2.7/logging/__init__.py", line 723, in format
>>>>>>     return fmt.format(record)
>>>>>>   File "/usr/lib/python2.7/logging/__init__.py", line 464, in format
>>>>>>     record.message = record.getMessage()
>>>>>>   File "/usr/lib/python2.7/logging/__init__.py", line 328, in
>>>>>> getMessage
>>>>>>     msg = msg % self.args
>>>>>> TypeError: not all arguments converted during string formatting
>>>>>> Logged from file agent.py, line 331
>>>>>>
>>>>>> On Fri, Dec 26, 2014 at 3:03 PM, Imesh Gunaratne <imesh@apache.org>
>>>>>> wrote:
>>>>>>
>>>>>>> As I found there was an issue in agent code, it was using PROVIDER
>>>>>>> constant to fetch the INTERNAL parameter. I cannot see PROVIDER
parameter
>>>>>>> being using in the agent.
>>>>>>>
>>>>>>> Thanks
>>>>>>>
>>>>>>> On Fri, Dec 26, 2014 at 2:32 PM, Gayan Gunarathne <gayang@wso2.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi Imesh,
>>>>>>>>
>>>>>>>> Attachment is the sample launch_params that I have tested.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Gayan
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, Dec 26, 2014 at 2:22 PM, Gayan Gunarathne <gayang@wso2.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi Imesh,
>>>>>>>>>
>>>>>>>>> Is that launch_params contains the PROVIDER parameter?
I guess
>>>>>>>>> this comes when the provider is not available in the
payload. Seems like we
>>>>>>>>> are showing wrong error message.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Gayan
>>>>>>>>>
>>>>>>>>> On Fri, Dec 26, 2014 at 2:05 PM, Imesh Gunaratne <imesh@apache.org
>>>>>>>>> > wrote:
>>>>>>>>>
>>>>>>>>>> Thanks Gayan, yes it was not there in launch_params,
I just fixed
>>>>>>>>>> it and seeing the below error:
>>>>>>>>>>
>>>>>>>>>> [2014-12-26 08:31:17,311] INFO
>>>>>>>>>> {cartridgeagentconfiguration.py:__init__} -  INTERNAL
payload parameter is
>>>>>>>>>> not found
>>>>>>>>>> [2014-12-26 08:31:17,311] INFO
>>>>>>>>>> {cartridgeagentconfiguration.py:__init__} - Cartridge
agent configuration
>>>>>>>>>> initialized
>>>>>>>>>> Traceback (most recent call last):
>>>>>>>>>>   File "/usr/lib/python2.7/logging/__init__.py",
line 850, in emit
>>>>>>>>>>     msg = self.format(record)
>>>>>>>>>>   File "/usr/lib/python2.7/logging/__init__.py",
line 723, in
>>>>>>>>>> format
>>>>>>>>>>     return fmt.format(record)
>>>>>>>>>>   File "/usr/lib/python2.7/logging/__init__.py",
line 464, in
>>>>>>>>>> format
>>>>>>>>>>     record.message = record.getMessage()
>>>>>>>>>>   File "/usr/lib/python2.7/logging/__init__.py",
line 328, in
>>>>>>>>>> getMessage
>>>>>>>>>>     msg = msg % self.args
>>>>>>>>>> TypeError: not all arguments converted during string
formatting
>>>>>>>>>>
>>>>>>>>>> On Fri, Dec 26, 2014 at 1:49 PM, Gayan Gunarathne
<
>>>>>>>>>> gayang@wso2.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Imesh,
>>>>>>>>>>>
>>>>>>>>>>> This is happen when CLUSTER_INSTANCE_ID not contain
as the
>>>>>>>>>>> payload parameter of the cartridge agent.I think
we can first check the
>>>>>>>>>>> agent payload and see CLUSTER_INSTANCE_ID is
there in the agent
>>>>>>>>>>> payload.
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Gayan
>>>>>>>>>>>
>>>>>>>>>>> On Fri, Dec 26, 2014 at 1:39 PM, Imesh Gunaratne
<
>>>>>>>>>>> imesh@apache.org> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> I fixed the above issue in cartridge agent
and re-created the
>>>>>>>>>>>> base image, now I'm seeing another error:
>>>>>>>>>>>>
>>>>>>>>>>>> root@e82ebdc6-8cd5-11e4-85d5-080027f35f1b:~#
cat
>>>>>>>>>>>> /tmp/agent.screen.log
>>>>>>>>>>>> Traceback (most recent call last):
>>>>>>>>>>>>   File "agent.py", line 32, in <module>
>>>>>>>>>>>>     from modules.extensions import defaultextensionhandler
>>>>>>>>>>>>   File
>>>>>>>>>>>> "/mnt/apache-stratos-python-cartridge-agent-4.1.0-SNAPSHOT/modules/extensions/defaultextensionhandler.py",
>>>>>>>>>>>> line 22, in <module>
>>>>>>>>>>>>     from ..util import extensionutils, cartridgeagentutils
>>>>>>>>>>>>   File
>>>>>>>>>>>> "/mnt/apache-stratos-python-cartridge-agent-4.1.0-SNAPSHOT/modules/util/extensionutils.py",
>>>>>>>>>>>> line 29, in <module>
>>>>>>>>>>>>     cartridge_agent_config =
>>>>>>>>>>>> cartridgeagentconfiguration.CartridgeAgentConfiguration()
>>>>>>>>>>>>   File
>>>>>>>>>>>> "/mnt/apache-stratos-python-cartridge-agent-4.1.0-SNAPSHOT/modules/config/cartridgeagentconfiguration.py",
>>>>>>>>>>>> line 349, in __init__
>>>>>>>>>>>>     CartridgeAgentConfiguration.instance
=
>>>>>>>>>>>> CartridgeAgentConfiguration.__CartridgeAgentConfiguration()
>>>>>>>>>>>>   File
>>>>>>>>>>>> "/mnt/apache-stratos-python-cartridge-agent-4.1.0-SNAPSHOT/modules/config/cartridgeagentconfiguration.py",
>>>>>>>>>>>> line 226, in __init__
>>>>>>>>>>>>     raise RuntimeError(ex)
>>>>>>>>>>>> RuntimeError: Cannot find the value of required
parameter:
>>>>>>>>>>>> 'CLUSTER_INSTANCE_ID'
>>>>>>>>>>>>
>>>>>>>>>>>> I can see CLUSTER_INSTANCE_ID being passed
to the container,
>>>>>>>>>>>> may be there is something wrong in the agent
code, will investigate.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks
>>>>>>>>>>>>
>>>>>>>>>>>> On Fri, Dec 26, 2014 at 2:40 AM, Imesh Gunaratne
<
>>>>>>>>>>>> imesh@apache.org> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Devs,
>>>>>>>>>>>>>
>>>>>>>>>>>>> I fixed some of the issues in Kubernetes
Iaas and tested the
>>>>>>>>>>>>> Kubernetes workflow with sample php docker
image. Now replication
>>>>>>>>>>>>> controllers, services and containers
get created without a problem. We can
>>>>>>>>>>>>> also access the php server in the container.
>>>>>>>>>>>>>
>>>>>>>>>>>>> However I noticed that agent does not
send the instance
>>>>>>>>>>>>> started event to CEP. Following error
was found in agent screen log:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Traceback (most recent call last):
>>>>>>>>>>>>>   File "agent.py", line 21, in <module>
>>>>>>>>>>>>>     from cartridgeagent.cartridgeagent.modules
import
>>>>>>>>>>>>> eventsubscriber
>>>>>>>>>>>>> ImportError: No module named
>>>>>>>>>>>>> cartridgeagent.cartridgeagent.modules
>>>>>>>>>>>>>
>>>>>>>>>>>>> May be agent was not updated in the php
docker image. Will
>>>>>>>>>>>>> check this tomorrow.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Imesh Gunaratne
>>>>>>>>>>>>>
>>>>>>>>>>>>> Technical Lead, WSO2
>>>>>>>>>>>>> Committer & PMC Member, Apache Stratos
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Imesh Gunaratne
>>>>>>>>>>>>
>>>>>>>>>>>> Technical Lead, WSO2
>>>>>>>>>>>> Committer & PMC Member, Apache Stratos
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>>
>>>>>>>>>>> Gayan Gunarathne
>>>>>>>>>>> Technical Lead
>>>>>>>>>>> WSO2 Inc. (http://wso2.com)
>>>>>>>>>>> email  : gayang@wso2.com  | mobile : +94 766819985
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Imesh Gunaratne
>>>>>>>>>>
>>>>>>>>>> Technical Lead, WSO2
>>>>>>>>>> Committer & PMC Member, Apache Stratos
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>>
>>>>>>>>> Gayan Gunarathne
>>>>>>>>> Technical Lead
>>>>>>>>> WSO2 Inc. (http://wso2.com)
>>>>>>>>> email  : gayang@wso2.com  | mobile : +94 766819985
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>>
>>>>>>>> Gayan Gunarathne
>>>>>>>> Technical Lead
>>>>>>>> WSO2 Inc. (http://wso2.com)
>>>>>>>> email  : gayang@wso2.com  | mobile : +94 766819985
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Imesh Gunaratne
>>>>>>>
>>>>>>> Technical Lead, WSO2
>>>>>>> Committer & PMC Member, Apache Stratos
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Imesh Gunaratne
>>>>>>
>>>>>> Technical Lead, WSO2
>>>>>> Committer & PMC Member, Apache Stratos
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Imesh Gunaratne
>>>>>
>>>>> Technical Lead, WSO2
>>>>> Committer & PMC Member, Apache Stratos
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Imesh Gunaratne
>>>>
>>>> Technical Lead, WSO2
>>>> Committer & PMC Member, Apache Stratos
>>>>
>>>
>>>
>>>
>>> --
>>> Lakmal Warusawithana
>>> Vice President, Apache Stratos
>>> Director - Cloud Architecture; WSO2 Inc.
>>> Mobile : +94714289692
>>> Blog : http://lakmalsview.blogspot.com/
>>>
>>>
>>
>>
>> --
>> Imesh Gunaratne
>>
>> Technical Lead, WSO2
>> Committer & PMC Member, Apache Stratos
>>
>
>
>
> --
> Rajkumar Rajaratnam
> Committer & PMC Member, Apache Stratos
> Software Engineer, WSO2
>
> Mobile : +94777568639
> Blog : rajkumarr.com
>



-- 
Imesh Gunaratne

Technical Lead, WSO2
Committer & PMC Member, Apache Stratos

Mime
View raw message