stratos-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From david hbase <davidhb...@gmail.com>
Subject Re: Stratos 4.0 can not deploy the Multi-Tenant Service
Date Tue, 10 Mar 2015 07:51:23 GMT
Hi, Team:

I have fixed all error, but LB still not send out the LB LB statistics to
Stratos:

1. Stratos logs:
[2015-03-10 07:46:24,855] DEBUG
{org.apache.stratos.autoscaler.monitor.ClusterMonitor} -  Scale rule will
not run since the LB statistics have not received before this cycle for
network partition openstack-region1
[2015-03-10 07:46:53,190] DEBUG
{org.apache.stratos.autoscaler.message.receiver.health.AutoscalerHealthStatEventReceiver}
-  Average Rif event: [cluster] lb.lb.domain [network-partition] zone-1
[value] 0.0
[2015-03-10 07:46:53,190] DEBUG
{org.apache.stratos.autoscaler.message.receiver.health.AutoscalerHealthStatEventReceiver}
-  Network partition context is not available for : [network partition]
zone-1
[2015-03-10 07:46:53,199] DEBUG
{org.apache.stratos.autoscaler.message.receiver.health.AutoscalerHealthStatEventReceiver}
-  Gradient of Rif event: [cluster] lb.lb.domain [network-partition] zone-1
[value] 0.0
[2015-03-10 07:46:53,199] DEBUG
{org.apache.stratos.autoscaler.message.receiver.health.AutoscalerHealthStatEventReceiver}
-  Network partition context is not available for : [network partition]
zone-1
[2015-03-10 07:46:53,393] DEBUG
{org.apache.stratos.autoscaler.message.receiver.health.AutoscalerHealthStatEventReceiver}
-  Second derivative of Rif event: [cluster] lb.lb.domain
[network-partition] zone-1 [value] 0.0
[2015-03-10 07:46:53,394] DEBUG
{org.apache.stratos.autoscaler.message.receiver.health.AutoscalerHealthStatEventReceiver}
-  Network partition context is not available for : [network partition]
zone-1
[2015-03-10 07:46:54,855] DEBUG
{org.apache.stratos.autoscaler.monitor.ClusterMonitor} -  Cluster monitor
is running.. ClusterMonitor [clusterId=lb.lb.domain, serviceId=lb,
deploymentPolicy=Deployment Policy [id]economyDeployment [partitions]
[org.apache.stratos.cloud.controller.stub.deployment.partition.Partition@274ba6a4],
autoscalePolicy=ASPolicy [id=economyPolicy, displayName=null,
description=null], lbReferenceType=null]
[2015-03-10 07:46:54,856] DEBUG
{org.apache.stratos.autoscaler.monitor.ClusterMonitor} -  Running minimum
check for partition zone-1
[2015-03-10 07:46:54,856] DEBUG
{org.apache.stratos.autoscaler.rule.RuleLog} -  Running minimum rule:
[network-partition] openstack-region1 [partition] zone-1
[2015-03-10 07:46:54,856] DEBUG
{org.apache.stratos.autoscaler.rule.RuleLog} -  [min-check]
[network-partition] openstack-region1 [partition] zone-1 Non terminated
member count: 1
[2015-03-10 07:46:54,856] DEBUG
{org.apache.stratos.autoscaler.rule.RuleLog} -  [min-check]
[network-partition] openstack-region1 [partition] zone-1 Minimum member
count: 1
[2015-03-10 07:46:54,857] DEBUG
{org.apache.stratos.autoscaler.rule.RuleLog} -  Running obsolete instances
rule: [partition] zone-1 [network-partition] openstack-region1
[2015-03-10 07:46:54,857] DEBUG
{org.apache.stratos.autoscaler.rule.RuleLog} -  [obsolete-check]
[network-partition] openstack-region1 [partition] zone-1 Obsolete member
count: 0
[2015-03-10 07:46:54,857] DEBUG
{org.apache.stratos.autoscaler.rule.AutoscalerRuleEvaluator} -  Minimum
check executed for : org.apache.stratos.autoscaler.PartitionContext@6bf159e8
[2015-03-10 07:46:54,857] DEBUG
{org.apache.stratos.autoscaler.monitor.ClusterMonitor} -  Scale rule will
not run since the LB statistics have not received before this cycle for
network partition openstack-region1
[2015-03-10 07:47:24,857] DEBUG
{org.apache.stratos.autoscaler.monitor.ClusterMonitor} -  Cluster monitor
is running.. ClusterMonitor [clusterId=lb.lb.domain, serviceId=lb,
deploymentPolicy=Deployment Policy [id]economyDeployment [partitions]
[org.apache.stratos.cloud.controller.stub.deployment.partition.Partition@274ba6a4],
autoscalePolicy=ASPolicy [id=economyPolicy, displayName=null,
description=null], lbReferenceType=null]
[2015-03-10 07:47:24,858] DEBUG
{org.apache.stratos.autoscaler.monitor.ClusterMonitor} -  Running minimum
check for partition zone-1
[2015-03-10 07:47:24,858] DEBUG
{org.apache.stratos.autoscaler.rule.RuleLog} -  Running minimum rule:
[network-partition] openstack-region1 [partition] zone-1
[2015-03-10 07:47:24,858] DEBUG
{org.apache.stratos.autoscaler.rule.RuleLog} -  [min-check]
[network-partition] openstack-region1 [partition] zone-1 Non terminated
member count: 1
[2015-03-10 07:47:24,859] DEBUG
{org.apache.stratos.autoscaler.rule.RuleLog} -  [min-check]
[network-partition] openstack-region1 [partition] zone-1 Minimum member
count: 1
[2015-03-10 07:47:24,859] DEBUG
{org.apache.stratos.autoscaler.rule.RuleLog} -  Running obsolete instances
rule: [partition] zone-1 [network-partition] openstack-region1
[2015-03-10 07:47:24,859] DEBUG
{org.apache.stratos.autoscaler.rule.RuleLog} -  [obsolete-check]
[network-partition] openstack-region1 [partition] zone-1 Obsolete member
count: 0
[2015-03-10 07:47:24,859] DEBUG
{org.apache.stratos.autoscaler.rule.AutoscalerRuleEvaluator} -  Minimum
check executed for : org.apache.stratos.autoscaler.PartitionContext@6bf159e8
[2015-03-10 07:47:24,859] DEBUG
{org.apache.stratos.autoscaler.monitor.ClusterMonitor} -  Scale rule will
not run since the LB statistics have not received before this cycle for
network partition openstack-region1


2. LB logs:
root@lblbdoma-59:~# /mnt/apache-stratos-load-balancer-4.0.0/bin/stratos.sh
JAVA_HOME environment variable is set to /usr/lib/jvm/java-7-openjdk-amd64
CARBON_HOME environment variable is set to
/mnt/apache-stratos-load-balancer-4.0.0
[2015-03-10 07:38:38,577]  INFO
{org.wso2.carbon.server.extensions.PatchInstaller} -  Patch changes
detected
[2015-03-10 07:38:40,233]  INFO
{org.wso2.carbon.server.util.PatchUtils.console} -  Backed up plugins to
patch0000
[2015-03-10 07:38:41,537]  INFO
{org.wso2.carbon.server.util.PatchUtils.console} -  Patch verification
started
[2015-03-10 07:38:41,831]  INFO
{org.wso2.carbon.server.util.PatchUtils.console} -  Patch verification
successfully completed without encountering any issues.
[2015-03-10 07:38:46,194]  INFO - CarbonCoreActivator Starting WSO2
Carbon...
[2015-03-10 07:38:46,196]  INFO - CarbonCoreActivator Operating System :
Linux 3.2.0-74-virtual, amd64
[2015-03-10 07:38:46,196]  INFO - CarbonCoreActivator Java Home        :
/usr/lib/jvm/java-7-openjdk-amd64/jre
[2015-03-10 07:38:46,197]  INFO - CarbonCoreActivator Java Version     :
1.7.0_75
[2015-03-10 07:38:46,197]  INFO - CarbonCoreActivator Java VM          :
OpenJDK 64-Bit Server VM 24.75-b04,Oracle Corporation
[2015-03-10 07:38:46,197]  INFO - CarbonCoreActivator Carbon Home      :
/mnt/apache-stratos-load-balancer-4.0.0
[2015-03-10 07:38:46,197]  INFO - CarbonCoreActivator Java Temp Dir    :
/mnt/apache-stratos-load-balancer-4.0.0/tmp
[2015-03-10 07:38:46,197]  INFO - CarbonCoreActivator User             :
root, en-US, Etc/UTC
[2015-03-10 07:38:46,349]  WARN - SystemValidator Could not validate the
system for configuration parameter : CPU
[2015-03-10 07:38:46,351]  WARN - ValidationResultPrinter RAM size (MB):
2003 of the system is below the recommended minimum size :2048
[2015-03-10 07:38:46,357]  WARN - ValidationResultPrinter The default
keystore (wso2carbon.jks) is currently being used. To maximize security
when deploying to a production environment, configure a new keystore with a
unique password in the production server profile.
[2015-03-10 07:38:46,357]  WARN - ValidationResultPrinter Swap Memory size
(MB): 0 of the system is below the recommended minimum size :2048
[2015-03-10 07:38:46,366]  INFO - AgentHolder Agent created !
[2015-03-10 07:38:46,411]  INFO - AgentDS Successfully deployed Agent Client
[2015-03-10 07:39:33,357]  INFO - EmbeddedRegistryService Configured
Registry in 225ms
[2015-03-10 07:39:33,847]  INFO - RegistryCoreServiceComponent Registry
Mode    : READ-WRITE
[2015-03-10 07:39:34,793]  INFO - UserStoreMgtDSComponent Carbon
UserStoreMgtDSComponent activated successfully.
[2015-03-10 07:39:34,972]  INFO - ClusterBuilder Clustering has been
disabled
[2015-03-10 07:39:35,273]  INFO - CarbonAxisConfigurator No modules
directory was found under
/mnt/apache-stratos-load-balancer-4.0.0/repository/deployment/server.
[2015-03-10 07:39:35,326]  INFO - PassThroughHttpSSLSender Initializing
Pass-through HTTP/S Sender...
[2015-03-10 07:39:35,344]  INFO - ClientConnFactoryBuilder HTTPS Loading
Identity Keystore from : repository/resources/security/wso2carbon.jks
[2015-03-10 07:39:35,349]  INFO - ClientConnFactoryBuilder HTTPS Loading
Trust Keystore from : repository/resources/security/client-truststore.jks
[2015-03-10 07:39:35,380]  INFO - PassThroughHttpSSLSender Pass-through
HTTPS Sender started...
[2015-03-10 07:39:35,380]  INFO - PassThroughHttpSender Initializing
Pass-through HTTP/S Sender...
[2015-03-10 07:39:35,381]  INFO - PassThroughHttpSender Pass-through HTTP
Sender started...
[2015-03-10 07:39:35,398]  INFO - PassThroughHttpSSLListener Initializing
Pass-through HTTP/S Listener...
[2015-03-10 07:39:35,415]  INFO - PassThroughHttpListener Initializing
Pass-through HTTP/S Listener...
[2015-03-10 07:39:35,501]  INFO - ModuleDeployer Deploying module:
addressing-1.6.1-wso2v10 -
file:/mnt/apache-stratos-load-balancer-4.0.0/repository/deployment/client/modules/addressing-1.6.1-wso2v10.mar
[2015-03-10 07:39:35,508]  INFO - ModuleDeployer Deploying module:
rampart-1.6.1-wso2v1 -
file:/mnt/apache-stratos-load-balancer-4.0.0/repository/deployment/client/modules/rampart-1.6.1-wso2v1.mar
[2015-03-10 07:39:35,513]  INFO - TCPTransportSender TCP Sender started
[2015-03-10 07:39:37,032]  INFO - CarbonServerManager Repository       :
/mnt/apache-stratos-load-balancer-4.0.0/repository/deployment/server/
[2015-03-10 07:39:37,195]  INFO - PermissionUpdater Permission cache
updated for tenant -1234
[2015-03-10 07:39:37,297]  INFO - ServiceBusInitializer Starting ESB...
[2015-03-10 07:39:37,335]  INFO - ServiceBusInitializer Initializing Apache
Synapse...
[2015-03-10 07:39:37,346]  INFO - SynapseControllerFactory Using Synapse
home : /mnt/apache-stratos-load-balancer-4.0.0/.
[2015-03-10 07:39:37,353]  INFO - SynapseControllerFactory Using
synapse.xml location :
/mnt/apache-stratos-load-balancer-4.0.0/././repository/deployment/server/synapse-configs/default
[2015-03-10 07:39:37,353]  INFO - SynapseControllerFactory Using server
name : localhost
[2015-03-10 07:39:37,360]  INFO - SynapseControllerFactory The timeout
handler will run every : 15s
[2015-03-10 07:39:37,371]  INFO - Axis2SynapseController Initializing
Synapse at : Tue Mar 10 07:39:37 UTC 2015
[2015-03-10 07:39:37,377]  INFO - CarbonSynapseController Loading the
mediation configuration from the file system
[2015-03-10 07:39:37,379]  INFO - MultiXMLConfigurationBuilder Building
synapse configuration from the synapse artifact repository at :
././repository/deployment/server/synapse-configs/default
[2015-03-10 07:39:37,385]  INFO - XMLConfigurationBuilder Generating the
Synapse configuration model by parsing the XML configuration
[2015-03-10 07:39:37,732]  INFO - SynapseConfigurationBuilder Loaded
Synapse configuration from the artifact repository at :
././repository/deployment/server/synapse-configs/default
[2015-03-10 07:39:37,735]  INFO - Axis2SynapseController Loading mediator
extensions...
[2015-03-10 07:39:37,736]  INFO - Axis2SynapseController Deploying the
Synapse service...
[2015-03-10 07:39:37,737]  INFO - Axis2SynapseController Deploying Proxy
services...
[2015-03-10 07:39:37,737]  INFO - Axis2SynapseController Deploying
EventSources...
[2015-03-10 07:39:37,762]  INFO - ServerManager Server ready for
processing...
[2015-03-10 07:39:37,850]  WARN - LoadBalancerConfiguration Endpoint
timeout not found, using default: 15000
[2015-03-10 07:39:37,852]  INFO - SynapseConfigurator Configuring synapse
main sequence...
[2015-03-10 07:39:37,938]  INFO - SynapseConfigurator Synapse main sequence
configured successfully
[2015-03-10 07:39:37,948]  INFO - LoadBalancerServiceComponent Tenant
receiver thread started
[2015-03-10 07:39:37,967]  INFO - LoadBalancerServiceComponent Topology
receiver thread started
[2015-03-10 07:39:37,967]  INFO - TenantEventMessageDelegator Tenant event
message delegator started
[2015-03-10 07:39:37,974]  INFO - LoadBalancerTopologyEventReceiver Load
balancer topology receiver thread started
[2015-03-10 07:39:38,060]  INFO - TopologyEventMessageDelegator Topology
event message delegator started
[2015-03-10 07:39:38,183]  INFO - LoadBalancerServiceComponent Load
balancer statistics notifier thread started
[2015-03-10 07:39:38,189]  INFO - LoadBalancerServiceComponent Load
balancer service component is activated
[2015-03-10 07:39:38,677]  INFO - PassThroughHttpSSLListener Starting
Pass-through HTTPS Listener...
[2015-03-10 07:39:38,701]  INFO - PassThroughHttpSSLListener Pass-through
HTTPS Listener started on 0:0:0:0:0:0:0:0:443
[2015-03-10 07:39:38,701]  INFO - PassThroughHttpListener Starting
Pass-through HTTP Listener...
[2015-03-10 07:39:38,709]  INFO - PassThroughHttpListener Pass-through HTTP
Listener started on 0:0:0:0:0:0:0:0:80
[2015-03-10 07:39:38,915]  INFO - TaskServiceImpl Task service starting in
STANDALONE mode...
[2015-03-10 07:39:39,026]  INFO - JMXServerManager JMX Service URL  :
service:jmx:rmi://localhost:11111/jndi/rmi://localhost:9999/jmxrmi
[2015-03-10 07:39:39,026]  INFO - StartupFinalizerServiceComponent
Server           :  Apache Stratos LB-4.0.0
[2015-03-10 07:39:39,027]  INFO - StartupFinalizerServiceComponent WSO2
Carbon started in 60 sec
[2015-03-10 07:39:39,317]  INFO - CarbonUIServiceComponent Mgt Console URL
: https://10.203.50.158:9443/carbon/
[2015-03-10 07:40:01,081]  INFO - CompleteTenantMessageProcessor Tenant
initialized
[2015-03-10 07:40:01,093]  INFO - CompleteTopologyMessageProcessor Topology
initialized

Please help, Thanks
-D


On Mon, Mar 9, 2015 at 10:02 PM, david hbase <davidhbase@gmail.com> wrote:

> Hi, Team:
>
> I found the ERROR with Stratos 4.0 loadbalancer as following, and this
> error blocked the LB statistics to CEP,
>
> Please help, Thanks
> -D
>
> [2015-03-10 04:59:46,057] ERROR - TopicSubscriber Error while subscribing
> to the topic: topology
> javax.naming.NoInitialContextException: Failed to create InitialContext
> using factory specified in hash table. [Root exception is
> java.lang.ClassNotFoundException: class
> org.apache.activemq.jndi.ActiveMQInitialContextFactory not found]
>     at
> org.wso2.carbon.context.internal.CarbonContextDataHolder$CarbonInitialJNDIContextFactoryBuilder.createInitialContextFactory(CarbonContextDataHolder.java:737)
>     at
> javax.naming.spi.NamingManager.getInitialContext(NamingManager.java:681)
>     at
> javax.naming.InitialContext.getDefaultInitCtx(InitialContext.java:307)
>     at javax.naming.InitialContext.init(InitialContext.java:242)
>     at javax.naming.InitialContext.<init>(InitialContext.java:216)
>     at
> org.apache.stratos.messaging.broker.connect.TopicConnector.init(TopicConnector.java:59)
>     at
> org.apache.stratos.messaging.broker.subscribe.TopicSubscriber.doSubscribe(TopicSubscriber.java:63)
>     at
> org.apache.stratos.messaging.broker.subscribe.TopicSubscriber.run(TopicSubscriber.java:116)
>     at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.ClassNotFoundException: class
> org.apache.activemq.jndi.ActiveMQInitialContextFactory not found
>     at
> org.wso2.carbon.context.internal.CarbonContextDataHolder.classForName(CarbonContextDataHolder.java:1450)
>     at
> org.wso2.carbon.context.internal.CarbonContextDataHolder.access$700(CarbonContextDataHolder.java:103)
>     at
> org.wso2.carbon.context.internal.CarbonContextDataHolder$CarbonInitialJNDIContextFactoryBuilder.createInitialContextFactory(CarbonContextDataHolder.java:690)
>     ... 8 more
> [2015-03-10 04:59:46,060]  INFO - TopicSubscriber Will try to subscribe
> again in 30 sec
> [2015-03-10 04:59:46,061]  INFO - TopicSubscriber Will try to subscribe
> again in 30 sec
>
> On Mon, Mar 9, 2015 at 7:36 AM, Rajkumar Rajaratnam <rajkumarr@wso2.com>
> wrote:
>
>> Hi David,
>>
>> You can set the member expiry timeout in
>> Stratos-Home/repository/conf/autoscaler.xml via expiryTimeout element
>> (Highlighted below)
>>
>> <configuration>
>> 	<autoscaler>
>> 	    <rulesEvaluator>
>> 	        <schedule>
>> 		    <initialDelay>30</initialDelay>
>> 	            <period>15</period>
>> 	        </schedule>
>> 	    </rulesEvaluator>
>> 	    <cloudController>
>> 	        <hostname>localhost</hostname>
>> 		<port>9444</port>
>> 		<!-->CC client timout in ms<-->
>>                 <clientTimeout>300000</clientTimeout>
>> 	    </cloudController>
>>             <stratosManager>
>> 	        <hostname>localhost</hostname>
>> 		<port>9445</port>
>> 		<!--SM client timeout in ms-->
>>                 <clientTimeout>300000</clientTimeout>
>> 	    </stratosManager>
>> 	    <member>
>> 		<!-->member expiry timeout (time waited till activation) in ms<-->
>> 		*<expiryTimeout>900000</expiryTimeout>*
>> 	    </member>
>> 	    <!-- monitor task interval -->
>> 	    <monitorInterval>90000</monitorInterval>
>> 	</autoscaler>
>> </configuration>
>>
>> Thanks.
>>
>>
>> On Mon, Mar 9, 2015 at 7:56 PM, david hbase <davidhbase@gmail.com> wrote:
>>
>>> Hi, Imesh and Lakmal
>>>
>>> Thanks for you support, I will get more detail debug logs.
>>> An further issue is how can we disable / change the Member expriy time?
>>>
>>> [2015-03-08 09:41:10,723] DEBUG
>>> {org.apache.stratos.autoscaler.PartitionContext} -  Member expiry time is
>>> set to: 900000
>>>
>>> Thanks
>>> -D
>>>
>>> On Mon, Mar 9, 2015 at 2:35 AM, Lakmal Warusawithana <lakmal@wso2.com>
>>> wrote:
>>>
>>>>
>>>>
>>>> On Mon, Mar 9, 2015 at 2:39 PM, Imesh Gunaratne <imesh@apache.org>
>>>> wrote:
>>>>
>>>>> Hi Lakmal,
>>>>>
>>>>> On Mon, Mar 9, 2015 at 12:20 PM, Lakmal Warusawithana <lakmal@wso2.com
>>>>> > wrote:
>>>>>>
>>>>>>
>>>>>> On Mon, Mar 9, 2015 at 12:16 PM, Imesh Gunaratne <imesh@apache.org>
>>>>>>  wrote:
>>>>>>
>>>>>>> Hi David,
>>>>>>>
>>>>>>> We might need to check why the load balancer is not sending
>>>>>>> statistics to CEP. Try to enable debug logs in the load balancer
and run,
>>>>>>> it should trace the statistics sent to CEP.
>>>>>>>
>>>>>>> No AFAIK, it is not possible to configure Autoscaler to disregard
>>>>>>> load balancer statistics. I see this as a very valid requirement
(to
>>>>>>> exclude any of the scaling factors), may be we can consider adding
this to
>>>>>>> 4.1.0. Would you mind reporting this as a feature request?
>>>>>>>
>>>>>>>
>>>>>> I'm bit confuse here! I can scale up a cluster without having a LB.
>>>>>> (in 4.1.0) which mean LB is not a MUST right?
>>>>>>
>>>>>
>>>>> Yes, scaling up would work but scaling down might not work if
>>>>> statistics for all the autoscaling factors are available:
>>>>>
>>>>> scaleUp : Boolean() from ((rifReset && (rifPredictedValue >
>>>>> rifAverageLimit * 0.8)) || (mcReset && (mcPredictedValue >
mcAverageLimit *
>>>>> 0.8)) || (laReset && (laPredictedValue > laAverageLimit *
0.8)))
>>>>> scaleDown : Boolean() from ((rifReset && (rifPredictedValue <
>>>>> rifAverageLimit * 0.1)) && (mcReset && (mcPredictedValue
< mcAverageLimit *
>>>>> 0.1)) && (laReset && (laPredictedValue < laAverageLimit
* 0.1)))
>>>>>
>>>>>
>>>>> https://github.com/apache/stratos/blob/4.0.0/products/stratos/modules/distribution/src/main/conf/scaling.drl
>>>>>
>>>>>
>>>> IMO, we should fix this. We should support scale up/down with selected
>>>> autoscaling parameters.
>>>>
>>>>
>>>>
>>>>> Thanks
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Lakmal Warusawithana
>>>> Vice President, Apache Stratos
>>>> Director - Cloud Architecture; WSO2 Inc.
>>>> Mobile : +94714289692
>>>> Blog : http://lakmalsview.blogspot.com/
>>>>
>>>>
>>>
>>
>>
>> --
>> Rajkumar Rajaratnam
>> Committer & PMC Member, Apache Stratos
>> Software Engineer, WSO2
>>
>> Mobile : +94777568639
>> Blog : rajkumarr.com
>>
>
>

Mime
View raw message