Return-Path: X-Original-To: apmail-karaf-user-archive@minotaur.apache.org Delivered-To: apmail-karaf-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 70AEF185AF for ; Wed, 6 Jan 2016 16:16:48 +0000 (UTC) Received: (qmail 51000 invoked by uid 500); 6 Jan 2016 16:16:48 -0000 Delivered-To: apmail-karaf-user-archive@karaf.apache.org Received: (qmail 50958 invoked by uid 500); 6 Jan 2016 16:16:48 -0000 Mailing-List: contact user-help@karaf.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@karaf.apache.org Delivered-To: mailing list user@karaf.apache.org Received: (qmail 50948 invoked by uid 99); 6 Jan 2016 16:16:48 -0000 Received: from Unknown (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 06 Jan 2016 16:16:48 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id A63D518048D for ; Wed, 6 Jan 2016 16:16:47 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -0.673 X-Spam-Level: X-Spam-Status: No, score=-0.673 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RP_MATCHES_RCVD=-0.554, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (1024-bit key) header.d=wellsfargo.com Received: from mx1-us-east.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id W4BuKznyVPkb for ; Wed, 6 Jan 2016 16:16:33 +0000 (UTC) Received: from mxdcmx01e.wellsfargo.com (mxdcmx01e.wellsfargo.com [167.138.239.98]) by mx1-us-east.apache.org (ASF Mail Server at mx1-us-east.apache.org) with ESMTPS id A159442AC6 for ; Wed, 6 Jan 2016 16:16:33 +0000 (UTC) Received: from mxidlx02.wellsfargo.com (mxidlx02.wellsfargo.com [162.102.225.33]) by mxdcmx01e.wellsfargo.com (Sentrion-MTA-4.3.1/Sentrion-MTA-4.3.1) with ESMTP id u06GFx94019692 for ; Wed, 6 Jan 2016 16:16:27 GMT DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=wellsfargo.com; s=2011-05-wfb; t=1452096987; bh=4I/jv++WT8DqLsadeBG+F/wETcWzYQ4t60Rn+kh8qok=; h=From:To:Subject:Date:Message-ID:References:In-Reply-To: Content-Type:Content-Transfer-Encoding:MIME-Version; b=l4S5AMAfiDC8+Q4PPp/DAk3d+igbiBC6zeye2egaPf4UmVz7UEYXrOU7EgZxa+LGR h/4+GogWfjYPqh5T3y98HVfMtfjNY995LJWnhBjJEzrtLJ5k6/1tEnyvOYOOmSO+PU JSYNlEsEr545c4OKRWs9tpeoRgwpE8Kc2zZ69fXU= Received: from mxidli01.wellsfargo.com ( [162.103.179.209]) by mxidlx02.wellsfargo.com (postmaster@wellsfargo.com) with SMTP id 3F.CD.19352.ADD3D865; Wed, 6 Jan 2016 16:16:26 +0000 (GMT) X-AuditID: a266e121-f79bd6d000004b98-e5-568d3ddafecc Received: from mxicmx02.wellsfargo.com ( [162.102.137.63]) by mxidli01.wellsfargo.com (postmaster@wellsfargo.com) with SMTP id 30.98.01422.ADD3D865; Wed, 6 Jan 2016 16:16:26 +0000 (GMT) Received: from MSGEXSIL4111.ent.wfb.bank.corp (msgexsil4111.wellsfargo.com [162.103.224.43]) by mxicmx02.wellsfargo.com (Sentrion-MTA-4.3.1/Sentrion-MTA-4.3.1) with ESMTP id u06GGP1u006985 for ; Wed, 6 Jan 2016 16:16:25 GMT Received: from MSGEXSIL1119.ent.wfb.bank.corp ([169.254.1.116]) by MSGEXSIL4111.ent.wfb.bank.corp ([162.103.224.43]) with mapi id 14.03.0224.002; Wed, 6 Jan 2016 11:16:24 -0500 From: To: Subject: RE: Cellar clustering issue Thread-Topic: Cellar clustering issue Thread-Index: AQHRQHZO4S5Zciseuku11pzdJZIZ0p7rcHwAgABXVgD//8eJgIAAWAAAgAEXODCAAFXBgP//uRgQgABgcACAASGcoIAAVl4A//+t2NAACqGyAAAKbwSA//+t8YCAAEvN8P//vg2AgABSGMD//7EKAIAAU3Yw//+vlAAACCuvQAAFuKIAABXi/lA= Date: Wed, 6 Jan 2016 16:16:23 +0000 Message-ID: <443BCE38E921434394B45FF0813FA57850562471@MSGEXSIL1119.ent.wfb.bank.corp> References: <443BCE38E921434394B45FF0813FA5785053E4DB@MSGEXSIL1119.ent.wfb.bank.corp> <568BCB49.6020004@nanthrax.net> <443BCE38E921434394B45FF0813FA57850561618@MSGEXSIL1119.ent.wfb.bank.corp> <568BE0B4.4010409@nanthrax.net> <443BCE38E921434394B45FF0813FA5785056213D@MSGEXSIL1119.ent.wfb.bank.corp> <568D1C18.1020102@nanthrax.net> <443BCE38E921434394B45FF0813FA57850562184@MSGEXSIL1119.ent.wfb.bank.corp> <568D1E86.1010800@nanthrax.net> <443BCE38E921434394B45FF0813FA578505621B7@MSGEXSIL1119.ent.wfb.bank.corp> <568D1FB5.9000005@nanthrax.net> <443BCE38E921434394B45FF0813FA57850562275@MSGEXSIL1119.ent.wfb.bank.corp> <568D27F9.5010101@nanthrax.net> <443BCE38E921434394B45FF0813FA578505622FB@MSGEXSIL1119.ent.wfb.bank.corp> <568D2A9A.8070908@nanthrax.net> <443BCE38E921434394B45FF0813FA57850562341@MSGEXSIL1119.ent.wfb.bank.corp> <568D2D26.5000701@nanthrax.net> <443BCE38E921434394B45FF0813FA57850562448@MSGEXSIL1119.ent.wfb.bank.corp> <568D3D96.9020905@nanthrax.net> In-Reply-To: <568D3D96.9020905@nanthrax.net> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [170.13.180.66] Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFrrBIsWRmVeSWpSXmKPExsWyKHVHu+4t294wg8kNRhYT1r5gdGD0eLP5 CmsAY5SiTUpqTmZZapG+nU1SZUFicbFucppCYk6OrVJJUWmqkr5dgmLGtuP9jAV9m5gq3k15 zNzAuOoVYxcjJ4eEgInEvtsb2SBsMYkL99YD2VwcQgJ7GSV27P/KBlPUduMMM0RiD6PEksZP 7BDOL0aJ38+3MEE4Wxgltq1YBNbCJqAisX/iArAdIgLSEu+aXrF2MXJwCAPF26ZVQIRVJe59 +cMI0isicIxR4mnLPWaQBAtQzZKlm8FsXoFQicO3e8DmCAk8Z5fYN0MExOYU0JGYcvsXE4jN CHT391NrwGxmAXGJW0/mM0GcLSCxZM95ZghbVOLl43+sELaixNnz21kh6vUkbkydwgZha0ss W/gaaq+gxMmZT1hAjpMQmM4ucX7KaWiACUlMvbCIHcKWlDi44gbLBEbpWUh2z0IydxaSubOQ zF3AyLKKUTy3IjMlp8LASK88NSenOC2xKD1fLzk/dxMjKMLTHiruYNw/yfkQowAHoxIP76bV tqFCrIllxZW5hxhNgaE0kVlKNDkfmEbySuINTQwsjC1NTI0MTS1NlMR560CKBdITS1KzU1ML Uovii0pzUosPMTJxcEo1MCpf/nssZ/uOu+ktLVf4YmSd1bOlAyZG8ymYbJlV2RXccplXJHzD vd3yPGwHpCyc5ENuq8yeccCp8MzDKcvXh7J5y6/4uj+NL9UrfZ2LoIrN51+nHVar6Vb3fstS nrbwRkzc3OKSvn8x89VPcHQ5u7lziOe8E9jPzpZvdiiu86xJ4LX2qEvsSizFGYmGWsxFxYkA 94HbHOsCAAA= X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFtrJIsWRmVeSWpSXmKPExsWyKK3TXveWbW+YwacJAhYT1r5gdGD0eLP5 CmsAY5SDTUZqYkpqkUJqXnJ+SmZeuq2SZ7C/roWFqaWuoZJCXmJuqq1ShG64m5Oui0+Aboi/ u5JCWWJOKVA00jVYSd8uwSFj2/F+xoK+TUwV76Y8Zm5gXPWKsYuRk0NCwESi7cYZZghbTOLC vfVsXYxcHEICexglljR+YodwfjFK/H6+hQnC2cIosW3FIjaQFjYBFYn9ExeAjRIRkJZ41/SK tYuRg0MYKN42rQIirCpx78sfRpBeEYFjjBJPW+6BrWMBqlmydDOYzSsQKnH4dg/YHCGB5+wS +2aIgNicAjoSU27/YgKxGYHO+35qDZjNLCAucevJfCaIswUkluw5D/WCqMTLx/9YIWxFibPn t7NC1OtJ3Jg6hQ3C1pZYtvA11F5BiZMzn7BMYBSbhWTsLCQts5C0zELSsoCRZRWjeG5FZkpO poGhXnlqTk5xWmJRer5ecn7uJkZQjKZvvriDsW2J8yFGAQ5GJR7evF8TQoRYE8uKK3MPMZoC A2Ais5Rocj4wEeSVxBuaGFgYW5obmJpZGpkpifPKPrULFRJITyxJzU5NLUgtii8qzUktPsTI xMEp1cAY/ODX9eUFZlmZj42/79q9V1++1vzBmvz7D9n5NCxlZW8J5DMv5rMwE1Y3Ckv3vfhR 1V/klmGYfAHHdHPWtWy21wqab17btYJvOXNjrUxGtHnXvAlX/00JXmaq81KQ/c2ZKWmbCp6c +DPT6cf6JSfiLEraFT7u+fdWzEbq4MpEw3ntlfx87ZlKLMUZiYZazEXFiQBGAeE3zAIAAA== X-WFB-DLP-TOG: YES X-CFilter-Loop: Reflected No, you got it :) Regards, Barry=20 -----Original Message----- From: Jean-Baptiste Onofr=E9 [mailto:jb@nanthrax.net]=20 Sent: Wednesday, January 06, 2016 11:15 AM To: user@karaf.apache.org Subject: Re: Cellar clustering issue Cool, finally you got it ;) Regards JB On 01/06/2016 05:11 PM, barry.barnett@wellsfargo.com wrote: > It works!!!!!! > > Regards, > > Barry > > > -----Original Message----- > From: Jean-Baptiste Onofr=E9 [mailto:jb@nanthrax.net] > Sent: Wednesday, January 06, 2016 10:05 AM > To: user@karaf.apache.org > Subject: Re: Cellar clustering issue > > Both on each node > > On 01/06/2016 03:55 PM, barry.barnett@wellsfargo.com wrote: >> Just put one member, the remote one? Not the local? Or do I put both t= he box1 and box2 as members? >> >> Regards, >> >> Barry >> >> >> -----Original Message----- >> From: Jean-Baptiste Onofr=E9 [mailto:jb@nanthrax.net] >> Sent: Wednesday, January 06, 2016 9:54 AM >> To: user@karaf.apache.org >> Subject: Re: Cellar clustering issue >> >> OK, so, now, disable multicast, and enable tcp-ip with member containing= IP of the different host. >> >> Regards >> JB >> >> On 01/06/2016 03:49 PM, barry.barnett@wellsfargo.com wrote: >>> Yes, still doesn't see the remote node from either box. >>> >>> Regards, >>> >>> Barry >>> >>> >>> -----Original Message----- >>> From: Jean-Baptiste Onofr=E9 [mailto:jb@nanthrax.net] >>> Sent: Wednesday, January 06, 2016 9:43 AM >>> To: user@karaf.apache.org >>> Subject: Re: Cellar clustering issue >>> >>> Did you do cluster:node-list to check the nodes that the cluster is abl= e to see ? >>> >>> On 01/06/2016 03:39 PM, barry.barnett@wellsfargo.com wrote: >>>> Ok, so I try to do a cluster:group-set dev box2, from box1... >>>> >>>> cluster:group-set dev box2:5701 >>>> Cluster node box2:5701 doesn't exist >>>> >>>> 06 Jan 2016 08:33:51,141 | DEBUG | Thread-213 | LoggingCommandSe= ssionListener | 21 - org.apache.karaf.shell.console - 2.4.3 | Command: '= cluster:group-set dev box2' returned 'null' >>>> >>>> >>>> >>>> Regards, >>>> >>>> Barry >>>> >>>> >>>> -----Original Message----- >>>> From: Jean-Baptiste Onofr=E9 [mailto:jb@nanthrax.net] >>>> Sent: Wednesday, January 06, 2016 9:08 AM >>>> To: user@karaf.apache.org >>>> Subject: Re: Cellar clustering issue >>>> >>>> Please, can you try: >>>> >>>> 1. disable tcp-ip and use multicast 2. disable interfaces 3. send=20 >>>> the debug log message on each box 4. result of cluster:node-list >>>> >>>> Thanks, >>>> Regards >>>> JB >>>> >>>> On 01/06/2016 03:05 PM, barry.barnett@wellsfargo.com wrote: >>>>> In the tcpip stanza, I only have member for box1 and box2. I don't s= pecify interface. Should I also include interface there? With just the me= mbers, its still not picking up the remote node from either box. >>>>> >>>>> Regards, >>>>> >>>>> Barry >>>>> >>>>> >>>>> -----Original Message----- >>>>> From: Jean-Baptiste Onofr=E9 [mailto:jb@nanthrax.net] >>>>> Sent: Wednesday, January 06, 2016 9:03 AM >>>>> To: user@karaf.apache.org >>>>> Subject: Re: Cellar clustering issue >>>>> >>>>> If you mean inside , yes correct. >>>>> >>>>> or in can help (depending of your ne= twork configuration). >>>>> >>>>> Regards >>>>> JB >>>>> >>>>> >>>>> On 01/06/2016 02:58 PM, barry.barnett@wellsfargo.com wrote: >>>>>> So to connect one node to another remote node, it is not necessary t= o specify interface? Just set to false and allow to bind on all? >>>>>> >>>>>> Regards, >>>>>> >>>>>> Barry >>>>>> >>>>>> >>>>>> -----Original Message----- >>>>>> From: Jean-Baptiste Onofr=E9 [mailto:jb@nanthrax.net] >>>>>> Sent: Wednesday, January 06, 2016 8:52 AM >>>>>> To: user@karaf.apache.org >>>>>> Subject: Re: Cellar clustering issue >>>>>> >>>>>> Hi Barry, >>>>>> >>>>>> Your interface configuration is not correct. >>>>>> >>>>>> It's the network interface of your machine. >>>>>> >>>>>> By default, Hazelcast binds on all interface on your machine (0.0.0.= 0). >>>>>> >>>>>> You use to specify on which "local" interface you want = to bind. >>>>>> For instance, you have eth0 (192.169.1.1) and eth1 (192.168.134.10) = on your machine. You want to bind on eth1, so you do: >>>>>> >>>>>> >>>>>> 192.168.134.10 >>>>>> >>>>>> >>>>>> If it's not your case, I advise to disable interface in order to bin= d on all interfaces (0.0.0.0). >>>>>> >>>>>> As Hazelcast doesn't start, it explains why the Cellar ClusterManage= r service is not present. >>>>>> >>>>>> Regards >>>>>> JB >>>>>> >>>>>> On 01/06/2016 02:46 PM, barry.barnett@wellsfargo.com wrote: >>>>>>> My interface is set to: >>>>>>> >>>>>>> IPBox1 >>>>>>> IPBox2 >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> 06 Jan 2016 08:34:10,339 | ERROR | FelixStartLevel = | AddressPicker | 250 - com.hazelcast - 2.6.9 | Hazelca= st CANNOT start on this node. No matching network interface found. >>>>>>> Interface matching must be either disabled or updated in the hazelc= ast.xml config file. >>>>>>> 06 Jan 2016 08:34:10,340 | ERROR | FelixStartLevel = | AddressPicker | 250 - com.hazelcast - 2.6.9 | Hazelca= st CANNOT start on this node. No matching network interface found. >>>>>>> Interface matching must be either disabled or updated in the hazelc= ast.xml config file. >>>>>>> java.lang.RuntimeException: Hazelcast CANNOT start on this node. No= matching network interface found. >>>>>>> Interface matching must be either disabled or updated in the hazelc= ast.xml config file. >>>>>>> at com.hazelcast.impl.AddressPicker.pickAddress(Addr= essPicker.java:147) >>>>>>> at com.hazelcast.impl.AddressPicker.pickAddress(Addr= essPicker.java:51) >>>>>>> at com.hazelcast.impl.Node.(Node.java:144) >>>>>>> at com.hazelcast.impl.FactoryImpl.(FactoryImpl= .java:386) >>>>>>> at com.hazelcast.impl.FactoryImpl.newHazelcastInstan= ceProxy(FactoryImpl.java:133) >>>>>>> at com.hazelcast.impl.FactoryImpl.newHazelcastInstan= ceProxy(FactoryImpl.java:119) >>>>>>> at com.hazelcast.impl.FactoryImpl.newHazelcastInstan= ceProxy(FactoryImpl.java:104) >>>>>>> at com.hazelcast.core.Hazelcast.newHazelcastInstance= (Hazelcast.java:507)[250:com.hazelcast:2.6.9] >>>>>>> at org.apache.karaf.cellar.hazelcast.factory.Hazelca= stServiceFactory.buildInstance(HazelcastServiceFactory.java:107)[253:org.ap= ache.karaf.cellar.hazelcast:2.3.6] >>>>>>> at org.apache.karaf.cellar.hazelcast.factory.Hazelca= stServiceFactory.getInstance(HazelcastServiceFactory.java:92)[253:org.apach= e.karaf.cellar.hazelcast:2.3.6] >>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Nati= ve Method)[:1.8.0_45] >>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke(Nativ= eMethodAccessorImpl.java:62)[:1.8.0_45] >>>>>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(D= elegatingMethodAccessorImpl.java:43)[:1.8.0_45] >>>>>>> at java.lang.reflect.Method.invoke(Method.java:497)[= :1.8.0_45] >>>>>>> at org.apache.aries.blueprint.utils.ReflectionUtils.= invoke(ReflectionUtils.java:297)[18:org.apache.aries.blueprint.core:1.4.3] >>>>>>> at org.apache.aries.blueprint.container.BeanRecipe.i= nvoke(BeanRecipe.java:958)[18:org.apache.aries.blueprint.core:1.4.3] >>>>>>> at org.apache.aries.blueprint.container.BeanRecipe.g= etInstance(BeanRecipe.java:298)[18:org.apache.aries.blueprint.core:1.4.3] >>>>>>> at org.apache.aries.blueprint.container.BeanRecipe.i= nternalCreate2(BeanRecipe.java:806)[18:org.apache.aries.blueprint.core:1.4.= 3] >>>>>>> at org.apache.aries.blueprint.container.BeanRecipe.i= nternalCreate(BeanRecipe.java:787)[18:org.apache.aries.blueprint.core:1.4.3= ] >>>>>>> at org.apache.aries.blueprint.di.AbstractRecipe$1.ca= ll(AbstractRecipe.java:79)[18:org.apache.aries.blueprint.core:1.4.3] >>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.ja= va:266)[:1.8.0_45] >>>>>>> at org.apache.aries.blueprint.di.AbstractRecipe.crea= te(AbstractRecipe.java:88)[18:org.apache.aries.blueprint.core:1.4.3] >>>>>>> at org.apache.aries.blueprint.di.RefRecipe.internalC= reate(RefRecipe.java:62)[18:org.apache.aries.blueprint.core:1.4.3] >>>>>>> at org.apache.aries.blueprint.di.AbstractRecipe.crea= te(AbstractRecipe.java:106)[18:org.apache.aries.blueprint.core:1.4.3] >>>>>>> at org.apache.aries.blueprint.container.ServiceRecip= e.createService(ServiceRecipe.java:284)[18:org.apache.aries.blueprint.core:= 1.4.3] >>>>>>> at org.apache.aries.blueprint.container.ServiceRecip= e.internalGetService(ServiceRecipe.java:251)[18:org.apache.aries.blueprint.= core:1.4.3] >>>>>>> at org.apache.aries.blueprint.container.ServiceRecip= e.internalCreate(ServiceRecipe.java:148)[18:org.apache.aries.blueprint.core= :1.4.3] >>>>>>> at org.apache.aries.blueprint.container.ServiceRecipe= .internalCreate(ServiceRecipe.java:148)[18:org.apache.aries.blueprint.core:= 1.4.3] at org.apache.aries.blueprint.di.AbstractRecipe$1.call(Abstra= ctRecipe.java:79)[18:org.apache.aries.blueprint.core:1.4.3] >>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.ja= va:266)[:1.8.0_45] >>>>>>> at org.apache.aries.blueprint.di.AbstractRecipe.crea= te(AbstractRecipe.java:88)[18:org.apache.aries.blueprint.core:1.4.3] = at org.apache.aries.blueprint.container.BlueprintRepository.createInstance= s(BlueprintRepository.java:245)[18:org.apache.aries.blueprint.core:1.4.3] = at org.apache.aries.blueprint.container.BlueprintRepository.createAll= (BlueprintRepository.java:183)[18:org.apache.aries.blueprint.core:1.4.3] = at org.apache.aries.blueprint.container.BlueprintContainerImpl.instant= iateEagerComponents(BlueprintContainerImpl.java:682)[18:org.apache.aries.bl= ueprint.core:1.4.3] at org.apache.aries.blueprint.container.Blueprin= tContainerImpl.doRun(BlueprintContainerImpl.java:377)[18:org.apache.aries.b= lueprint.core:1.4.3] at org.apache.aries.blueprint.container.Bluepri= ntContainerImpl.run(BlueprintContainerImpl.java:269)[18:org.apache.aries.bl= ueprint.core:1.4.3] at org.apache.aries.blueprint.container.Blueprin= tExtender.createContain e r > ( > B >> l >> u >>> e >>> p >>>> r >>>> i >>>>> n >>>>> t >>>>>> E >>>>>> xtender.java:294)[18:org.apache.aries.blueprint.core:1.4.3] a= t org.apache.aries.blueprint.container.BlueprintExtender.createContainer(Bl= ueprintExtender.java:263)[18:org.apache.aries.blueprint.core:1.4.3] = at org.apache.aries.blueprint.container.BlueprintExtender.modifiedBundle(Bl= ueprintExtender.java:253)[18:org.apache.aries.blueprint.core:1.4.3] = at org.apache.aries.util.tracker.hook.BundleHookBundleTracker$Tracked.custo= mizerModified(BundleHookBundleTracker.java:500)[13:org.apache.aries.util:1.= 1.0] at org.apache.aries.util.tracker.hook.BundleHookBundleTracker$T= racked.customizerModified(BundleHookBundleTracker.java:433)[13:org.apache.a= ries.util:1.1.0] at org.apache.aries.util.tracker.hook.BundleHookBun= dleTracker$AbstractTracked.track(BundleHookBundleTracker.java:725)[13:org.a= pache.aries.util:1.1.0] at org.apache.aries.util.tracker.hook.Bundle= HookBundleTracker$Tracked.bundleChanged(BundleHookBundleTracker.java:463)[1= 3:org.apache.aries.util : > 1 > >> . >> >>> 1 >>> >>>> . >>>> >>>>> 0 >>>>> ] >>>>>> >>>>>> at >>>>>> org.apache.aries.util.tracker.hook.BundleHookBundleTracker$Bundle >>>>>> Ev >>>>>> e >>>>>> n >>>>>> t >>>>>> Hook.event(BundleHookBundleTracker.java:422)[13:org.apache.aries.uti= l: >>>>>> 1.1.0] >>>>>>> at org.apache.felix.framework.util.SecureAction.invo= keBundleEventHook(SecureAction.java:1127)[org.apache.felix.framework-4.4.1.= jar:] at org.apache.felix.framework.util.EventDispatcher.createWhite= listFromHooks(EventDispatcher.java:696)[org.apache.felix.framework-4.4.1.ja= r:] >>>>>>> at org.apache.felix.framework.util.EventDispatcher.f= ireBundleEvent(EventDispatcher.java:484)[org.apache.felix.framework-4.4.1.j= ar:] >>>>>>> at org.apache.felix.framework.Felix.fireBundleEvent(= Felix.java:4429)[org.apache.felix.framework-4.4.1.jar:] >>>>>>> at org.apache.felix.framework.Felix.startBundle(Feli= x.java:2100)[org.apache.felix.framework-4.4.1.jar:] at org.apache.fe= lix.framework.Felix.setActiveStartLevel(Felix.java:1299)[org.apache.felix.f= ramework-4.4.1.jar:] >>>>>>> at >>>>>>> org.apache.felix.framework.FrameworkStartLevelImpl.run(Framework >>>>>>> St a r t L=20 >>>>>>> evelImpl.java:304)[org.apache.felix.framework-4.4.1.jar:] >>>>>>> at org.apache.felix.framework.Felix.setActiveStartLevel(Felix.java:= 1299)[org.apache.felix.framework-4.4.1.jar:] >>>>>>> at org.apache.felix.framework.FrameworkStartLevelImp= l.run(FrameworkStartLevelImpl.java:304)[org.apache.felix.framework-4.4.1.ja= r:] >>>>>>> at java.lang.Thread.run(Thread.java:745)[:1.8.0_45] >>>>>>> 06 Jan 2016 08:34:10,349 | ERROR | FelixStartLevel = | ServiceRecipe | 18 - org.apache.aries.blueprint.core = - 1.4.3 | Error retrieving service from ServiceRecipe[name=3D'.component-1'= ] >>>>>>> org.osgi.service.blueprint.container.ComponentDefinitionException: = Error when instantiating bean hazelcast of class com.hazelcast.core.Hazelca= st >>>>>>> at org.apache.aries.blueprint.container.BeanRecipe.g= etInstance(BeanRecipe.java:300)[18:org.apache.aries.blueprint.core:1.4.3] >>>>>>> at org.apache.aries.blueprint.container.BeanRecipe.i= nternalCreate2(BeanRecipe.java:806)[18:org.apache.aries.blueprint.core:1.4.= 3] >>>>>>> at org.apache.aries.blueprint.container.BeanRecipe.i= nternalCreate(BeanRecipe.java:787)[18:org.apache.aries.blueprint.core:1.4.3= ] >>>>>>> at org.apache.aries.blueprint.di.AbstractRecipe$1.ca= ll(AbstractRecipe.java:79)[18:org.apache.aries.blueprint.core:1.4.3] >>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.ja= va:266)[:1.8.0_45] >>>>>>> at org.apache.aries.blueprint.di.AbstractRecipe.crea= te(AbstractRecipe.java:88)[18:org.apache.aries.blueprint.core:1.4.3] = at org.apache.aries.blueprint.di.RefRecipe.internalCreate(RefRecipe.java:6= 2)[18:org.apache.aries.blueprint.core:1.4.3] >>>>>>> at org.apache.aries.blueprint.di.AbstractRecipe.crea= te(AbstractRecipe.java:106)[18:org.apache.aries.blueprint.core:1.4.3] = at org.apache.aries.blueprint.container.ServiceRecipe.createService(Servi= ceRecipe.java:284)[18:org.apache.aries.blueprint.core:1.4.3] at org.= apache.aries.blueprint.container.ServiceRecipe.internalGetService(ServiceRe= cipe.java:251)[18:org.apache.aries.blueprint.core:1.4.3] >>>>>>> at org.apache.aries.blueprint.container.ServiceRecip= e.internalCreate(ServiceRecipe.java:148)[18:org.apache.aries.blueprint.core= :1.4.3] at org.apache.aries.blueprint.di.AbstractRecipe$1.call(Abstr= actRecipe.java:79)[18:org.apache.aries.blueprint.core:1.4.3] >>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.ja= va:266)[:1.8.0_45] at org.apache.aries.blueprint.di.AbstractRecipe.c= reate(AbstractRecipe.java:88)[18:org.apache.aries.blueprint.core:1.4.3] = at org.apache.aries.blueprint.container.BlueprintRepository.createInsta= nces(BlueprintRepository.java:245)[18:org.apache.aries.blueprint.core:1.4.3= ] at org.apache.aries.blueprint.container.BlueprintRepository.create= All(BlueprintRepository.java:183)[18:org.apache.aries.blueprint.core:1.4.3]= at org.apache.aries.blueprint.container.BlueprintContainerImpl.inst= antiateEagerComponents(BlueprintContainerImpl.java:682)[18:org.apache.aries= .blueprint.core:1.4.3] at org.apache.aries.blueprint.container.Bluep= rintContainerImpl.doRun(BlueprintContainerImpl.java:377)[18:org.apache.arie= s.blueprint.core:1.4.3] at org.apache.aries.blueprint.container.Blue= printContainerImpl.run(BlueprintContainerImpl.java:269)[18:org.apache.aries= .blueprint.core:1.4.3] >>> >>> a >>>> t >>>> >>>>> o >>>>> r >>>>>> g >>>>>> .apache.aries.blueprint.container.BlueprintExtender.createContain >>>>>>> >>>>>>> >>>>>>> Regards, >>>>>>> >>>>>>> Barry >>>>>>> >>>>>>> >>>>>>> -----Original Message----- >>>>>>> From: Jean-Baptiste Onofr=E9 [mailto:jb@nanthrax.net] >>>>>>> Sent: Tuesday, January 05, 2016 10:27 AM >>>>>>> To: user@karaf.apache.org >>>>>>> Subject: Re: Cellar clustering issue >>>>>>> >>>>>>> Can you set log level in DEBUG and send karaf.log to me ? >>>>>>> >>>>>>> It looks like bundle-hazelcast doesn't expose the ClusterManager se= rvice. >>>>>>> >>>>>>> Regards >>>>>>> JB >>>>>>> >>>>>>> On 01/05/2016 03:42 PM, barry.barnett@wellsfargo.com wrote: >>>>>>>> Ok, now I've put in the stanza for required-members. But when I d= o that, and have interfaces enabled, I get the following: >>>>>>>> >>>>>>>> 05 Jan 2016 09:40:20,739 | INFO | Thread-198 = | ReferenceRecipe | 18 - org.apache.aries.blueprint.core= - 1.4.3 | No matching service for optional OSGi service reference (objectC= lass=3Dorg.apache.karaf.cellar.core.ClusterManager) >>>>>>>> 05 Jan 2016 09:40:20,741 | ERROR | Thread-198 = | Console | 21 - org.apache.karaf.shell.console = - 2.4.3 | Exception caught while executing command >>>>>>>> org.osgi.service.blueprint.container.ServiceUnavailableException: = No matching service for optional OSGi service reference: (objectClass=3Dorg= .apache.karaf.cellar.core.ClusterManager) >>>>>>>> at org.apache.aries.blueprint.container.ReferenceR= ecipe.getService(ReferenceRecipe.java:236) >>>>>>>> at org.apache.aries.blueprint.container.ReferenceR= ecipe.access$000(ReferenceRecipe.java:55) >>>>>>>> at org.apache.aries.blueprint.container.ReferenceR= ecipe$ServiceDispatcher.call(ReferenceRecipe.java:298) >>>>>>>> at Proxy2882e1b3_fe00_4c30_818e_5ad671ebc492.listN= odes(Unknown Source) >>>>>>>> at org.apache.karaf.cellar.shell.NodesListCommand.= doExecute(NodesListCommand.java:29) >>>>>>>> at org.apache.karaf.shell.console.OsgiCommandSuppo= rt.execute(OsgiCommandSupport.java:38) >>>>>>>> at org.apache.felix.gogo.commands.basic.AbstractCo= mmand.execute(AbstractCommand.java:35) >>>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Na= tive Method)[:1.8.0_45] >>>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke(Nat= iveMethodAccessorImpl.java:62)[:1.8.0_45] >>>>>>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke= (DelegatingMethodAccessorImpl.java:43)[:1.8.0_45] >>>>>>>> at java.lang.reflect.Method.invoke(Method.java:497= )[:1.8.0_45] >>>>>>>> at org.apache.aries.proxy.impl.ProxyHandler$1.invo= ke(ProxyHandler.java:54) >>>>>>>> at org.apache.aries.proxy.impl.ProxyHandler.invoke= (ProxyHandler.java:119) >>>>>>>> at org.apache.karaf.shell.console.commands.$Bluepr= intCommand1144753357.execute(Unknown Source)[21:org.apache.karaf.shell.cons= ole:2.4.3] >>>>>>>> at org.apache.felix.gogo.runtime.CommandProxy.exec= ute(CommandProxy.java:78)[21:org.apache.karaf.shell.console:2.4.3] >>>>>>>> at org.apache.felix.gogo.runtime.Closure.executeCm= d(Closure.java:477)[21:org.apache.karaf.shell.console:2.4.3] >>>>>>>> at org.apache.felix.gogo.runtime.Closure.executeSt= atement(Closure.java:403)[21:org.apache.karaf.shell.console:2.4.3] >>>>>>>> at org.apache.felix.gogo.runtime.Pipe.run(Pipe.jav= a:108)[21:org.apache.karaf.shell.console:2.4.3] >>>>>>>> at org.apache.felix.gogo.runtime.Closure.execute(C= losure.java:183)[21:org.apache.karaf.shell.console:2.4.3] >>>>>>>> at org.apache.felix.gogo.runtime.Closure.execute(C= losure.java:120)[21:org.apache.karaf.shell.console:2.4.3] >>>>>>>> at org.apache.felix.gogo.runtime.CommandSessionImp= l.execute(CommandSessionImpl.java:92) >>>>>>>> at org.apache.karaf.shell.console.jline.Console.ru= n(Console.java:195) >>>>>>>> at org.apache.karaf.shell.ssh.ShellFactoryImpl$She= llImpl$1.runConsole(ShellFactoryImpl.java:167)[36:org.apache.karaf.shell.ss= h:2.4.3] >>>>>>>> at org.apache.karaf.shell.ssh.ShellFactoryImpl$She= llImpl$1$1.run(ShellFactoryImpl.java:126) >>>>>>>> at java.security.AccessController.doPrivileged(Nat= ive Method)[:1.8.0_45] >>>>>>>> at org.apache.karaf.jaas.modules.JaasHelper.doAs(J= aasHelper.java:47)[20:org.apache.karaf.jaas.modules:2.4.3] >>>>>>>> at >>>>>>>> org.apache.karaf.shell.ssh.ShellFactoryImpl$ShellImpl$1.run(She >>>>>>>> ll F a c t o=20 >>>>>>>> ryImpl.java:124)[36:org.apache.karaf.shell.ssh:2.4.3] >>>>>>>> >>>>>>>> Regards, >>>>>>>> >>>>>>>> Barry >>>>>>>> >>>>>>>> -----Original Message----- >>>>>>>> From: Jean-Baptiste Onofr=E9 [mailto:jb@nanthrax.net] >>>>>>>> Sent: Tuesday, January 05, 2016 8:55 AM >>>>>>>> To: user@karaf.apache.org >>>>>>>> Subject: Re: Cellar clustering issue >>>>>>>> >>>>>>>> As I'm not able to reproduce your issue, it's not easy to figure i= t out. >>>>>>>> >>>>>>>> Clearly, the problem is that the nodes don't see each other, and I= suspect we're missing something obvious on the network configuration. So y= es, tweaking the tcp-ip configuration can help. >>>>>>>> >>>>>>>> The only weird thing for me is the fact that you have a Cellar bun= dle in failed state. Is it still the case ? >>>>>>>> >>>>>>>> Regards >>>>>>>> JB >>>>>>>> >>>>>>>> On 01/05/2016 02:50 PM, barry.barnett@wellsfargo.com wrote: >>>>>>>>> Should I try the following? >>>>>>>>> >>>>>>>>> >>>>>>>>> IPofBox1 >>>>>>>>> IPofBox1 >>>>>>>>> IPofBox1,IPofBox2 >>>>>>>>> >>>>>>>>> >>>>>>>>> I currently only use: >>>>>>>>> >>>>>>>>> IPofBox1 >>>>>>>>> IPofBox2 >>>>>>>>> >>>>>>>>> >>>>>>>>> Regards, >>>>>>>>> >>>>>>>>> Barry >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -----Original Message----- >>>>>>>>> From: Jean-Baptiste Onofr=E9 [mailto:jb@nanthrax.net] >>>>>>>>> Sent: Monday, January 04, 2016 11:09 AM >>>>>>>>> To: user@karaf.apache.org >>>>>>>>> Subject: Re: Cellar clustering issue >>>>>>>>> >>>>>>>>> Do you mind to provide a link to the karaf log ? >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Regards >>>>>>>>> JB >>>>>>>>> >>>>>>>>> On 01/04/2016 04:59 PM, barry.barnett@wellsfargo.com wrote: >>>>>>>>>> Iptables is not enabled on the Linux boxes. >>>>>>>>>> >>>>>>>>>> Config on hazelcast.xml for Box1. Box2 is a mirror image basica= lly: >>>>>>>>>> >>>>>>>>>> >>>>>>>>> xsi:schemaLocation=3D"http://www.hazelcast.com/schema/config haz= elcast-config-2.5.xsd" >>>>>>>>>> xmlns=3D"http://www.hazelcast.com/schema/co= nfig" >>>>>>>>>> xmlns:xsi=3D"http://www.w3.org/2001/XMLSche= ma-instance"> >>>>>>>>>> >>>>>>>>>> dev >>>>>>>>>> pass >>>>>>>>>> >>>>>>>>>> http://localh= ost:8080/mancenter >>>>>>>>>> >>>>>>>>>> 5701 >>>>>>>>>> >>>>>>>>>> 0 >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> 224.2.2.3 >>>>>>>>>> 54327 >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> IPforBox1:5701 >>>>>>>>>> IPforBox2:5701 >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> my-access-key >>>>>>>>>> my-secret-key >>>>>>>>>> >>>>>>>>>> us-west-1 >>>>>>>>>> >>>>>>>>>> ec2.amazonaws.com >>>>>>>>>> >>>>>>>>>> hazelcast-sg >>>>>>>>>> type >>>>>>>>>> hz-nodes >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> IPforBox1 >>>>>>>>>> IPforBox2 >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> PBEWithMD5AndDES >>>>>>>>>> >>>>>>>>>> thesalt >>>>>>>>>> >>>>>>>>>> thepass >>>>>>>>>> >>>>>>>>>> 19 >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> RSA/NONE/PKCS1PADDING >>>>>>>>>> >>>>>>>>>> thekeypass >>>>>>>>>> >>>>>>>>>> local >>>>>>>>>> >>>>>>>>>> JKS >>>>>>>>>> >>>>>>>>>> thestorepass >>>>>>>>>> >>>>>>>>>> keystore >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> 16 >>>>>>>>>> 64 >>>>>>>>>> 60 >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> 0 >>>>>>>>>> >>>>>>>>>> default >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> 1 >>>>>>>>>> >>>>>>>>>> 0 >>>>>>>>>> >>>>>>>>>> 0 >>>>>>>>>> >>>>>>>>>> NONE >>>>>>>>>> >>>>>>>>>> 0 >>>>>>>>>> >>>>>>>>>> 25 >>>>>>>>>> >>>>>>>>>> hz.ADD_NEW_ENTRY >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>