Return-Path: X-Original-To: apmail-helix-user-archive@minotaur.apache.org Delivered-To: apmail-helix-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 5465C10BE2 for ; Fri, 31 Jan 2014 22:56:14 +0000 (UTC) Received: (qmail 98877 invoked by uid 500); 31 Jan 2014 22:56:13 -0000 Delivered-To: apmail-helix-user-archive@helix.apache.org Received: (qmail 98795 invoked by uid 500); 31 Jan 2014 22:56:12 -0000 Mailing-List: contact user-help@helix.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@helix.apache.org Delivered-To: mailing list user@helix.apache.org Received: (qmail 98693 invoked by uid 99); 31 Jan 2014 22:56:11 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 31 Jan 2014 22:56:11 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of g.kishore@gmail.com designates 74.125.82.179 as permitted sender) Received: from [74.125.82.179] (HELO mail-we0-f179.google.com) (74.125.82.179) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 31 Jan 2014 22:56:07 +0000 Received: by mail-we0-f179.google.com with SMTP id q58so186202wes.10 for ; Fri, 31 Jan 2014 14:55:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=Fn9NxsJ/oJ5pAsExgHzeAhu15KQrJytRFnS720WVtF8=; b=i5tNtMD1n3fvK8SegaSbEGuLS/88IC62GO/noNZVnV9vanJ8g1xvF99Ip9KmQyaQK4 1AhoW3KT5w+1YwBHiYsUxAy7Alotatugzdvbrzbn3OZx0uc6VbB9eLKlFHv3IjxjbfEF UeUcyY42PXXuQ5QV37tzT2oyTONjQiIX//cm9/CSMk3FegxbDBWOIaVQcSrRu/FYHMac l4tGxFYDmLmI4gYLhyedEn/ApRMv4GeiG5qRnsirAlozEm3fj07Orb1aBK/ickEowNyR zGYe5rNkMHjYNgA+cXaLMzdsmJvtgN0LwZMXsrDEgGdK8RZPFe5SARE33Zfm7fRwCypG Rn+Q== MIME-Version: 1.0 X-Received: by 10.194.185.165 with SMTP id fd5mr29245wjc.95.1391208945749; Fri, 31 Jan 2014 14:55:45 -0800 (PST) Received: by 10.194.78.143 with HTTP; Fri, 31 Jan 2014 14:55:45 -0800 (PST) In-Reply-To: References: Date: Fri, 31 Jan 2014 14:55:45 -0800 Message-ID: Subject: Re: question about helix-agent From: kishore g To: Benoit Vanderbeck Cc: user@helix.apache.org Content-Type: multipart/alternative; boundary=047d7bdc7e2a05122004f14c13c2 X-Virus-Checked: Checked by ClamAV on apache.org --047d7bdc7e2a05122004f14c13c2 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable By the way, if you prefer Python there is a Helix python agent. It should not hard to add the ability to run a command on every transition using python agent. https://pypi.python.org/pypi/pyhelix On Mon, Jan 27, 2014 at 5:29 AM, Benoit Vanderbeck < benoit.vanderbeck@vaph.be> wrote: > Thanks, > > your explanation did clarify. > Will keep on playing with helix. > > regards, > > B > > > On 23 January 2014 19:15, kishore g wrote: > >> Not sure if subscribed to the mailing list. You can subscribe by sending >> email to user-subscribe@helix.apache.org >> >> ---------- Forwarded message ---------- >> From: kishore g >> Date: Thu, Jan 23, 2014 at 9:30 AM >> Subject: Re: question about helix-agent >> To: user@helix.apache.org >> >> >> Hi, >> >> The command needs to set on a per resource basis or at a cluster level. >> The per resource config is specified under >> /CONFIGS/RESOURCE/ or /CONFIGS/CLUSTER/. >> >> You did set it in the right place >> /VAPH_DEV/CONFIGS/RESOURCE/simpleHttpServer >> >> But the configuration keys need to be slightly different. >> >> Take a look at the integration test TestHelixAgent >> >> The configuration looks like this >> TestHelixAgent_test/CONFIGS/CLUSTER/TestHelixAgent_test >> >> { >> "id" : "TestHelixAgent_test", >> "listFields" : { >> }, >> "mapFields" : { >> }, >> "simpleFields" : { >> "SLAVE-MASTER.command" : "simpleHttpClient.py SLAVE-MASTER", >> "SLAVE-MASTER.command.timeout" : "0", >> "SLAVE-MASTER.command.workingDir" : >> "/Users/kgopalak/Documents/projects/incubator-helix/helix-agent/src/main= /scripts/integration-test/script", >> "OFFLINE-SLAVE.command" : "simpleHttpClient.py OFFLINE-SLAVE", >> "OFFLINE-SLAVE.command.workingDir" : >> "/Users/kgopalak/Documents/projects/incubator-helix/helix-agent/src/main= /scripts/integration-test/script", >> "MASTER-SLAVE.command" : "simpleHttpClient.py MASTER-SLAVE", >> "MASTER-SLAVE.command.workingDir" : >> "/Users/kgopalak/Documents/projects/incubator-helix/helix-agent/src/main= /scripts/integration-test/script", >> "SLAVE-OFFLINE.command" : "simpleHttpClient.py SLAVE-OFFLINE", >> "SLAVE-OFFLINE.command.workingDir" : >> "/Users/kgopalak/Documents/projects/incubator-helix/helix-agent/src/main= /scripts/integration-test/script", >> "OFFLINE-DROPPED.command" : "nop" >> } >> } >> >> You can set these configs using helix admin api. >> >> Via java api >> HelixConfigScope scope =3D >> new >> HelixConfigScopeBuilder(ConfigScopeProperty.CLUSTER).forCluster(clusterN= ame).build(); >> CommandConfig.Builder builder =3D new CommandConfig.Builder(); >> CommandConfig cmdConfig =3D >> builder.setTransition("SLAVE", "MASTER") >> .setCommand("simpleHttpClient.py SLAVE-MASTER") >> .setCommandWorkingDir(workingDir).setCommandTimeout("0= ") >> .setPidFile(pidFile) >> .build(); >> configAccessor.set(scope, cmdConfig.toKeyValueMap()); >> >> Using Helix Admin >> USAGE: --setConfig > myCluster,testDB) KeyValueMap(e.g. k1=3Dv1,k2=3Dv2)> Set configs >> example: >> ./helix-admin.sh --zkSvr localhost:2181 --setConfig CLUSTER >> VAPH_DEV SLAVE-MASTER.command=3D,SLAVE-MASTER.workingDir=3D, >> SLAVE-MASTER.command.pidfile=3D >> >> >> Let us know if this does not work. >> >> thanks, >> Kishore G >> >> >> >> >> >> >> >> >> On Thu, Jan 23, 2014 at 4:58 AM, Benoit Vanderbeck < >> benoit.vanderbeck@vaph.be> wrote: >> >>> Hi, >>> >>> i'm trying to run the helix-agent. But I can't seem to figure out how t= o >>> configure the command that should be run when the statetransition chang= es >>> from offline to online. >>> I have tried with a simple setup : One instance and one resource >>> >>> IdealState for simpleHttpServer: >>> { >>> "id" : "simpleHttpServer", >>> "mapFields" : { >>> "simpleHttpServer_0" : { >>> "vbxl003_80" : "ONLINE" >>> }, >>> "simpleHttpServer_1" : { >>> "vbxl003_80" : "ONLINE" >>> }, >>> "simpleHttpServer_2" : { >>> "vbxl003_80" : "ONLINE" >>> } >>> }, >>> "listFields" : { >>> "simpleHttpServer_0" : [ "vbxl003_80" ], >>> "simpleHttpServer_1" : [ "vbxl003_80" ], >>> "simpleHttpServer_2" : [ "vbxl003_80" ] >>> }, >>> "simpleFields" : { >>> "COMMAND" : "/tmp/simpleHttpServer.py", >>> "IDEAL_STATE_MODE" : "AUTO", >>> "NUM_PARTITIONS" : "3", >>> "OFFLINE-ONLINE.COMMAND" : "/tmp/simpleHttpServer.py", >>> "REBALANCE_MODE" : "SEMI_AUTO", >>> "REPLICAS" : "1", >>> "STATE_MODEL_DEF_REF" : "OnlineOffline", >>> "STATE_MODEL_FACTORY_NAME" : "DEFAULT" >>> } >>> } >>> >>> ExternalView for simpleHttpServer: >>> { >>> "id" : "simpleHttpServer", >>> "mapFields" : { >>> }, >>> "listFields" : { >>> }, >>> "simpleFields" : { >>> "BUCKET_SIZE" : "0" >>> } >>> >>> >>> But when I start the agent I get following error : >>> >>> 677 [pool-2-thread-1] ERROR >>> org.apache.helix.messaging.handling.HelixStateTransitionHandler - >>> Exception while executing a state transition task simpleHttpServer_0 >>> java.lang.reflect.InvocationTargetException >>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>> at >>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.ja= va:39) >>> at >>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccesso= rImpl.java:25) >>> at java.lang.reflect.Method.invoke(Method.java:597) >>> at >>> org.apache.helix.messaging.handling.HelixStateTransitionHandler.invoke(= HelixStateTransitionHandler.java:299) >>> at >>> org.apache.helix.messaging.handling.HelixStateTransitionHandler.handleM= essage(HelixStateTransitionHandler.java:253) >>> at >>> org.apache.helix.messaging.handling.HelixTask.call(HelixTask.java:85) >>> at org.apache.helix.messaging.handling.HelixTask.call(HelixTask.java:42= ) >>> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) >>> at java.util.concurrent.FutureTask.run(FutureTask.java:138) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecut= or.java:886) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.j= ava:908) >>> at java.lang.Thread.run(Thread.java:662) >>> Caused by: java.lang.Exception: Unable to find command for transition >>> from:OFFLINE to:ONLINE >>> >>> >>> So the agent can not find what command should be executed when going >>> from OFFLINE to ONLINE. >>> >>> I also tried to edit the configs tree in the znode >>> >>> get /VAPH_DEV/CONFIGS/RESOURCE/simpleHttpServer >>> { >>> "id" : "simpleHttpServer", >>> "simpleFields" : { >>> "COMMAND" : "/tmp/simpleHttpServer.py", >>> "WORKING_DIR" : "/tmp", >>> "PID" : "/tmp/mypid", >>> "OFFLINE-ONLINE.COMMAND" : "/tmp/simpleHttpServer.py" >>> }, >>> "listFields" : { >>> }, >>> "mapFields" : { >>> } >>> } >>> >>> >>> But that didn't work either. >>> >>> Can someone point me in the right direction ? Where en how do define th= e >>> command that the agent should execute. >>> >>> Regards, >>> >>> Beno=EEt >>> -- >>> Beno=EEt Vanderbeck >>> Systeembeheerder VAPH >>> tel: +32 2 225 84 80 >>> >>> >>> Raadpleeg uw VAPH-dossier online via http://mijn.vaph.be. >>> ------------------------------------------------------------ >>> --------------------- >>> http://www.vaph.be/disclaimer >> >> >> >> > > > -- > Beno=EEt Vanderbeck > Systeembeheerder VAPH > tel: +32 2 225 84 80 > > > Raadpleeg uw VAPH-dossier online via http://mijn.vaph.be. > ------------------------------------------------------------ > --------------------- > http://www.vaph.be/disclaimer > --047d7bdc7e2a05122004f14c13c2 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
By the way, if you prefer Python there is a Helix python a= gent. It should not hard to add the ability to run a command on every trans= ition using python agent.

https://pypi.python.org/pypi/pyhelix



On = Mon, Jan 27, 2014 at 5:29 AM, Benoit Vanderbeck <benoit.vanderbeck= @vaph.be> wrote:
Thanks,

= your explanation did clarify.
Will keep on playing with helix.

regards,

B


On 23 January 2014 19:15, kishore g <g.kishore@gmail.com> = wrote:
Not sure if subscribed to the mailing list. You can subscr= ibe by sending email to user-subscribe@helix.apache.org

---------- Forwarded message ----------
From: kishore g <g.kishore@gmail.com>
Date: Thu, Jan 23, 2014 at 9:30 AM
Subject: Re: questio= n about helix-agent
To: user@helix.a= pache.org


Hi,

The c= ommand needs to set on a per resource basis or at a cluster level. The per = resource config is specified under /CONFIGS/RESOURCE/<resource_name> = or /CONFIGS/CLUSTER/<cluster_name>.

You did set it in the right place /VAPH_DEV/CONFIGS/RESOURCE= /simpleHttpServer

But the configuration keys need to be s= lightly different.

Take a look at the integration test Te= stHelixAgent

The configuration looks like this TestHelixAgent_test/CONFIG= S/CLUSTER/TestHelixAgent_test

{
=A0 "id" : "TestHe= lixAgent_test",
=A0 "listFields" : {
=A0 },
=A0 &qu= ot;mapFields" : {
=A0 },
=A0 "simpleFields" : {
=A0=A0=A0 "SLAVE-MASTER.= command" : "simpleHttpClient.py SLAVE-MASTER",
=A0=A0=A0 = "SLAVE-MASTER.command.timeout" : "0",
=A0=A0=A0 &quo= t;SLAVE-MASTER.command.workingDir" : "/Users/kgopalak/Documents/p= rojects/incubator-helix/helix-agent/src/main/scripts/integration-test/scrip= t",
=A0=A0=A0 "OFFLINE-SLAVE.command" : "simpleHttpClient.py OFF= LINE-SLAVE",
=A0=A0=A0 "OFFLINE-SLAVE.command.workingDir"= : "/Users/kgopalak/Documents/projects/incubator-helix/helix-agent/src= /main/scripts/integration-test/script",
=A0=A0=A0 "MASTER-SLAVE.command" : "simpleHttpClient.py MAST= ER-SLAVE",
=A0=A0=A0 "MASTER-SLAVE.command.workingDir" : = "/Users/kgopalak/Documents/projects/incubator-helix/helix-agent/src/ma= in/scripts/integration-test/script",
=A0=A0=A0 "SLAVE-OFFLINE.command" : "simpleHttpClient.py SLA= VE-OFFLINE",
=A0=A0=A0 "SLAVE-OFFLINE.command.workingDir"= : "/Users/kgopalak/Documents/projects/incubator-helix/helix-agent/src= /main/scripts/integration-test/script",
=A0=A0=A0 "OFFLINE-DROPPED.command" : "nop"
=A0 }}

You can set these configs using helix admin api.
Via java api
=A0=A0=A0 HelixConfigScope scope =3D
=A0=A0= =A0=A0=A0=A0=A0 new HelixConfigScopeBuilder(ConfigScopeProperty.CLUSTER).fo= rCluster(clusterName).build();
=A0=A0=A0 CommandConfig.Builder builder =3D new CommandConfig.Builder();=A0=A0=A0 CommandConfig cmdConfig =3D
=A0=A0=A0=A0=A0=A0=A0 builder.set= Transition("SLAVE", "MASTER")
=A0=A0=A0=A0=A0=A0=A0= =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 .setCommand("simpleHttpClient.py SLAVE-= MASTER")
=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 .setCommandWorkingDir(w= orkingDir).setCommandTimeout("0")
=A0=A0=A0=A0=A0=A0=A0=A0=A0= =A0=A0 =A0 =A0 =A0 .setPidFile(pidFile)
=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0= =A0=A0=A0=A0=A0=A0=A0 .build();
=A0=A0=A0 configAccessor.set(scope, cmdC= onfig.toKeyValueMap());

Using Helix Admin
USAGE: --setConfig <ConfigScope(e.g.= RESOURCE) ConfigScopeArgs(e.g. myCluster,testDB) KeyValueMap(e.g. k1=3Dv1,= k2=3Dv2)>=A0=A0=A0=A0=A0 Set configs
example:=A0
=A0=A0=A0=A0=A0 = ./helix-admin.sh --zkSvr localhost:2181 --setConfig CLUSTER VAPH_DEV SLAVE-= MASTER.command=3D<cmd>,SLAVE-MASTER.workingDir=3D<dir>, SLAVE-M= ASTER.command.pidfile=3D<pid_file>


Let us know if this does not work.

thanks,
Kis= hore G





<= div>


On Thu, Jan 23, 2014 at 4:58 AM, Benoit Vanderbeck <benoit.vanderb= eck@vaph.be> wrote:
Hi,

i'm trying to run the helix-age= nt. But I can't seem to figure out how to configure the command that sh= ould be run when the statetransition changes from offline to online.
I have tried with a simple setup : One instance and one resource
=

IdealState for simpleHttpServer:
{
=A0 "id" : "simp= leHttpServer",
=A0 "mapFields&quo= t; : {
=A0 =A0 "simpleHttpServer_0&quo= t; : {
=A0 =A0 =A0 "vbxl003_80" : "ONLIN= E"
=A0 =A0 },
=A0 =A0 "simpleHttpServer_1" : {
=A0 =A0 =A0 "vbxl003_80" : "ONLINE"
=A0 =A0 },
=A0 = =A0 "simpleHttpServer_2" : {
=A0 = =A0 =A0 "vbxl003_80" : "ONLINE"
<= div>
=A0 =A0 }
=A0 },
=A0 "= ;listFields" : {
=A0 =A0 "simpleH= ttpServer_0" : [ "vbxl003_80" ],
=
=A0 =A0 "simpleHttpServer_1" : [ "vbxl003_80" ],
<= /div>
=A0 =A0 "simpleHttpServer_2" : [ "= vbxl003_80" ]
=A0 },
=
=A0 "simpleFields" : {
=A0 = =A0 "COMMAND" : "/tmp/simpleHttpServer.py",
=
=A0 =A0 "IDEAL_STATE_MODE" : "AUTO"= ;,
=A0 =A0 "NUM_PARTITIONS" : "3&quo= t;,
=A0 =A0 "OFFLINE-ONLINE.COMMAND&qu= ot; : "/tmp/simpleHttpServer.py",
=A0 =A0 "REBALANCE_MODE" : "SEMI_AUTO",
=A0 =A0 "REPLICAS" : "1",
=A0 =A0 "STATE_MODEL_DEF_REF" : &qu= ot;OnlineOffline",
=A0 =A0 "STATE= _MODEL_FACTORY_NAME" : "DEFAULT"
=A0 }
}

ExternalView f= or simpleHttpServer:
{
=A0 "id" : "simpleHttpServer",
=A0 "mapFields" : {
=
=A0 },
=A0 "listFields" : {<= /div>
=A0 },
=A0 "= ;simpleFields" : {
=A0 =A0 "BUCKE= T_SIZE" : "0"
=A0 }


But when I = start the agent I get following error :

677 = =A0[pool-2-thread-1] ERROR org.apache.helix.messaging.handling.HelixStateTr= ansitionHandler =A0- Exception while executing a state transition task simp= leHttpServer_0
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke= 0(Native Method)
at = sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:3= 9)
at sun.reflect.Delegating= MethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
<= span style=3D"white-space:pre-wrap"> at java.lang.reflect.Method.inv= oke(Method.java:597)
at org.apache.helix.messa= ging.handling.HelixStateTransitionHandler.invoke(HelixStateTransitionHandle= r.java:299)
at org.a= pache.helix.messaging.handling.HelixStateTransitionHandler.handleMessage(He= lixStateTransitionHandler.java:253)
at org.apache.helix.messa= ging.handling.HelixTask.call(HelixTask.java:85)
at org.apache.helix.messaging.handling.HelixTas= k.call(HelixTask.java:42)
at java.util.concurrent.F= utureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.= java:138)
at java.util.concurrent.T= hreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoo= lExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(T= hread.java:662)
Caused by: java.lang.Exception: Unable to find co= mmand for transition from:OFFLINE to:ONLINE


So the agent can not find what command should be execut= ed when going from OFFLINE to ONLINE.

I also tried= to edit the configs tree in the znode

get /VAPH_DEV/CONFIGS/RESOURCE/simpleHttpServer
<= /div>
{
=A0 "id" : = "simpleHttpServer",
=A0 "sim= pleFields" : {
=A0 =A0 "COMMAND" : "/tmp/simpleH= ttpServer.py",
=A0 =A0 "WORKING_D= IR" : "/tmp",
=A0 =A0 "= PID" : "/tmp/mypid",
=A0 =A0 "OFFLINE-ONLINE.COMMAND" : &qu= ot;/tmp/simpleHttpServer.py"
=A0 },
=A0 "listFields" : {
=A0 },
=A0 "mapFields" = : {
=A0 }
}=

But that didn't= work either.

Can someone point me in the right direction ? Where en = how do define the command that the agent should execute.

Regards,

Beno=EEt
--
Beno=EEt Vanderbeck
Systeembeheerder VAPH


Raadpleeg uw VAPH-dossier online via=A0http://mijn.vaph.be.
------------------------------------------= ---------------------------------------
http://www.vaph.be/d= isclaimer





--
Beno=EEt Vanderbeck
Systeembeheerder VAPH


Raadpleeg uw VAPH-dossier online via=A0http://mijn.vaph.be.
------------------------------------------= ---------------------------------------
http://www.vaph.be/d= isclaimer
--047d7bdc7e2a05122004f14c13c2--