incubator-s4-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From kishore g <g.kish...@gmail.com>
Subject Re: Information about Apache S4 and Helix
Date Thu, 09 Jan 2014 19:29:52 GMT
I will take a stab at making S4-110-new branch working.

Repartitioning is dependent on the application and is hard to implement.
Helix architecture does not restrict one from implementing repartitioning.
In general, partitioning splitting is hard and a common work around is to
over partition and simply move partition around when new machines are added.

thanks,
Kishore G


On Thu, Jan 9, 2014 at 8:22 AM, Sergio Vavassori <svavassori@conwet.com>wrote:

> On 12 December 2013 18:54, kishore g <g.kishore@gmail.com> wrote:
>
> >
> >
> https://git-wip-us.apache.org/repos/asf?p=incubator-s4.git;a=blob;f=subprojects/s4-tools/src/main/java/org/apache/s4/tools/helix/CreateTask.java;h=36a45bc5bef5e4c4f56bfb4c740abb315197ca21;hb=59bcb94cc727cb1600074e6da81330953aeaa566
> >
> > When you add the task, you can use pe type s the name of the task and
> > specify the streams it is interested in.
> >
> > Give it a try :-)
> >
>
> Hello,
>
> First of all I'm sorry for the delay in reply but Christmas vacation didn't
> help.
>
> I tried to use the CreateTask but it raised a ZK Exception about S4 cluster
> not present (even when it was created manually).
>
> Is it correct that CreateTask creates all resources under
> S4HelixConstants.HELIX_CLUSTER_NAME and a property "GROUP" with the
> clusterName instead of using the clusterName as reference for
> ConfigScopeBuilder?
>
> Then I found this repository: https://github.com/kishoreg/incubator-s4 and
> even if it's older and needed some patches to make it work, I was able to
> run it. Nevertheless the application didn't respond as supposed. (echo
> "Bob" | nc localhost 16000 didn't return).
> In details I did the following:
>
> ./gradlew install
> ./gradlew s4-tools:installApp
>
> ./s4 zkServer &
> ./s4 newCluster -c=cluster1 -nbNodes=2 -flp=12000
> ./s4 newCluster -c=cluster2 -nbNodes=1 -flp=16000
> ./s4 createTask  -zk localhost:2181 -c cluster1 -id names -t consumer
> -p 4 -r 1 -s names
> ./s4 createTask  -zk localhost:2181 -c cluster2 -id names -t producer
> -p 1 -r 0 -s names
>
> ./s4 newApp myApp -parentDir=/tmp
> cd /tmp/myApp
> ./s4 s4r -a=hello.HelloApp -b=/tmp/myApp/build.gradle myApp
>
> ./s4 deployApp -c=cluster1 -s4r=/tmp/myApp/build/libs/myApp.s4r
> -appName=myApp -zk=localhost:2181 &
> ./s4 adapter -c=cluster2 -zk=localhost:2181
> -p=s4.adapter.output.stream=names,s4.instance.name=localhost_16000 &
>
> (I had to add the "-p" parameters because guava was requiring a value for
> that)
>
> and these are their statuses
>
>
> ./s4 status -c=cluster1
> -----------------------
>
> calling referenced s4 script : /home/sergio/upm/repos/s4-helix/s4
> 1 [main] INFO org.apache.helix.manager.zk.ZKHelixManager  - Create a
> zk-based cluster manager. clusterName:cluster1, instanceName:ADMIN,
> type:ADMINISTRATOR, zkSvr:localhost:2181
>
> 57 [main] INFO org.apache.helix.messaging.handling.HelixTaskExecutor
> - Adding msg factory for type TASK_REPLY threadpool size 40
> 59 [main] INFO org.apache.helix.manager.zk.ZKHelixManager  -
> ClusterManager.connect()
>
> 135 [main] INFO org.apache.helix.manager.zk.ZkStateChangeListener  -
> KeeperState: SyncConnected, zookeeper:State:CONNECTED Timeout:30000
> sessionid:0x14376f2de0a002b local:/127.0.0.1:41285
> remoteserver:localhost/127.0.0.1:2181 lastZxid:0 xid:1 sent:1 recv:1
> queuedpkts:0 pendingresp:0 queuedevents:0
>
> 136 [main] INFO org.apache.helix.manager.zk.ZKHelixManager  - Handling
> new session, session id:14376f2de0a002b, instance:ADMIN, instanceTye:
> ADMINISTRATOR, cluster: cluster1
> 136 [main] INFO org.apache.helix.manager.zk.ZKHelixManager  -
> State:CONNECTED Timeout:30000 sessionid:0x14376f2de0a002b
> local:/127.0.0.1:41285 remoteserver:localhost/127.0.0.1:2181
> lastZxid:0 xid:1 sent:1 recv:1 queuedpkts:0 pendingresp:0
> queuedevents:0
>
> Cluster Status
>
> ----------------------------------------------------------------------------------------------------------------------------------
>
>                 Nodes
>
>     Cluster Name           Nodes          Active
>
> --------------------------------------------------------------------------------
>                                                        Node id
>                 Host                         Port    Active
>
>
> ----------------------------------------------------------------------------------------------------------------------------------
>       cluster1         2                   2        localhost_12000
>                 localhost                   12000       Y
>
>                                                     localhost_12001
>                 localhost                   12001       Y
> App Status
>
> ----------------------------------------------------------------------------------------------------------------------------------
>
>         Name              Cluster
>             URI
>
> ----------------------------------------------------------------------------------------------------------------------------------
>
>        myApp              cluster1
> file:/tmp/myApp/build/libs/myApp.s4r
>
> ----------------------------------------------------------------------------------------------------------------------------------
>
>                      Node id                            DEPLOYED
>
> ----------------------------------------------------------------------------------------------------------------------------------
>                  localhost_12000                           Y
>
>                  localhost_12001                           Y
>
> ----------------------------------------------------------------------------------------------------------------------------------
> App Status
>
> ----------------------------------------------------------------------------------------------------------------------------------
>
>         Name              Cluster
>             URI
>
> ----------------------------------------------------------------------------------------------------------------------------------
>
>       adapter             cluster1
> file:/tmp/myApp/build/libs/myApp.s4r
>
> ----------------------------------------------------------------------------------------------------------------------------------
>
>                      Node id                            DEPLOYED
>
> ----------------------------------------------------------------------------------------------------------------------------------
>                  localhost_12000                           Y
>
>                  localhost_12001                           Y
>
> ----------------------------------------------------------------------------------------------------------------------------------
> Task Status
>
> ----------------------------------------------------------------------------------------------------------------------------------
>
>       Task Id             Cluster
>         Description
>
> ----------------------------------------------------------------------------------------------------------------------------------
>
>        names              cluster1
>        names producer
>
> ----------------------------------------------------------------------------------------------------------------------------------
>
>                     Partition                            State
>
> ----------------------------------------------------------------------------------------------------------------------------------
>
>                      names_0
>      localhost_12000:STANDBY, localhost_12001:LEADER
>                      names_1
>      localhost_12000:LEADER, localhost_12001:STANDBY
>
>                      names_2
>      localhost_12000:STANDBY, localhost_12001:LEADER
>                      names_3
>      localhost_12000:LEADER, localhost_12001:STANDBY
>
>
> ----------------------------------------------------------------------------------------------------------------------------------
>
>
>
> ./s4 status -c=cluster2
> -----------------------
>
> calling referenced s4 script : /home/sergio/upm/repos/s4-helix/s4
> 0 [main] INFO org.apache.helix.manager.zk.ZKHelixManager  - Create a
> zk-based cluster manager. clusterName:cluster2, instanceName:ADMIN,
> type:ADMINISTRATOR, zkSvr:localhost:2181
>
> 57 [main] INFO org.apache.helix.messaging.handling.HelixTaskExecutor
> - Adding msg factory for type TASK_REPLY threadpool size 40
> 60 [main] INFO org.apache.helix.manager.zk.ZKHelixManager  -
> ClusterManager.connect()
>
> 133 [main] INFO org.apache.helix.manager.zk.ZkStateChangeListener  -
> KeeperState: SyncConnected, zookeeper:State:CONNECTED Timeout:30000
> sessionid:0x14376f2de0a0030 local:/127.0.0.1:41291
> remoteserver:localhost/127.0.0.1:2181 lastZxid:0 xid:1 sent:1 recv:1
> queuedpkts:0 pendingresp:0 queuedevents:0
>
> 133 [main] INFO org.apache.helix.manager.zk.ZKHelixManager  - Handling
> new session, session id:14376f2de0a0030, instance:ADMIN, instanceTye:
> ADMINISTRATOR, cluster: cluster2
> 133 [main] INFO org.apache.helix.manager.zk.ZKHelixManager  -
> State:CONNECTED Timeout:30000 sessionid:0x14376f2de0a0030
> local:/127.0.0.1:41291 remoteserver:localhost/127.0.0.1:2181
> lastZxid:0 xid:1 sent:1 recv:1 queuedpkts:0 pendingresp:0
> queuedevents:0
>
> Cluster Status
>
> ----------------------------------------------------------------------------------------------------------------------------------
>
>                 Nodes
>
>     Cluster Name           Nodes          Active
>
> --------------------------------------------------------------------------------
>                                                        Node id
>                 Host                         Port    Active
>
>
> ----------------------------------------------------------------------------------------------------------------------------------
>       cluster2         1                   1        localhost_16000
>                 localhost                   16000       Y
>
> App Status
>
> ----------------------------------------------------------------------------------------------------------------------------------
>         Name              Cluster
>             URI
>
>
> ----------------------------------------------------------------------------------------------------------------------------------
>       adapter             cluster2
> file:/tmp/myApp/build/libs/myApp.s4r
>
>
> ----------------------------------------------------------------------------------------------------------------------------------
>                      Node id                            DEPLOYED
>
> ----------------------------------------------------------------------------------------------------------------------------------
>
>                  localhost_16000                           Y
>
> ----------------------------------------------------------------------------------------------------------------------------------
> Task Status
>
> ----------------------------------------------------------------------------------------------------------------------------------
>
>       Task Id             Cluster
>         Description
>
> ----------------------------------------------------------------------------------------------------------------------------------
>
>        names              cluster2
>        names producer
>
> ----------------------------------------------------------------------------------------------------------------------------------
>
>                     Partition                            State
>
> ----------------------------------------------------------------------------------------------------------------------------------
>
>                      names_0
>                   localhost_16000:LEADER
>
> ------------------------------------------------------------------------------------------------------------------------
>
>
> Am I missing something?
>
> Is S4-110-new branch (commit 59bcb94cc727cb1600074e6da81330953aeaa566 ) the
> correct one to use or is best to use the other git repo?
> In the case it's the S4-110-new branch, would you mind to provide a working
> command-list for the same basic example?
>
> I saw there is a request to support repartitioning in Helix, but nobody
> replied (
>
> http://mail-archives.apache.org/mod_mbox/helix-dev/201304.mbox/%3CJIRA.12640790.1365088831943.107882.1365088877867@arcas%3E
> ).
>
> Is it too much application-dependent or are there any problems with Helix
> architecture?
>
> Best Regards,
> Sergio Vavassori
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message