incubator-s4-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sergio Vavassori <svavass...@conwet.com>
Subject Re: Information about Apache S4 and Helix
Date Thu, 09 Jan 2014 16:22:10 GMT
On 12 December 2013 18:54, kishore g <g.kishore@gmail.com> wrote:

>
> https://git-wip-us.apache.org/repos/asf?p=incubator-s4.git;a=blob;f=subprojects/s4-tools/src/main/java/org/apache/s4/tools/helix/CreateTask.java;h=36a45bc5bef5e4c4f56bfb4c740abb315197ca21;hb=59bcb94cc727cb1600074e6da81330953aeaa566
>
> When you add the task, you can use pe type s the name of the task and
> specify the streams it is interested in.
>
> Give it a try :-)
>

Hello,

First of all I'm sorry for the delay in reply but Christmas vacation didn't
help.

I tried to use the CreateTask but it raised a ZK Exception about S4 cluster
not present (even when it was created manually).

Is it correct that CreateTask creates all resources under
S4HelixConstants.HELIX_CLUSTER_NAME and a property "GROUP" with the
clusterName instead of using the clusterName as reference for
ConfigScopeBuilder?

Then I found this repository: https://github.com/kishoreg/incubator-s4 and
even if it's older and needed some patches to make it work, I was able to
run it. Nevertheless the application didn't respond as supposed. (echo
"Bob" | nc localhost 16000 didn't return).
In details I did the following:

./gradlew install
./gradlew s4-tools:installApp

./s4 zkServer &
./s4 newCluster -c=cluster1 -nbNodes=2 -flp=12000
./s4 newCluster -c=cluster2 -nbNodes=1 -flp=16000
./s4 createTask  -zk localhost:2181 -c cluster1 -id names -t consumer
-p 4 -r 1 -s names
./s4 createTask  -zk localhost:2181 -c cluster2 -id names -t producer
-p 1 -r 0 -s names

./s4 newApp myApp -parentDir=/tmp
cd /tmp/myApp
./s4 s4r -a=hello.HelloApp -b=/tmp/myApp/build.gradle myApp

./s4 deployApp -c=cluster1 -s4r=/tmp/myApp/build/libs/myApp.s4r
-appName=myApp -zk=localhost:2181 &
./s4 adapter -c=cluster2 -zk=localhost:2181
-p=s4.adapter.output.stream=names,s4.instance.name=localhost_16000 &

(I had to add the "-p" parameters because guava was requiring a value for
that)

and these are their statuses


./s4 status -c=cluster1
-----------------------

calling referenced s4 script : /home/sergio/upm/repos/s4-helix/s4
1 [main] INFO org.apache.helix.manager.zk.ZKHelixManager  - Create a
zk-based cluster manager. clusterName:cluster1, instanceName:ADMIN,
type:ADMINISTRATOR, zkSvr:localhost:2181

57 [main] INFO org.apache.helix.messaging.handling.HelixTaskExecutor
- Adding msg factory for type TASK_REPLY threadpool size 40
59 [main] INFO org.apache.helix.manager.zk.ZKHelixManager  -
ClusterManager.connect()

135 [main] INFO org.apache.helix.manager.zk.ZkStateChangeListener  -
KeeperState: SyncConnected, zookeeper:State:CONNECTED Timeout:30000
sessionid:0x14376f2de0a002b local:/127.0.0.1:41285
remoteserver:localhost/127.0.0.1:2181 lastZxid:0 xid:1 sent:1 recv:1
queuedpkts:0 pendingresp:0 queuedevents:0

136 [main] INFO org.apache.helix.manager.zk.ZKHelixManager  - Handling
new session, session id:14376f2de0a002b, instance:ADMIN, instanceTye:
ADMINISTRATOR, cluster: cluster1
136 [main] INFO org.apache.helix.manager.zk.ZKHelixManager  -
State:CONNECTED Timeout:30000 sessionid:0x14376f2de0a002b
local:/127.0.0.1:41285 remoteserver:localhost/127.0.0.1:2181
lastZxid:0 xid:1 sent:1 recv:1 queuedpkts:0 pendingresp:0
queuedevents:0

Cluster Status
----------------------------------------------------------------------------------------------------------------------------------

                Nodes

    Cluster Name           Nodes          Active
--------------------------------------------------------------------------------
                                                       Node id
                Host                         Port    Active

----------------------------------------------------------------------------------------------------------------------------------
      cluster1         2                   2        localhost_12000
                localhost                   12000       Y

                                                    localhost_12001
                localhost                   12001       Y
App Status
----------------------------------------------------------------------------------------------------------------------------------

        Name              Cluster
            URI
----------------------------------------------------------------------------------------------------------------------------------

       myApp              cluster1
file:/tmp/myApp/build/libs/myApp.s4r
----------------------------------------------------------------------------------------------------------------------------------

                     Node id                            DEPLOYED
----------------------------------------------------------------------------------------------------------------------------------
                 localhost_12000                           Y

                 localhost_12001                           Y
----------------------------------------------------------------------------------------------------------------------------------
App Status
----------------------------------------------------------------------------------------------------------------------------------

        Name              Cluster
            URI
----------------------------------------------------------------------------------------------------------------------------------

      adapter             cluster1
file:/tmp/myApp/build/libs/myApp.s4r
----------------------------------------------------------------------------------------------------------------------------------

                     Node id                            DEPLOYED
----------------------------------------------------------------------------------------------------------------------------------
                 localhost_12000                           Y

                 localhost_12001                           Y
----------------------------------------------------------------------------------------------------------------------------------
Task Status
----------------------------------------------------------------------------------------------------------------------------------

      Task Id             Cluster
        Description
----------------------------------------------------------------------------------------------------------------------------------

       names              cluster1
       names producer
----------------------------------------------------------------------------------------------------------------------------------

                    Partition                            State
----------------------------------------------------------------------------------------------------------------------------------

                     names_0
     localhost_12000:STANDBY, localhost_12001:LEADER
                     names_1
     localhost_12000:LEADER, localhost_12001:STANDBY

                     names_2
     localhost_12000:STANDBY, localhost_12001:LEADER
                     names_3
     localhost_12000:LEADER, localhost_12001:STANDBY

----------------------------------------------------------------------------------------------------------------------------------



./s4 status -c=cluster2
-----------------------

calling referenced s4 script : /home/sergio/upm/repos/s4-helix/s4
0 [main] INFO org.apache.helix.manager.zk.ZKHelixManager  - Create a
zk-based cluster manager. clusterName:cluster2, instanceName:ADMIN,
type:ADMINISTRATOR, zkSvr:localhost:2181

57 [main] INFO org.apache.helix.messaging.handling.HelixTaskExecutor
- Adding msg factory for type TASK_REPLY threadpool size 40
60 [main] INFO org.apache.helix.manager.zk.ZKHelixManager  -
ClusterManager.connect()

133 [main] INFO org.apache.helix.manager.zk.ZkStateChangeListener  -
KeeperState: SyncConnected, zookeeper:State:CONNECTED Timeout:30000
sessionid:0x14376f2de0a0030 local:/127.0.0.1:41291
remoteserver:localhost/127.0.0.1:2181 lastZxid:0 xid:1 sent:1 recv:1
queuedpkts:0 pendingresp:0 queuedevents:0

133 [main] INFO org.apache.helix.manager.zk.ZKHelixManager  - Handling
new session, session id:14376f2de0a0030, instance:ADMIN, instanceTye:
ADMINISTRATOR, cluster: cluster2
133 [main] INFO org.apache.helix.manager.zk.ZKHelixManager  -
State:CONNECTED Timeout:30000 sessionid:0x14376f2de0a0030
local:/127.0.0.1:41291 remoteserver:localhost/127.0.0.1:2181
lastZxid:0 xid:1 sent:1 recv:1 queuedpkts:0 pendingresp:0
queuedevents:0

Cluster Status
----------------------------------------------------------------------------------------------------------------------------------

                Nodes

    Cluster Name           Nodes          Active
--------------------------------------------------------------------------------
                                                       Node id
                Host                         Port    Active

----------------------------------------------------------------------------------------------------------------------------------
      cluster2         1                   1        localhost_16000
                localhost                   16000       Y

App Status
----------------------------------------------------------------------------------------------------------------------------------
        Name              Cluster
            URI

----------------------------------------------------------------------------------------------------------------------------------
      adapter             cluster2
file:/tmp/myApp/build/libs/myApp.s4r

----------------------------------------------------------------------------------------------------------------------------------
                     Node id                            DEPLOYED
----------------------------------------------------------------------------------------------------------------------------------

                 localhost_16000                           Y
----------------------------------------------------------------------------------------------------------------------------------
Task Status
----------------------------------------------------------------------------------------------------------------------------------

      Task Id             Cluster
        Description
----------------------------------------------------------------------------------------------------------------------------------

       names              cluster2
       names producer
----------------------------------------------------------------------------------------------------------------------------------

                    Partition                            State
----------------------------------------------------------------------------------------------------------------------------------

                     names_0
                  localhost_16000:LEADER
------------------------------------------------------------------------------------------------------------------------


Am I missing something?

Is S4-110-new branch (commit 59bcb94cc727cb1600074e6da81330953aeaa566 ) the
correct one to use or is best to use the other git repo?
In the case it's the S4-110-new branch, would you mind to provide a working
command-list for the same basic example?

I saw there is a request to support repartitioning in Helix, but nobody
replied (
http://mail-archives.apache.org/mod_mbox/helix-dev/201304.mbox/%3CJIRA.12640790.1365088831943.107882.1365088877867@arcas%3E).

Is it too much application-dependent or are there any problems with Helix
architecture?

Best Regards,
Sergio Vavassori

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message