storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mayur Mohite <mayur.moh...@vizury.com>
Subject Only single task executing each bolt and spout
Date Tue, 01 Apr 2014 12:40:55 GMT
Hi,

We are trying to deploy a trident topology on a cluster of 3 machines.
However when we run this topology on a cluster, corresponding to each bolt
and spout there is exactly one task that is processing the tuples. Even
though we have set the parallelism hint > 1 at all the places in the
topology, several executors and tasks are created but the tuples are
processed by exactly 1 task for each spout and bolt.

The topology code is as follows -

        config.setNumWorkers(8);
        config.setMaxSpoutPending(10000);

        TridentKafkaConfig kafkaConfig = new
TridentKafkaConfig(brokerHosts, "test", "storm");
        TransactionalTridentKafkaSpout sp = new
TransactionalTridentKafkaSpout(kafkaConfig);
        topology.newStream("kafka", sp)
                .parallelismHint(2)
                .each(new Fields("str") ,new ComputeAttributes(), new
Fields("d1", "d2", "d3", "d4", "d5", "d6", "d7", "d8", "d9", "d10", "m1"))
.parallelismHint(4).groupBy(new Fields("d1", "d2", "d3","d4", "d5", "d6",
"d7"))
                .persistentAggregate (MysqlState.newFactory(mysqlconfig),
new Fields("d1", "d2", "d3", "d4", "d5", "d6", "d7", "d8", "m1"),
new AggregateCountryLevelData(), new Fields("sum"))
                .parallelismHint(4);


-Mayur

Mime
View raw message