spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rachana Srivastava <Rachana.Srivast...@markmonitor.com>
Subject Not all KafkaReceivers processing the data Why?
Date Wed, 14 Sep 2016 19:33:38 GMT
Hello all,

I have created a Kafka topic with 5 partitions.  And I am using createStream receiver API
like following.   But somehow only one receiver is getting the input data. Rest of receivers
are not processign anything.  Can you please help?

JavaPairDStream<String, String> messages = null;

            if(sparkStreamCount > 0){
                // We create an input DStream for each partition of the topic, unify those
streams, and then repartition the unified stream.
                List<JavaPairDStream<String, String>> kafkaStreams = new ArrayList<JavaPairDStream<String,
String>>(sparkStreamCount);
                for (int i = 0; i < sparkStreamCount; i++) {
                                kafkaStreams.add( KafkaUtils.createStream(jssc, contextVal.getString(KAFKA_ZOOKEEPER),
contextVal.getString(KAFKA_GROUP_ID), kafkaTopicMap));
                }
                messages = jssc.union(kafkaStreams.get(0), kafkaStreams.subList(1, kafkaStreams.size()));
            }
            else{
                messages =  KafkaUtils.createStream(jssc, contextVal.getString(KAFKA_ZOOKEEPER),
contextVal.getString(KAFKA_GROUP_ID), kafkaTopicMap);
            }



[cid:image001.png@01D20E84.3558F520]





Mime
View raw message