apex-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sanjay Pujare <san...@datatorrent.com>
Subject Re: [EXTERNAL] Re: KafkaSinglePortInputOperator
Date Wed, 07 Dec 2016 17:54:15 GMT
Just out of curiosity – what was the problem? Why were the ssl.* project properties not seen
by the Kafka consumer?

 

From: "Raja.Aravapalli" <Raja.Aravapalli@target.com>
Reply-To: <users@apex.apache.org>
Date: Wednesday, December 7, 2016 at 7:56 AM
To: "users@apex.apache.org" <users@apex.apache.org>
Subject: Re: [EXTERNAL] Re: KafkaSinglePortInputOperator

 

 

Thanks a tonnn for the support today Chaitanya!!

 

We were able to successfully download messages from Kafka SSL Secured Topics!!

 

Thanks you very much!!

 

 

Regards,

Raja.

 

From: Chaitanya Chebolu <chaitanya@datatorrent.com>
Reply-To: "users@apex.apache.org" <users@apex.apache.org>
Date: Wednesday, December 7, 2016 at 11:28 AM
To: "users@apex.apache.org" <users@apex.apache.org>
Subject: Re: [EXTERNAL] Re: KafkaSinglePortInputOperator

 

Raja, 

 

  Issue is the SSL properties(ssl.*.*) are not reflected to Kafka consumer.

  Could you please share the complete project ?

 

Thanks,

Chaitanya

  

 

On Wed, Dec 7, 2016 at 7:39 AM, Raja.Aravapalli <Raja.Aravapalli@target.com> wrote:

Hi Chaitanya, 

 

Any other thoughts on how I can fix this ??

 

Are Apex doesn’t yet support SSL secured topics ?

 

 

Thanks a lot.

 

Regards,

Raja.

 

From: "Raja.Aravapalli" <Raja.Aravapalli@target.com>
Reply-To: "users@apex.apache.org" <users@apex.apache.org>
Date: Tuesday, December 6, 2016 at 5:32 PM


To: "users@apex.apache.org" <users@apex.apache.org>
Subject: Re: [EXTERNAL] Re: KafkaSinglePortInputOperator

 

 

 

I added the below line as said…  I cannot see any exceptions also!!!!

 

Still nothing is happening L

 

I am not sure, why these below as always showing as null… even though I set them in my Application.java
class!! Any help on how to set these properties ???

 

ssl.keystore.location = null

ssl.truststore.location = null

ssl.keystore.password = null

 

 

Thanks a lot in advance.

 

Regards,

Raja.

 

From: Chaitanya Chebolu <chaitanya@datatorrent.com>
Reply-To: "users@apex.apache.org" <users@apex.apache.org>
Date: Tuesday, December 6, 2016 at 5:17 PM
To: "users@apex.apache.org" <users@apex.apache.org>
Subject: Re: [EXTERNAL] Re: KafkaSinglePortInputOperator

 

Raja, 

 

   Please set the consumerProps to the KafkaSinglePortInputOperator.

   Add the below line in your application:

  KafkaSinglePortInputOperator in = dag.addOperator("in", new KafkaSinglePortInputOperator());

  ------

   in.setConsumerProps(props);

 

 Please let me know, if you are still facing issues.

 

Regards,

Chaitanya

   

 

 

On Tue, Dec 6, 2016 at 5:00 PM, Raja.Aravapalli <Raja.Aravapalli@target.com> wrote:

 

Find below the log I am observing:

 

2016-12-06 05:17:37,264 INFO  kafka.AbstractKafkaInputOperator (AbstractKafkaInputOperator.java:initPartitioner(311))
- Initialize Partitioner

2016-12-06 05:17:37,265 INFO  kafka.AbstractKafkaInputOperator (AbstractKafkaInputOperator.java:initPartitioner(324))
- Actual Partitioner is class org.apache.apex.malhar.kafka.OneToOnePartitioner

2016-12-06 05:17:37,280 INFO  consumer.ConsumerConfig (AbstractConfig.java:logAll(165)) -
ConsumerConfig values: 

                metric.reporters = []

                metadata.max.age.ms = 300000

                value.deserializer = class org.apache.kafka.common.serialization.ByteArrayDeserializer

                group.id = org.apache.apex.malhar.kafka.AbstractKafkaInputOperatorMETA_GROUP

                partition.assignment.strategy = [org.apache.kafka.clients.consumer.RangeAssignor]

                reconnect.backoff.ms = 50

                sasl.kerberos.ticket.renew.window.factor = 0.8

                max.partition.fetch.bytes = 1048576

                bootstrap.servers = [10.66.137.116:9093]

                retry.backoff.ms = 100

                sasl.kerberos.kinit.cmd = /usr/bin/kinit

                sasl.kerberos.service.name = null

                sasl.kerberos.ticket.renew.jitter = 0.05

                ssl.keystore.type = JKS

                ssl.trustmanager.algorithm = PKIX

                enable.auto.commit = false

                ssl.key.password = null

                fetch.max.wait.ms = 500

                sasl.kerberos.min.time.before.relogin = 60000

                connections.max.idle.ms = 540000

                ssl.truststore.password = null

                session.timeout.ms = 30000

                metrics.num.samples = 2

                client.id = 

                ssl.endpoint.identification.algorithm = null

                key.deserializer = class org.apache.kafka.common.serialization.ByteArrayDeserializer

                ssl.protocol = TLS

                check.crcs = true

                request.timeout.ms = 40000

                ssl.provider = null

                ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]

                ssl.keystore.location = null

                heartbeat.interval.ms = 3000

                auto.commit.interval.ms = 5000

                receive.buffer.bytes = 32768

                ssl.cipher.suites = null

                ssl.truststore.type = JKS

                security.protocol = PLAINTEXT

                ssl.truststore.location = null

                ssl.keystore.password = null

                ssl.keymanager.algorithm = SunX509

                metrics.sample.window.ms = 30000

                fetch.min.bytes = 1024

                send.buffer.bytes = 131072

                auto.offset.reset = latest

 

2016-12-06 05:17:37,385 INFO  utils.AppInfoParser (AppInfoParser.java:<init>(82)) -
Kafka version : 0.9.0.0

2016-12-06 05:17:37,385 INFO  utils.AppInfoParser (AppInfoParser.java:<init>(83)) -
Kafka commitId : fc7243c2af4b2b4a

 

 

 

Regards,

Raja.

 

From: Chaitanya Chebolu <chaitanya@datatorrent.com>
Reply-To: "users@apex.apache.org" <users@apex.apache.org>
Date: Tuesday, December 6, 2016 at 4:28 PM
To: "users@apex.apache.org" <users@apex.apache.org>
Subject: [EXTERNAL] Re: KafkaSinglePortInputOperator

 

Hi Raja, 

 

  Could you please share the Application Master logs and Kafka operator container logs.

 

Regards,

Chaitanya

 

On Tue, Dec 6, 2016 at 4:17 PM, Raja.Aravapalli <Raja.Aravapalli@target.com> wrote:

 

Hi Team, 

 

I am using “KafkaSinglePortInputOperator” to connect to a SSL Secured topic in Kafka 0.9!!

 

Unfortunately… my apex application is not going to “RUNNING” state…!! Only staying
in ACCEPTED State and then going into FAILED statie!! I don’t see much information in the
logs…!! L

 

Can someone please help fix the issue…. We have immediate need to read messages from kafka
0.9 SSL configured topics… 

 

Please advise!

 

 

Thanks very much in advance.

 

 

Regards,

Raja.

 

 

 


Mime
View raw message