apex-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chaitanya Chebolu <chaita...@datatorrent.com>
Subject Re: [EXTERNAL] Re: KafkaSinglePortInputOperator
Date Thu, 08 Dec 2016 06:58:12 GMT
There was a bug in Malhar-3.4.0 and is fixed in Malhar-3.6.0.
JIRA details of this issue is here
<https://issues.apache.org/jira/browse/APEXMALHAR-2241>.

Regards,
Chaitanya

On Wed, Dec 7, 2016 at 11:24 PM, Sanjay Pujare <sanjay@datatorrent.com>
wrote:

> Just out of curiosity – what was the problem? Why were the ssl.* project
> properties not seen by the Kafka consumer?
>
>
>
> *From: *"Raja.Aravapalli" <Raja.Aravapalli@target.com>
> *Reply-To: *<users@apex.apache.org>
> *Date: *Wednesday, December 7, 2016 at 7:56 AM
>
> *To: *"users@apex.apache.org" <users@apex.apache.org>
> *Subject: *Re: [EXTERNAL] Re: KafkaSinglePortInputOperator
>
>
>
>
>
> Thanks a tonnn for the support today Chaitanya!!
>
>
>
> We were able to successfully download messages from Kafka SSL Secured
> Topics!!
>
>
>
> Thanks you very much!!
>
>
>
>
>
> Regards,
>
> Raja.
>
>
>
> *From: *Chaitanya Chebolu <chaitanya@datatorrent.com>
> *Reply-To: *"users@apex.apache.org" <users@apex.apache.org>
> *Date: *Wednesday, December 7, 2016 at 11:28 AM
> *To: *"users@apex.apache.org" <users@apex.apache.org>
> *Subject: *Re: [EXTERNAL] Re: KafkaSinglePortInputOperator
>
>
>
> Raja,
>
>
>
>   Issue is the SSL properties(ssl.*.*) are not reflected to Kafka consumer.
>
>   Could you please share the complete project ?
>
>
>
> Thanks,
>
> Chaitanya
>
>
>
>
>
> On Wed, Dec 7, 2016 at 7:39 AM, Raja.Aravapalli <
> Raja.Aravapalli@target.com> wrote:
>
> Hi Chaitanya,
>
>
>
> Any other thoughts on how I can fix this ??
>
>
>
> Are Apex doesn’t yet support SSL secured topics ?
>
>
>
>
>
> Thanks a lot.
>
>
>
> Regards,
>
> Raja.
>
>
>
> *From: *"Raja.Aravapalli" <Raja.Aravapalli@target.com>
> *Reply-To: *"users@apex.apache.org" <users@apex.apache.org>
> *Date: *Tuesday, December 6, 2016 at 5:32 PM
>
>
> *To: *"users@apex.apache.org" <users@apex.apache.org>
> *Subject: *Re: [EXTERNAL] Re: KafkaSinglePortInputOperator
>
>
>
>
>
>
>
> I added the below line as said…  I cannot see any exceptions also!!!!
>
>
>
> Still nothing is happening L
>
>
>
> I am not sure, why these below as always showing as null… even though I
> set them in my Application.java class!! Any help on how to set these
> properties ???
>
>
>
> ssl.keystore.location = null
>
> ssl.truststore.location = null
>
> ssl.keystore.password = null
>
>
>
>
>
> Thanks a lot in advance.
>
>
>
> Regards,
>
> Raja.
>
>
>
> *From: *Chaitanya Chebolu <chaitanya@datatorrent.com>
> *Reply-To: *"users@apex.apache.org" <users@apex.apache.org>
> *Date: *Tuesday, December 6, 2016 at 5:17 PM
> *To: *"users@apex.apache.org" <users@apex.apache.org>
> *Subject: *Re: [EXTERNAL] Re: KafkaSinglePortInputOperator
>
>
>
> Raja,
>
>
>
>    Please set the consumerProps to the KafkaSinglePortInputOperator.
>
>    Add the below line in your application:
>
>   KafkaSinglePortInputOperator in = dag.addOperator("in", new
> KafkaSinglePortInputOperator());
>
>   ------
>
>    in.setConsumerProps(props);
>
>
>
>  Please let me know, if you are still facing issues.
>
>
>
> Regards,
>
> Chaitanya
>
>
>
>
>
>
>
> On Tue, Dec 6, 2016 at 5:00 PM, Raja.Aravapalli <
> Raja.Aravapalli@target.com> wrote:
>
>
>
> Find below the log I am observing:
>
>
>
> 2016-12-06 05:17:37,264 INFO  kafka.AbstractKafkaInputOperator
> (AbstractKafkaInputOperator.java:initPartitioner(311)) - Initialize
> Partitioner
>
> 2016-12-06 05:17:37,265 INFO  kafka.AbstractKafkaInputOperator
> (AbstractKafkaInputOperator.java:initPartitioner(324)) - Actual
> Partitioner is class org.apache.apex.malhar.kafka.OneToOnePartitioner
>
> 2016-12-06 05:17:37,280 INFO  consumer.ConsumerConfig
> (AbstractConfig.java:logAll(165)) - ConsumerConfig values:
>
>                 metric.reporters = []
>
>                 metadata.max.age.ms = 300000
>
>                 value.deserializer = class org.apache.kafka.common.
> serialization.ByteArrayDeserializer
>
>                 group.id = org.apache.apex.malhar.kafka.
> AbstractKafkaInputOperatorMETA_GROUP
>
>                 partition.assignment.strategy = [org.apache.kafka.clients.
> consumer.RangeAssignor]
>
>                 reconnect.backoff.ms = 50
>
>                 sasl.kerberos.ticket.renew.window.factor = 0.8
>
>                 max.partition.fetch.bytes = 1048576
>
>                 bootstrap.servers = [10.66.137.116:9093]
>
>                 retry.backoff.ms = 100
>
>                 sasl.kerberos.kinit.cmd = /usr/bin/kinit
>
>                 sasl.kerberos.service.name = null
>
>                 sasl.kerberos.ticket.renew.jitter = 0.05
>
>                 ssl.keystore.type = JKS
>
>                 ssl.trustmanager.algorithm = PKIX
>
>                 enable.auto.commit = false
>
>                 ssl.key.password = null
>
>                 fetch.max.wait.ms = 500
>
>                 sasl.kerberos.min.time.before.relogin = 60000
>
>                 connections.max.idle.ms = 540000
>
>                 ssl.truststore.password = null
>
>                 session.timeout.ms = 30000
>
>                 metrics.num.samples = 2
>
>                 client.id =
>
>                 ssl.endpoint.identification.algorithm = null
>
>                 key.deserializer = class org.apache.kafka.common.
> serialization.ByteArrayDeserializer
>
>                 ssl.protocol = TLS
>
>                 check.crcs = true
>
>                 request.timeout.ms = 40000
>
>                 ssl.provider = null
>
>                 ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
>
>                 ssl.keystore.location = null
>
>                 heartbeat.interval.ms = 3000
>
>                 auto.commit.interval.ms = 5000
>
>                 receive.buffer.bytes = 32768
>
>                 ssl.cipher.suites = null
>
>                 ssl.truststore.type = JKS
>
>                 security.protocol = PLAINTEXT
>
>                 ssl.truststore.location = null
>
>                 ssl.keystore.password = null
>
>                 ssl.keymanager.algorithm = SunX509
>
>                 metrics.sample.window.ms = 30000
>
>                 fetch.min.bytes = 1024
>
>                 send.buffer.bytes = 131072
>
>                 auto.offset.reset = latest
>
>
>
> 2016-12-06 05:17:37,385 INFO  utils.AppInfoParser
> (AppInfoParser.java:<init>(82)) - Kafka version : 0.9.0.0
>
> 2016-12-06 05:17:37,385 INFO  utils.AppInfoParser
> (AppInfoParser.java:<init>(83)) - Kafka commitId : fc7243c2af4b2b4a
>
>
>
>
>
>
>
> Regards,
>
> Raja.
>
>
>
> *From: *Chaitanya Chebolu <chaitanya@datatorrent.com>
> *Reply-To: *"users@apex.apache.org" <users@apex.apache.org>
> *Date: *Tuesday, December 6, 2016 at 4:28 PM
> *To: *"users@apex.apache.org" <users@apex.apache.org>
> *Subject: *[EXTERNAL] Re: KafkaSinglePortInputOperator
>
>
>
> Hi Raja,
>
>
>
>   Could you please share the Application Master logs and Kafka operator
> container logs.
>
>
>
> Regards,
>
> Chaitanya
>
>
>
> On Tue, Dec 6, 2016 at 4:17 PM, Raja.Aravapalli <
> Raja.Aravapalli@target.com> wrote:
>
>
>
> Hi Team,
>
>
>
> I am using “KafkaSinglePortInputOperator” to connect to a SSL Secured
> topic in Kafka 0.9!!
>
>
>
> Unfortunately… my apex application is not going to “RUNNING” state…!! Only
> staying in ACCEPTED State and then going into FAILED statie!! I don’t see
> much information in the logs…!! L
>
>
>
> Can someone please help fix the issue…. We have immediate need to read
> messages from kafka 0.9 SSL configured topics…
>
>
>
> Please advise!
>
>
>
>
>
> Thanks very much in advance.
>
>
>
>
>
> Regards,
>
> Raja.
>
>
>
>
>
>
>

Mime
View raw message