kylin-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Billy Liu <billy...@apache.org>
Subject Re: Kafka Streaming data - Error while building the Cube
Date Fri, 13 Oct 2017 06:35:13 GMT
And in Kylin tutorial, the topic is kylindemo, in your sample, the topic is
kylin_demo. Please double check the topic name.

2017-10-13 14:27 GMT+08:00 Billy Liu <billyliu@apache.org>:

> If you could package the source code, please try to add more debug log
> when retrieving partition info by Kafka consumer. Check which topic and how
> many partitions you got.
>
> 2017-10-12 23:19 GMT+08:00 Kumar, Manoj H <manoj.h.kumar@jpmorgan.com>:
>
>> I guess KafkaSource Class file where it has
>> enrichSourcePartitionBeforeBuild() to get partition values. There its
>> getting Error out. Do we know how can we test out to find why its coming as
>> 0 for start & end offset?
>>
>>
>>
>> Regards,
>>
>> Manoj
>>
>>
>>
>> *From:* Kumar, Manoj H
>> *Sent:* Thursday, October 12, 2017 3:35 PM
>> *To:* 'user@kylin.apache.org'
>> *Subject:* RE: Kafka Streaming data - Error while building the Cube
>>
>>
>>
>> Yes its there.. I could see the messages..
>>
>>
>>
>> Regards,
>>
>> Manoj
>>
>>
>>
>> *From:* Billy Liu [mailto:billyliu@apache.org <billyliu@apache.org>]
>> *Sent:* Thursday, October 12, 2017 3:11 PM
>>
>> *To:* user
>> *Subject:* Re: Kafka Streaming data - Error while building the Cube
>>
>>
>>
>> STREAMING_SALES_TABLE table reads messages from Kafka topic kylin_demoļ¼Œbut
>> got 0 message.
>>
>>
>>
>> Could you check if the topic has incoming message: bin/kafka-console-consumer.sh
>> --zookeeper localhost:2181 --bootstrap-server localhost:9092 --topic
>> kylin_demo
>>
>>
>>
>> 2017-10-12 17:19 GMT+08:00 Kumar, Manoj H <manoj.h.kumar@jpmorgan.com>:
>>
>> Pls. find below information about consumer config from Kylin log file.
>>
>>
>>
>> 2017-10-11 02:11:43,787 INFO  [pool-8-thread-1]
>> threadpool.DefaultScheduler:123 : Job Fetcher: 0 should running, 0
>> actual running, 0 stopped, 0 ready, 1 already succeed, 0 error, 0
>> discarded, 0 others
>>
>> 2017-10-11 02:12:13,783 INFO  [pool-8-thread-1]
>> threadpool.DefaultScheduler:123 : Job Fetcher: 0 should running, 0
>> actual running, 0 stopped, 0 ready, 1 already succeed, 0 error, 0
>> discarded, 0 others
>>
>> 2017-10-11 02:12:40,734 INFO  [http-bio-7070-exec-3]
>> streaming.StreamingManager:222 : Reloading Streaming Metadata from folder
>> kylin_metadata(key='/streaming')@kylin_metadata@hbase
>>
>> 2017-10-11 02:12:40,760 DEBUG [http-bio-7070-exec-3]
>> streaming.StreamingManager:247 : Loaded 1 StreamingConfig(s)
>>
>> 2017-10-11 02:12:43,789 INFO  [pool-8-thread-1]
>> threadpool.DefaultScheduler:123 : Job Fetcher: 0 should running, 0
>> actual running, 0 stopped, 0 ready, 1 already succeed, 0 error, 0
>> discarded, 0 others
>>
>> 2017-10-11 02:13:13,788 INFO  [pool-8-thread-1]
>> threadpool.DefaultScheduler:123 : Job Fetcher: 0 should running, 0
>> actual running, 0 stopped, 0 ready, 1 already succeed, 0 error, 0
>> discarded, 0 others
>>
>> 2017-10-11 02:13:43,785 INFO  [pool-8-thread-1]
>> threadpool.DefaultScheduler:123 : Job Fetcher: 0 should running, 0
>> actual running, 0 stopped, 0 ready, 1 already succeed, 0 error, 0
>> discarded, 0 others
>>
>> 2017-10-11 02:14:13,789 INFO  [pool-8-thread-1]
>> threadpool.DefaultScheduler:123 : Job Fetcher: 0 should running, 0
>> actual running, 0 stopped, 0 ready, 1 already succeed, 0 error, 0
>> discarded, 0 others
>>
>> 2017-10-11 02:14:43,796 INFO  [pool-8-thread-1]
>> threadpool.DefaultScheduler:123 : Job Fetcher: 0 should running, 0
>> actual running, 0 stopped, 0 ready, 1 already succeed, 0 error, 0
>> discarded, 0 others
>>
>> 2017-10-11 02:15:03,911 DEBUG [http-bio-7070-exec-1]
>> controller.StreamingController:255 : Saving StreamingConfig
>> {"uuid":"8613b0e1-40ac-438c-bdf5-72be4d91c230","last_modifie
>> d":1507705685859,"version":"2.1.0","name":"DEFAULT.
>> STREAMING_SALES_TABLE","type":"kafka"}
>>
>> 2017-10-11 02:15:03,913 DEBUG [http-bio-7070-exec-1]
>> controller.StreamingController:273 : Saving KafkaConfig
>> {"uuid":"87dc6ab5-5141-4bd8-8e00-c16ec86dce41","last_modifie
>> d":1507705685916,"version":"2.1.0","name":"DEFAULT.
>> STREAMING_SALES_TABLE","clusters":[{"brokers":[{"id":"
>> 1","host":"sandbox","port":"9092"}]}],"topic":"kylin_demo","
>> timeout":60000,"parserName":"org.apache.kylin.source.kafka.
>> TimedJsonStreamParser","parserTimeStampField":null,"margin":
>> 0,"parserProperties":"tsColName=order_time"}
>>
>> 2017-10-11 02:15:03,963 DEBUG [pool-7-thread-1] cachesync.Broadcaster:132
>> : Servers in the cluster: [localhost:7070]
>>
>> 2017-10-11 02:15:04,000 DEBUG [pool-7-thread-1] cachesync.Broadcaster:139
>> : Announcing new broadcast event: BroadcastEvent{entity=streaming,
>> event=update, cacheKey=DEFAULT.STREAMING_SALES_TABLE}
>>
>> 2017-10-11 02:15:04,009 DEBUG [pool-7-thread-1] cachesync.Broadcaster:132
>> : Servers in the cluster: [localhost:7070]
>>
>> 2017-10-11 02:15:04,009 DEBUG [pool-7-thread-1] cachesync.Broadcaster:139
>> : Announcing new broadcast event: BroadcastEvent{entity=kafka,
>> event=update, cacheKey=DEFAULT.STREAMING_SALES_TABLE}
>>
>> 2017-10-11 02:15:04,164 DEBUG [http-bio-7070-exec-9]
>> cachesync.Broadcaster:236 : Done broadcasting metadata change:
>> entity=streaming, event=UPDATE, cacheKey=DEFAULT.STREAMING_SALES_TABLE
>>
>> 2017-10-11 02:15:04,192 DEBUG [http-bio-7070-exec-10]
>> cachesync.Broadcaster:236 : Done broadcasting metadata change:
>> entity=kafka, event=UPDATE, cacheKey=DEFAULT.STREAMING_SALES_TABLE
>>
>> 2017-10-11 02:15:13,789 INFO  [pool-8-thread-1]
>> threadpool.DefaultScheduler:123 : Job Fetcher: 0 should running, 0
>> actual running, 0 stopped, 0 ready, 1 already succeed, 0 error, 0
>> discarded, 0 others
>>
>> 2017-10-11 02:15:23,780 DEBUG [http-bio-7070-exec-7] kafka.KafkaSource:83
>> : Last segment doesn't exist, and didn't initiate the start offset, will
>> seek from topic's earliest offset.
>>
>>
>>
>> 2017-10-11 20:50:42,558 INFO  [http-bio-7070-exec-8]
>> utils.AppInfoParser:83 : Kafka version : 0.10.2-kafka-2.2.0
>>
>> 2017-10-11 20:50:42,563 INFO  [http-bio-7070-exec-8]
>> utils.AppInfoParser:84 : Kafka commitId : unknown
>>
>> 2017-10-11 20:50:42,570 DEBUG [http-bio-7070-exec-8]
>> kafka.KafkaSource:105 : Seek end offsets from topic
>>
>> 2017-10-11 20:50:42,570 INFO  [http-bio-7070-exec-8]
>> consumer.ConsumerConfig:196 : ConsumerConfig values:
>>
>>         auto.commit.interval.ms = 5000
>>
>>         auto.offset.reset = latest
>>
>>         bootstrap.servers = [localhost:9092]
>>
>>         check.crcs = true
>>
>>         client.id =
>>
>>         connections.max.idle.ms = 540000
>>
>>         enable.auto.commit = false
>>
>>         exclude.internal.topics = true
>>
>>         fetch.max.bytes = 52428800
>>
>>         fetch.max.wait.ms = 500
>>
>>         fetch.min.bytes = 1
>>
>>         group.id = streaming_cube
>>
>>         heartbeat.interval.ms = 3000
>>
>>         interceptor.classes = null
>>
>>         internal.leave.group.on.close = true
>>
>>         key.deserializer = class org.apache.kafka.common.serial
>> ization.StringDeserializer
>>
>>         max.partition.fetch.bytes = 1048576
>>
>>         max.poll.interval.ms = 300000
>>
>>         max.poll.records = 500
>>
>>         metadata.max.age.ms = 300000
>>
>>         metric.reporters = []
>>
>>         metrics.num.samples = 2
>>
>>         metrics.recording.level = INFO
>>
>>         metrics.sample.window.ms = 30000
>>
>>         partition.assignment.strategy = [class
>> org.apache.kafka.clients.consumer.RangeAssignor]
>>
>>         receive.buffer.bytes = 65536
>>
>>         reconnect.backoff.ms = 50
>>
>>         request.timeout.ms = 305000
>>
>>         retry.backoff.ms = 100
>>
>>         sasl.jaas.config = null
>>
>>         sasl.kerberos.kinit.cmd = /usr/bin/kinit
>>
>>         sasl.kerberos.min.time.before.relogin = 60000
>>
>>         sasl.kerberos.service.name = null
>>
>>         sasl.kerberos.ticket.renew.jitter = 0.05
>>
>>         request.timeout.ms = 305000
>>
>>         retry.backoff.ms = 100
>>
>>         sasl.jaas.config = null
>>
>>         sasl.kerberos.kinit.cmd = /usr/bin/kinit
>>
>>         sasl.kerberos.min.time.before.relogin = 60000
>>
>>         sasl.kerberos.service.name = null
>>
>>         sasl.kerberos.ticket.renew.jitter = 0.05
>>
>>         sasl.kerberos.ticket.renew.window.factor = 0.8
>>
>>         sasl.mechanism = GSSAPI
>>
>>         security.protocol = PLAINTEXT
>>
>>         send.buffer.bytes = 131072
>>
>>         session.timeout.ms = 10000
>>
>>         ssl.cipher.suites = null
>>
>>         ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
>>
>>         ssl.endpoint.identification.algorithm = null
>>
>>         ssl.key.password = null
>>
>>         ssl.keymanager.algorithm = SunX509
>>
>>         ssl.keystore.location = null
>>
>>         ssl.keystore.password = null
>>
>>         ssl.keystore.type = JKS
>>
>>         ssl.protocol = TLS
>>
>>         ssl.provider = null
>>
>>         ssl.secure.random.implementation = null
>>
>>         ssl.trustmanager.algorithm = PKIX
>>
>>         ssl.truststore.location = null
>>
>>         ssl.truststore.password = null
>>
>>         ssl.truststore.type = JKS
>>
>>         value.deserializer = class org.apache.kafka.common.serial
>> ization.StringDeserializer
>>
>>
>>
>> 2017-10-11 20:50:42,573 INFO  [http-bio-7070-exec-8]
>> utils.AppInfoParser:83 : Kafka version : 0.10.2-kafka-2.2.0
>>
>> 2017-10-11 20:50:42,573 INFO  [http-bio-7070-exec-8]
>> utils.AppInfoParser:84 : Kafka commitId : unknown
>>
>> 2017-10-11 20:50:42,586 DEBUG [http-bio-7070-exec-8]
>> kafka.KafkaSource:107 : The end offsets are {0=0}
>>
>> 2017-10-11 20:50:42,588 ERROR [http-bio-7070-exec-8]
>> controller.CubeController:305 : No new message comes, startOffset =
>> endOffset:0
>>
>> java.lang.IllegalArgumentException: No new message comes, startOffset =
>> endOffset:0
>>
>>         at org.apache.kylin.source.kafka.KafkaSource.enrichSourcePartit
>> ionBeforeBuild(KafkaSource.java:134)
>>
>>         at org.apache.kylin.rest.service.JobService.submitJobInternal(J
>> obService.java:236)
>>
>> Regards,
>>
>> Manoj
>>
>>
>>
>> *From:* Billy Liu [mailto:billyliu@apache.org]
>> *Sent:* Thursday, October 12, 2017 1:06 PM
>> *To:* user
>> *Subject:* Re: Kafka Streaming data - Error while building the Cube
>>
>>
>>
>> Hi Kumar,
>>
>>
>>
>> Could you paste more Kafka Consumer related log in kylin.log? And also
>> check from the Kafka broker side, if the Kylin client has connected to
>> Broker.
>>
>>
>>
>> 2017-10-12 14:29 GMT+08:00 Kumar, Manoj H <manoj.h.kumar@jpmorgan.com>:
>>
>> Building the Cube from Kylin UI - Although Messages are there in Kafka
>> topic but Kylin is not able read the offset. Can someone help on this?
>>
>> 2017-10-11 20:50:42,573 INFO  [http-bio-7070-exec-8]
>> utils.AppInfoParser:83 : Kafka version : 0.10.2-kafka-2.2.0
>> 2017-10-11 20:50:42,573 INFO  [http-bio-7070-exec-8]
>> utils.AppInfoParser:84 : Kafka commitId : unknown
>> 2017-10-11 20:50:42,586 DEBUG [http-bio-7070-exec-8]
>> kafka.KafkaSource:107 : The end offsets are {0=0}
>> 2017-10-11 20:50:42,588 ERROR [http-bio-7070-exec-8]
>> controller.CubeController:305 : No new message comes, startOffset =
>> endOffset:0
>> java.lang.IllegalArgumentException: No new message comes, startOffset =
>> endOffset:0
>>         at org.apache.kylin.source.kafka.KafkaSource.enrichSourcePartit
>> ionBeforeBuild(KafkaSource.java:134)
>>         at org.apache.kylin.rest.service.JobService.submitJobInternal(J
>> obService.java:236)
>>         at org.apache.kylin.rest.service.JobService.submitJob(JobServic
>> e.java:208)
>>         at org.apache.kylin.rest.service.JobService$$FastClassBySpringC
>> GLIB$$83a44b2a.invoke(<generated>)
>>
>> Regards,
>> Manoj
>>
>>
>> This message is confidential and subject to terms at:
>> http://www.jpmorgan.com/emaildisclaimer including on confidentiality,
>> legal privilege, viruses and monitoring of electronic messages. If you are
>> not the intended recipient, please delete this message and notify the
>> sender immediately. Any unauthorized use is strictly prohibited.
>>
>>
>>
>> This message is confidential and subject to terms at: http://
>> www.jpmorgan.com/emaildisclaimer including on confidentiality, legal
>> privilege, viruses and monitoring of electronic messages. If you are not
>> the intended recipient, please delete this message and notify the sender
>> immediately. Any unauthorized use is strictly prohibited.
>>
>>
>>
>> This message is confidential and subject to terms at: http://
>> www.jpmorgan.com/emaildisclaimer including on confidentiality, legal
>> privilege, viruses and monitoring of electronic messages. If you are not
>> the intended recipient, please delete this message and notify the sender
>> immediately. Any unauthorized use is strictly prohibited.
>>
>
>

Mime
View raw message