kylin-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Xiaoxiang Yu <>
Subject Re: 答复: kafka构建 重置offset
Date Tue, 27 Aug 2019 06:19:21 GMT
Dear Katte:
  I think you are right. The offset rewrite logic is located at method enrichSourcePartitionBeforeBuild
of KafkaSource.

,I find that if sourceOffsetStart is set to 0, Kylin will try to find the latest/newest segment
partition offset for your next build.

,I find that if your sourceOffsetStart is set to negative number, Kylin will use the earliest
partition offset.

I think the log of will give your detailed message if kylin’s logger is
enabled at Debug Level.

Best wishes,
Xiaoxiang Yu

发件人: Katte <>
答复: "" <>
日期: 2019年8月26日 星期一 14:44
收件人: Xiaoxiang Yu <>, "" <>
主题: 答复: kafka构建 重置offset


我发现通过下面命令可以进行构建了,当我指定"sourceOffsetStart": -1,kylin会从offset:0的位置开始消费;如果指定"sourceOffsetStart":

curl -X PUT --user ADMIN:KYLIN -H "Content-Type: application/json;charset=utf-8" -d '{ "sourceOffsetStart":
-1, "sourceOffsetEnd": 9223372036854775807, "buildType": "BUILD"}' http://localhost:7070/kylin/api/cubes/CUBE_T2_0_4/build2

发送自 Windows 10 版邮件<>应用

发件人: Xiaoxiang Yu<>
发送时间: 2019年8月24日 18:56
主题: Re: kafka构建 重置offset

Dear Katte:
    As far as I can see, I think Kylin has the ability of resume the consumption in the right
position(right behind the latest segment's offset), even after you restart your kafka and
    I guess you may need to check the state of your kafka topic, you may use tools provided
at $KAFKA_HOME/bin to check current offset of your topic partitions, and make sure you can
consume the latest message by $KAFKA_HOME/bin/ . If you find you
Kafka is in right state and the error is caused by Kylin, please provide more detail like
kylin version, exception trace in kylin.log and your metadata for further analysis.

Best wishes,
Xiaoxiang Yu

发件人: Katte <>
答复: "" <>
日期: 2019年8月23日 星期五 19:31
收件人: "" <>
主题: kafka构建 重置offset


我通过kafka构建了一个 Cube,构建成功后最后的offset位置是10050.
之后,我再次build cube时,会提示说当前offset是10050,不能构建。

因为实际情况,我们有可能不定期 重的启kylin与kafka。

发送自 Windows 10 版邮件<>应用

View raw message