hudi-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] [incubator-hudi] amitsingh-10 commented on issue #1335: [SUPPORT] HoodieDeltaStreamer Kafka offset reset config not being read
Date Fri, 14 Feb 2020 11:07:20 GMT
amitsingh-10 commented on issue #1335: [SUPPORT] HoodieDeltaStreamer Kafka offset reset config
not being read
URL: https://github.com/apache/incubator-hudi/issues/1335#issuecomment-586213714
 
 
   Thanks for the reply @lamber-ken. I tried working with `0.5.0-incubating` however I am
getting the following error now :
   ```
   User class threw exception: org.apache.spark.SparkException: Offsets not available on leader
   ```
   
   The stacktrace for the same is :
   ```
   User class threw exception: org.apache.spark.SparkException: Offsets not available on leader:
OffsetRange(topic: 'test-topic', partition: 0, range: [0 -> 6667]),OffsetRange(topic: 'test-topic',
partition: 1, range: [0 -> 6667]),OffsetRange(topic: 'test-topic', partition: 2, range:
[0 -> 6666])
   org.apache.spark.SparkException: Offsets not available on leader: OffsetRange(topic: 'test-topic',
partition: 0, range: [0 -> 6667]),OffsetRange(topic: 'test-topic', partition: 1, range:
[0 -> 6667]),OffsetRange(topic: 'test-topic', partition: 2, range: [0 -> 6666])
   	at org.apache.spark.streaming.kafka.KafkaUtils$.org$apache$spark$streaming$kafka$KafkaUtils$$checkOffsets(KafkaUtils.scala:200)
   	at org.apache.spark.streaming.kafka.KafkaUtils$$anonfun$createRDD$1.apply(KafkaUtils.scala:253)
   	at org.apache.spark.streaming.kafka.KafkaUtils$$anonfun$createRDD$1.apply(KafkaUtils.scala:249)
   	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
   	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
   	at org.apache.spark.SparkContext.withScope(SparkContext.scala:699)
   	at org.apache.spark.streaming.kafka.KafkaUtils$.createRDD(KafkaUtils.scala:249)
   	at org.apache.spark.streaming.kafka.KafkaUtils$$anonfun$createRDD$3.apply(KafkaUtils.scala:338)
   	at org.apache.spark.streaming.kafka.KafkaUtils$$anonfun$createRDD$3.apply(KafkaUtils.scala:333)
   	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
   	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
   	at org.apache.spark.SparkContext.withScope(SparkContext.scala:699)
   	at org.apache.spark.streaming.kafka.KafkaUtils$.createRDD(KafkaUtils.scala:333)
   	at org.apache.spark.streaming.kafka.KafkaUtils.createRDD(KafkaUtils.scala)
   	at org.apache.hudi.utilities.sources.AvroKafkaSource.toRDD(AvroKafkaSource.java:67)
   	at org.apache.hudi.utilities.sources.AvroKafkaSource.fetchNewData(AvroKafkaSource.java:61)
   	at org.apache.hudi.utilities.sources.Source.fetchNext(Source.java:71)
   	at org.apache.hudi.utilities.deltastreamer.SourceFormatAdapter.fetchNewDataInAvroFormat(SourceFormatAdapter.java:61)
   	at org.apache.hudi.utilities.deltastreamer.DeltaSync.readFromSource(DeltaSync.java:292)
   	at org.apache.hudi.utilities.deltastreamer.DeltaSync.syncOnce(DeltaSync.java:214)
   	at org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.sync(HoodieDeltaStreamer.java:120)
   	at org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.main(HoodieDeltaStreamer.java:292)
   	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   	at java.lang.reflect.Method.invoke(Method.java:498)
   	at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:678)
   ```
   
   This was fixed by following [pull request](https://github.com/apache/incubator-hudi/pull/650)
and is available in the `0.5.0-incubating` version of Hudi. Correct me, if I am wrong. I have
changed the `auto.offset.reset=smallest` in the properties if this is relevant.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

Mime
View raw message