spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From koeninger <>
Subject [GitHub] spark pull request: [SPARK-4964] [Streaming] Exactly-once semantic...
Date Tue, 03 Feb 2015 23:19:34 GMT
Github user koeninger commented on the pull request:
    Just make the simplified createRDD return a static type of RDD[(K, V)],
    that's what I'm saying.
    You're already going to have to deal with those other type parameters in
    java if you want to call a more complex version of createRDD, because you
    have to know about the serializers and message handler.  Make the more
    complex version return KafkaRDD.
    I agree that casting to KafkaRDD is better than casting to
    HasOffsetRanges.  The abstract class is an unnecessary complication though.
    On Tue, Feb 3, 2015 at 5:11 PM, Tathagata Das <>
    > I spent some time talking to Patrick offline about this. If we expose the
    > KafkaRDD as is (while keeping its constructor private), then the simplified
    > createRDD would be
    > KafkaUtils.createRDD[K, V, KD, VD](....): KafkaRDD[K, V, KD, VD, (K, V)]
    > Imagine how one would use it in Java.
    > KafkaRDD<String, String, StringDecoder, StringDecode, Product2<String, String>>
rdd = KafkaUtils.createRDD(....)
    > That's not very Java friendly if you ask a Java developer. And we a huge
    > fraction of the community as Java developers. Furthemore, we want to add
    > Python API as well, and that also requires the interfaces to be
    > Java-friendly. Here is the alternative (I think) with what I proposed.
    > KafkaRDD<String, String> rdd = KafkaUtils.createRDD(....)
    > Much simpler.
    > Regarding casting, there are two cases,
    > 1. casting RDD generated from createRDD - If we take off HasOffsetRanges (
    > KafkaUtils.createRDD returns only RDD), then user have to cast. But if we
    > return abstract class KafkaRDD, then no casting necessary.
    > 2. casting RDD in DStream.foreachRDD - Casting is necessary either ways.
    > But isnt it more intuitive to write rdd.asInstanceOf[KafkaRDD] than
    > rdd.asInstanceOf[HasOffsetRanges]?
    > —
    > Reply to this email directly or view it on GitHub
    > <>.

If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at or file a JIRA ticket
with INFRA.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message