spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Asher Krim <ak...@hubspot.com>
Subject Re: Possible bug - Java iterator/iterable inconsistency
Date Thu, 19 Jan 2017 19:32:43 GMT
Thanks Sean!

On Thu, Jan 19, 2017 at 6:09 AM, Sean Owen <sowen@cloudera.com> wrote:

> Yes, confirmed that fixing it unfortunately causes trouble in Java 8. See
> https://issues.apache.org/jira/browse/SPARK-19287 for further discussion.
>
> On Wed, Jan 18, 2017 at 9:00 PM Sean Owen <sowen@cloudera.com> wrote:
>
>> Hm. Unless I am also totally missing or forgetting something, I think
>> you're right. The equivalent in PairRDDFunctions.scala operations on a
>> function from T to TraversableOnce[U] and a TraversableOnce is most like
>> java.util.Iterator.
>>
>> You can work around it by wrapping it in a faked IteratorIterable.
>>
>> I think this is fixable in the API by deprecating this method and adding
>> a new one that takes a FlatMapFunction. We'd have to triple-check in a test
>> that this doesn't cause an API compatibility problem with respect to Java 8
>> lambdas, but if that's settled, I think this could be fixed without
>> breaking the API.
>>
>> On Wed, Jan 18, 2017 at 8:50 PM Asher Krim <akrim@hubspot.com> wrote:
>>
>> In Spark 2 + Java + RDD api, the use of iterables was replaced with
>> iterators. I just encountered an inconsistency in `flatMapValues` that may
>> be a bug:
>>
>> `flatMapValues` (https://github.com/apache/spark/blob/master/core/src/
>> main/scala/org/apache/spark/api/java/JavaPairRDD.scala#L677) takes
>> a `FlatMapFunction` (https://github.com/apache/spark/blob/
>> 39e2bad6a866d27c3ca594d15e574a1da3ee84cc/core/src/main/java/
>> org/apache/spark/api/java/function/FlatMapFunction.java)
>>
>> The problem is that `FlatMapFunction` was changed to return an iterator,
>> but `rdd.flatMapValues` still expects an iterable. Am I using these
>> constructs correctly? Is there a workaround other than converting the
>> iterator to an iterable outside of the function?
>>
>> Thanks,
>> --
>> Asher Krim
>> Senior Software Engineer
>>
>>


-- 
Asher Krim
Senior Software Engineer

Mime
View raw message