ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Valentin Kulichenko <valentin.kuliche...@gmail.com>
Subject Re: Spark data frames integration merged
Date Wed, 03 Jan 2018 23:35:53 GMT
Revin,

I doubt IgniteRDD#getPrefferredLocations has any affect on data frames, but
this is an interesting point. Nikolay, as a developer of this
functionality, can you please comment on this?

-Val

On Wed, Jan 3, 2018 at 1:22 PM, Revin Chalil <rchalil@expedia.com> wrote:

> Thanks Val for the info on indexes with DF. Do you know if adding index /
> affinitykeys on the cache help with the join, when the IgniteRDD is joined
> with a spark DF? The below from docs say that
>
> “IgniteRDD also provides affinity information to Spark via
> getPrefferredLocations method so that RDD computations use data locality.”
>
> I was wondering, if the affinitykey on the cache can be utilized in the
> spark join?
>
>
> On 1/3/18, 12:27 PM, "vkulichenko" <valentin.kulichenko@gmail.com> wrote:
>
>     Indexes would not be used during joins, at least in current
> implementation.
>     Current integration is implemented as a regular Spark data source which
>     provides each relation separately. Spark then performs join by itself,
> so
>     Ignite indexes do not help.
>
>     The easiest way to get binaries would be to use a nightly build [1] ,
> but it
>     seems to be broken for some reason (latest is from May 31). I guess
> the only
>     option at the moment is to build from source.
>
>     [1]
>     https://builds.apache.org/view/H-L/view/Ignite/job/Ignite-nightly/
> lastSuccessfulBuild/
>
>     -Val
>
>
>
>     --
>     Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message