ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jörn Franke <jornfra...@gmail.com>
Subject Re: Spark Data Frame support in Ignite
Date Thu, 03 Aug 2017 06:45:43 GMT
I think the JDBC one is more inefficient, slower requires too much development effort. You
can also check the integration of Alluxio with Spark. 
Then, in general I think JDBC has never designed for large data volumes. It is for executing
queries and getting a small or aggregated result set back. Alternatively for inserting / updating
single rows. 

> On 3. Aug 2017, at 08:17, Dmitriy Setrakyan <dsetrakyan@apache.org> wrote:
> 
> Jorn, thanks for your feedback!
> 
> Can you explain how the direct support would be different from the JDBC
> support?
> 
> Thanks,
> D.
> 
>> On Thu, Aug 3, 2017 at 7:40 AM, Jörn Franke <jornfranke@gmail.com> wrote:
>> 
>> These are two different things. Spark applications themselves do not use
>> JDBC - it is more for non-spark applications to access Spark DataFrames.
>> 
>> A direct support by Ignite would make more sense. Although you have in
>> theory IGFS, if the user is using HDFS, which might not be the case. It is
>> now also very common to use Object stores, such as S3.
>> Direct support could be leverage for interactive analysis or different
>> Spark applications sharing data.
>> 
>>> On 3. Aug 2017, at 05:12, Dmitriy Setrakyan <dsetrakyan@apache.org>
>> wrote:
>>> 
>>> Igniters,
>>> 
>>> We have had the integration with Spark Data Frames on our roadmap for a
>>> while:
>>> https://issues.apache.org/jira/browse/IGNITE-3084
>>> 
>>> However, while browsing Spark documentation, I cam across the generic
>> JDBC
>>> data frame support in Spark:
>>> https://spark.apache.org/docs/latest/sql-programming-guide.
>> html#jdbc-to-other-databases
>>> 
>>> Given that Ignite has a JDBC driver, does it mean that it transitively
>> also
>>> supports Spark data frames? If yes, we should document it.
>>> 
>>> D.
>> 

Mime
View raw message