ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stephen Darlington <stephen.darling...@gridgain.com>
Subject Re: Write python dataframe to ignite table.
Date Tue, 29 Oct 2019 13:12:31 GMT
Ah, sorry. When I see “DataFrame” I immediately think of Spark.

While that would be a great addition, I don’t think the Python thin-client currently supports
DataFrames. I think your two options are to use Spark DataFrames or loop through, using put/putAll
to import that data. Three options if you include creating a patch for the thin-client ;)

Regards,
Stephen

> On 29 Oct 2019, at 10:04, Muhammed Favas <favas.muhammed@expeedsoftware.com> wrote:
> 
> Hi,
>  
> I have tried simple python program without using spark. First I read whole csv into python
dataframe using pandas library.
> Now I want to bulk insert the whole dataframe into ignite table without looping through.
>  
> The purpose of this test is to evaluate the best way(means faster wat) to bulk load csv
files into ignite.
>  
> Ignite COPY command I can not use here, because I need an option to do column mapping
while import csv files.
>  
>  
> Regards,
> Favas 
>  
> From: Stephen Darlington <stephen.darlington@gridgain.com> 
> Sent: Monday, October 28, 2019 5:05 PM
> To: user@ignite.apache.org
> Subject: Re: Write python dataframe to ignite table.
>  
> What have you tried? As long as your command-line includes the right JAR files it seems
to more-or-less just work for me:
>  
> https://medium.com/@sdarlington/the-trick-to-successfully-integrating-apache-ignite-and-pyspark-890e436d09ba
<https://medium.com/@sdarlington/the-trick-to-successfully-integrating-apache-ignite-and-pyspark-890e436d09ba>
>  
> Regards,
> Stephen
> 
> 
> On 22 Oct 2019, at 11:41, Muhammed Favas <favas.muhammed@expeedsoftware.com <mailto:favas.muhammed@expeedsoftware.com>>
wrote:
>  
> Hi,
>  
> Is there a way to bulk load python dataframe values to ignite table?
>  
> Regards,
> Favas 



Mime
View raw message