ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vladimir Ozerov (JIRA)" <j...@apache.org>
Subject [jira] [Created] (IGNITE-6917) SQL: implement COP command for efficient data loading
Date Wed, 15 Nov 2017 09:14:00 GMT
Vladimir Ozerov created IGNITE-6917:

             Summary: SQL: implement COP command for efficient data loading
                 Key: IGNITE-6917
                 URL: https://issues.apache.org/jira/browse/IGNITE-6917
             Project: Ignite
          Issue Type: Task
      Security Level: Public (Viewable by anyone)
          Components: sql
            Reporter: Vladimir Ozerov
             Fix For: 2.4

Inspired by Postgres [1]

Common use case - bulk data load through JDBC/ODBC interface. Currently it is only possible
to execute single commands one by one. We already can batch them to improve performance, but
there is still big room for improvement.

We should think of a completely new command - {{COPY}}. It will accept a file (or input stream
in general case) on the client side, then transfer data to the cluster, and then execute update
inside the cluster, e.g. through streamer. 

First of all we need to create quick and dirty prototype to assess potential performance improvement.
It speedup is confirmed, we should build base implementation which will accept only files.
But at the same time we should understand how it will evolve in future: multiple file formats
(probably including Hadoop formarts, e.g. Parquet), escape characters, input streams, etc..

[1] https://www.postgresql.org/docs/9.6/static/sql-copy.html

This message was sent by Atlassian JIRA

View raw message