phoenix-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (PHOENIX-5059) Use the Datasource v2 api in the spark connector
Date Mon, 07 Jan 2019 19:00:00 GMT

    [ https://issues.apache.org/jira/browse/PHOENIX-5059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16736183#comment-16736183
] 

ASF GitHub Bot commented on PHOENIX-5059:
-----------------------------------------

Github user twdsilva commented on the issue:

    https://github.com/apache/phoenix/pull/423
  
    @jmahonin  Thanks for the review. The Datasource V2 API is marked evolving and the API
changed from Spark 2.3 to 2.4 and it will change again in Spark 3.0. The DataSource v2 connector
in this patch will only work with Spark 2.4. 
    I deprecated but did not remove the current connector that uses the V1 API, so users should
still be able to use the current connector. 
    There is a JIRA to move all the connectors to their own repo (see [PHOENIX-5062](https://issues.apache.org/jira/browse/PHOENIX-5062))
which will allow us to release the connectors independently of phoenix core. I was planning
on working on that after this PR goes in. 
    
    For 2) are you referring to ```PhoenixRecordWritable```? That class is deprecated and
only used in the old connector. 
    Spark has code to convert from/to a ResultSet to/from InternalRow which I used (see ```SparkJdbcUtil```).
The setters methods are accessible, but there is a bug in handling ```ByteType```, so I had
to copy the code. This will be fixed in [SPARK-26499](https://issues.apache.org/jira/browse/SPARK-26499)
after which we can switch to just using Spark's ```JdbcUtils.resultSetToSparkInternalRows```
    The setters methods are not accessible, so I had to create a copy of them (see ```makeSetter```
in ```SparkJdbcUtil```). 



> Use the Datasource v2 api in the spark connector
> ------------------------------------------------
>
>                 Key: PHOENIX-5059
>                 URL: https://issues.apache.org/jira/browse/PHOENIX-5059
>             Project: Phoenix
>          Issue Type: New Feature
>            Reporter: Thomas D'Silva
>            Assignee: Thomas D'Silva
>            Priority: Major
>         Attachments: PHOENIX-5059-wip.patch
>
>
> Switching to the v2 api will enable pushing downing limits/aggregations and more in the
future.
> https://issues.apache.org/jira/browse/SPARK-15689



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message