spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Takeshi Yamamuro (JIRA)" <>
Subject [jira] [Commented] (SPARK-15816) SQL server based on Postgres protocol
Date Fri, 19 Aug 2016 08:56:21 GMT


Takeshi Yamamuro commented on SPARK-15816:

About 1.
Yea, we can support them in there; nested structs and maps are handled as json-fomatted strings.
Array formats also are handled as strings like '{x,x,x}'.
See tests for struct/map:

About 2.
ISTM it is not impossible though, what's the purpose to support a https mode?

> SQL server based on Postgres protocol
> -------------------------------------
>                 Key: SPARK-15816
>                 URL:
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>            Reporter: Reynold Xin
>         Attachments: New_SQL_Server_for_Spark.pdf
> At Spark Summit today this idea came up from a discussion: it would be great to investigate
the possibility of implementing a new SQL server using Postgres' protocol, in lieu of Hive
ThriftServer 2. I'm creating this ticket to track this idea, in case others have feedback.
> This server can have a simpler architecture, and allows users to leverage a wide range
of tools that are already available for Postgres (and many commercial database systems based
on Postgres).
> Some of the problems we'd need to figure out are:
> 1. What is the Postgres protocol? Is there an official documentation for it?
> 2. How difficult would it be to implement that protocol in Spark (JVM in particular).
> 3. How does data type mapping work?
> 4. How does system commands work? Would Spark need to support all of Postgres' commands?
> 5. Any restrictions in supporting nested data?

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message