apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ilya Ganelin (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (APEXMALHAR-1818) Integrate Calcite to support SQL
Date Tue, 10 May 2016 20:41:12 GMT

    [ https://issues.apache.org/jira/browse/APEXMALHAR-1818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15278850#comment-15278850
] 

Ilya Ganelin edited comment on APEXMALHAR-1818 at 5/10/16 8:40 PM:
-------------------------------------------------------------------

Based on our initial discussion, there are a number of key steps in unifying Calcite with
Apex

*SQL Translation*
First, we need a method to translate a SQL query representation into an Apex DAG.

1. SQL will be translated by Calcite into a relational algebra representation (e.g. JSON)
2. Malhar translates provided schema definitions (either dynamically communicated or statically
specified) into appropriately structured/typed input/output operators in the DAG and output
as a JSON config
3. Malhar translates the algebraic representation into an Apex/Malhar DAG representation output
(JSON). This DAG can be considered the “internal” DAG between the input and output operators
defined by the schema

*Testing*
The Calcite project is presently defining a new testing framework which will allow for offline
out-of-band testing of frameworks that work with Calcite. Testing wth the above translation
process is straightforward:

1. Compare the JSON relational algebra representation against the expected representation
2. Compare the JSON generated from the schema translation against the known good translation
3. Compare the JSON generated by the SQL to DAG translation against the expected DAG JSON
representation. 

The above approach should allow us to create a well modularized integration that can be readily
tested and supports dynamic and flexible reconfiguration. 

[~julianhyde] [~thw] Please review and let me know if I left anything out.


was (Author: ilganeli):
Based on our initial discussion, there are a number of key steps in unifying Calcite with
Apex

**SQL Translation**
First, we need a method to translate a SQL query representation into an Apex DAG.

1. SQL will be translated by Calcite into a relational algebra representation (e.g. JSON)
2. Malhar translates provided schema definitions (either dynamically communicated or statically
specified) into appropriately structured/typed input/output operators in the DAG and output
as a JSON config
3. Malhar translates the algebraic representation into an Apex/Malhar DAG representation output
(JSON). This DAG can be considered the “internal” DAG between the input and output operators
defined by the schema

**Testing**
The Calcite project is presently defining a new testing framework which will allow for offline
out-of-band testing of frameworks that work with Calcite. Testing wth the above translation
process is straightforward:

1. Compare the JSON relational algebra representation against the expected representation
2. Compare the JSON generated from the schema translation against the known good translation
3. Compare the JSON generated by the SQL to DAG translation against the expected DAG JSON
representation. 

The above approach should allow us to create a well modularized integration that can be readily
tested and supports dynamic and flexible reconfiguration. 

[~julianhyde] [~thw] Please review and let me know if I left anything out.

> Integrate Calcite to support SQL
> --------------------------------
>
>                 Key: APEXMALHAR-1818
>                 URL: https://issues.apache.org/jira/browse/APEXMALHAR-1818
>             Project: Apache Apex Malhar
>          Issue Type: New Feature
>          Components: query operators
>            Reporter: Amol
>            Assignee: Amol
>              Labels: roadmap
>
> Once we have ability to generate a subdag, we should take a look at integrating Calcite
into Apex. The operator that enables populate DAG, should use Calcite to generate the DAG,
given a SQL query.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message