flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Deepak Sharma <deepakmc...@gmail.com>
Subject Re: SQL for Flink
Date Wed, 14 Sep 2016 13:07:42 GMT
Thanks Greg .
I will start picking some of them.

Thanks
Deepak

On 14 Sep 2016 6:31 pm, "Greg Hogan" <code@greghogan.com> wrote:

> Hi Deepak,
>
> There are many open tickets for Flink's SQL API. Documentation is at
> https://ci.apache.org/projects/flink/flink-docs-master/dev/table_api.html.
>
> https://issues.apache.org/jira/issues/?jql=project%20%
> 3D%20FLINK%20AND%20resolution%20%3D%20Unresolved%20AND%
> 20component%20%3D%20%22Table%20API%20%26%20SQL%22%20ORDER%
> 20BY%20priority%20DESC
>
> Greg
>
> On Wed, Sep 14, 2016 at 12:27 PM, Deepak Sharma <deepakmca05@gmail.com>
> wrote:
>
>> +1
>> Yes.I agree to having SQL for Flink.
>> I can take up some tasks as well once this starts.
>>
>> Thanks
>> Deepak
>>
>> On Wed, Sep 14, 2016 at 3:47 PM, Radu Tudoran <radu.tudoran@huawei.com>
>> wrote:
>>
>>> Hi,
>>>
>>>
>>>
>>> As a follow up to multiple discussions that happened during Flink
>>> Forward about how SQL should be supported by Flink, I was thinking to make
>>> a couple of proposals.
>>>
>>> Disclaimer: I do not claim I have managed to synthesized all the
>>> discussions and probably a great deal of things are still missing
>>>
>>>
>>>
>>> *Why supporting SQL for Flink?*
>>>
>>> -          A goal to support SQL for Flink should be to enable larger
>>> adoption of Flink – particularly for data scientists / data engineers who
>>> might not want/know how to program against the existing APIs
>>>
>>> -          The main implication as I see from this is that SQL should
>>> serve as a translation tool of the data processing processing flow to a
>>> stream topology that will be executed by Flink
>>>
>>> -          This would require to support rather soon an SQL client for
>>> Flink
>>>
>>>
>>>
>>> *How many features should be supported?*
>>>
>>> -          In order to enable a (close to ) full benefit of the
>>> processing capabilities of Flink, I believe most of the processing types
>>> should be supported – this includes all different types of windows,
>>> aggregations, transformations, joins….
>>>
>>> -          I would propose that UDFs should also be supported such that
>>> one can easily add more complex computation if needed
>>>
>>> -          In the spirit of the extensibility that Flink supports for
>>> the operators, functions… such custom operators should be supported to
>>> replace the default implementations of the SQL logical operators
>>>
>>>
>>>
>>> *How much customization should be enabled?*
>>>
>>> -          Regarding customization this could be provided by
>>> configuration files. Such a configuration can cover the policies for how
>>> the triggers, evictors, parallelization …  will be done for the specific
>>> translation of the SQL query into Flink code
>>>
>>> -          In order to support the integration of custom operators for
>>> specific SQL logical operators, the users should be enabled also to provide
>>> translation RULES that will replace the default ones  (e.g. if a user want
>>> to define their own CUSTOM_TABLE_SCAN, it should be able to provide
>>> something like configuration.replaceRule(DataStreamScanRule.INSTANCE ,
>>> CUSTOM_TABLE_SCAN_Rule.INSTANCE) – or if the selection of the new
>>> translation rule can be handled from the cost than simply
>>> configuration.addRule( CUSTOM_TABLE_SCAN_Rule.INSTANCE)
>>>
>>>
>>>
>>> What do you think?
>>>
>>>
>>>
>>>
>>>
>>> Dr. Radu Tudoran
>>>
>>> Senior Research Engineer - Big Data Expert
>>>
>>> IT R&D Division
>>>
>>>
>>>
>>> [image: cid:image007.jpg@01CD52EB.AD060EE0]
>>>
>>> HUAWEI TECHNOLOGIES Duesseldorf GmbH
>>>
>>> European Research Center
>>>
>>> Riesstrasse 25, 80992 München
>>>
>>>
>>>
>>> E-mail: *radu.tudoran@huawei.com <radu.tudoran@huawei.com>*
>>>
>>> Mobile: +49 15209084330
>>>
>>> Telephone: +49 891588344173
>>>
>>>
>>>
>>> HUAWEI TECHNOLOGIES Duesseldorf GmbH
>>> Hansaallee 205, 40549 Düsseldorf, Germany, www.huawei.com
>>> Registered Office: Düsseldorf, Register Court Düsseldorf, HRB 56063,
>>> Managing Director: Bo PENG, Wanzhou MENG, Lifang CHEN
>>> Sitz der Gesellschaft: Düsseldorf, Amtsgericht Düsseldorf, HRB 56063,
>>> Geschäftsführer: Bo PENG, Wanzhou MENG, Lifang CHEN
>>>
>>> This e-mail and its attachments contain confidential information from
>>> HUAWEI, which is intended only for the person or entity whose address is
>>> listed above. Any use of the information contained herein in any way
>>> (including, but not limited to, total or partial disclosure, reproduction,
>>> or dissemination) by persons other than the intended recipient(s) is
>>> prohibited. If you receive this e-mail in error, please notify the sender
>>> by phone or email immediately and delete it!
>>>
>>>
>>>
>>
>>
>>
>> --
>> Thanks
>> Deepak
>> www.bigdatabig.com
>> www.keosha.net
>>
>
>

Mime
View raw message