calcite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Julian Hyde <jh...@apache.org>
Subject Re: streaming support for infinite streams
Date Tue, 21 Jul 2015 19:50:21 GMT
Ah yes. You're hitting the interpreter "cheap and dirty"
implementation of TableScan. I made the interpreter the simplest thing
that could possibly work, so I made every operator build a list. (I
know, I know. Enumerable uses iterators, and other implementations do
even better. But I wanted to fit it into one page of code.)

Can you log a jira case please?

The solution will be either to fix the interpreter to use iterators
(or similar) rather than lists, or to recognize that a query is
infinite and not use the interpreter.

Julian


On Tue, Jul 21, 2015 at 8:58 AM, Jesse Yates <jesse.k.yates@gmail.com> wrote:
> Hi,
>
> I've only been using Calcite for a short while and am trying to hook up my
> own streaming table. The problem I'm running into seems be, at its core, an
> impedance mismatch.
>
> Streams, by their very nature, are expected to be infinite (this what
> Julian is getting at in the stream tutorial
> <http://calcite.incubator.apache.org/docs/stream.html>). As such, they
> should send incremental results to along to the ResultSet.
>
> However, when running a simple query (e.g. select stream * from orders) and
> the StreamableTable returning a Enumerable<Object[]> from an infinite
> stream using the standard Linq4j tools, you end up quickly trying to store
> all the values of the stream in a ListSink in a TableScanNode.
> <https://github.com/apache/incubator-calcite/blob/3e50232b681e8dadb921580ee6f3e0376dd0f664/core/src/main/java/org/apache/calcite/interpreter/TableScanNode.java#L73>
>
> It seems like the TableScanNode needs to be made stream aware and the sink
> needs to forward results onward.
>
> Here is a simple test
> <https://github.com/jyates/incubator-calcite/commit/7111a0ed8456a567bd846660485bf509d77614fc>
> that verifies an infinite stream infinitely adds data to the ListSink (drop
> a breakpoint in TableScanNode to see it in action).
>
> As I mentioned, I'm still pretty new to Calcite, so any pointers to my
> being completely wrong would be much appreciated (or I'm happy to follow up
> in a JIRA if this looks like a bug).
>
> Thanks,
> Jesse Yates

Mime
View raw message