calcite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Julian Hyde <>
Subject Re: streaming support for infinite streams
Date Tue, 21 Jul 2015 19:50:21 GMT
Ah yes. You're hitting the interpreter "cheap and dirty"
implementation of TableScan. I made the interpreter the simplest thing
that could possibly work, so I made every operator build a list. (I
know, I know. Enumerable uses iterators, and other implementations do
even better. But I wanted to fit it into one page of code.)

Can you log a jira case please?

The solution will be either to fix the interpreter to use iterators
(or similar) rather than lists, or to recognize that a query is
infinite and not use the interpreter.


On Tue, Jul 21, 2015 at 8:58 AM, Jesse Yates <> wrote:
> Hi,
> I've only been using Calcite for a short while and am trying to hook up my
> own streaming table. The problem I'm running into seems be, at its core, an
> impedance mismatch.
> Streams, by their very nature, are expected to be infinite (this what
> Julian is getting at in the stream tutorial
> <>). As such, they
> should send incremental results to along to the ResultSet.
> However, when running a simple query (e.g. select stream * from orders) and
> the StreamableTable returning a Enumerable<Object[]> from an infinite
> stream using the standard Linq4j tools, you end up quickly trying to store
> all the values of the stream in a ListSink in a TableScanNode.
> <>
> It seems like the TableScanNode needs to be made stream aware and the sink
> needs to forward results onward.
> Here is a simple test
> <>
> that verifies an infinite stream infinitely adds data to the ListSink (drop
> a breakpoint in TableScanNode to see it in action).
> As I mentioned, I'm still pretty new to Calcite, so any pointers to my
> being completely wrong would be much appreciated (or I'm happy to follow up
> in a JIRA if this looks like a bug).
> Thanks,
> Jesse Yates

View raw message