calcite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jesse Yates <jesse.k.ya...@gmail.com>
Subject streaming support for infinite streams
Date Tue, 21 Jul 2015 15:58:31 GMT
Hi,

I've only been using Calcite for a short while and am trying to hook up my
own streaming table. The problem I'm running into seems be, at its core, an
impedance mismatch.

Streams, by their very nature, are expected to be infinite (this what
Julian is getting at in the stream tutorial
<http://calcite.incubator.apache.org/docs/stream.html>). As such, they
should send incremental results to along to the ResultSet.

However, when running a simple query (e.g. select stream * from orders) and
the StreamableTable returning a Enumerable<Object[]> from an infinite
stream using the standard Linq4j tools, you end up quickly trying to store
all the values of the stream in a ListSink in a TableScanNode.
<https://github.com/apache/incubator-calcite/blob/3e50232b681e8dadb921580ee6f3e0376dd0f664/core/src/main/java/org/apache/calcite/interpreter/TableScanNode.java#L73>

It seems like the TableScanNode needs to be made stream aware and the sink
needs to forward results onward.

Here is a simple test
<https://github.com/jyates/incubator-calcite/commit/7111a0ed8456a567bd846660485bf509d77614fc>
that verifies an infinite stream infinitely adds data to the ListSink (drop
a breakpoint in TableScanNode to see it in action).

As I mentioned, I'm still pretty new to Calcite, so any pointers to my
being completely wrong would be much appreciated (or I'm happy to follow up
in a JIRA if this looks like a bug).

Thanks,
Jesse Yates

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message