samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Julian Hyde" <jul...@hydromatic.net>
Subject Re: Review Request 33142: [SAMZA-561] Review in progress
Date Tue, 21 Apr 2015 03:27:35 GMT


> On April 14, 2015, 10:14 p.m., Yi Pan (Data Infrastructure) wrote:
> > samza-sql/src/main/java/org/apache/samza/sql/metadata/Stream.java, line 37
> > <https://reviews.apache.org/r/33142/diff/1/?file=926293#file926293line37>
> >
> >     What's the meaning of this parent?
> 
> Milinda Pathirage wrote:
>     parent identifies the system. For example 'kafka'. And this is similar to db schema.

Milinda, I think you should add doc to clarify the purpose of parent.

It will probably not be a good sign if you find yourself using parent (or even name) a lot.
Calcite relations are kind of like files - and the name and parent are like the inode part
of the file. Just like files, they should behave the same whatever you call them.

The RelOptTable that wraps a Table (or its subclass, Stream) remembers the fully-qualified
name for you.


> On April 14, 2015, 10:14 p.m., Yi Pan (Data Infrastructure) wrote:
> > samza-sql/src/main/java/org/apache/samza/sql/metadata/Stream.java, line 76
> > <https://reviews.apache.org/r/33142/diff/1/?file=926293#file926293line76>
> >
> >     I am not sure whether I fully understand this interface method: from Calcite
code base, it seems that this should return an iterator for the stream? This seems to be a
polling method to get the rows from the stream. However, Samza tasks are not polling the streams
to get the messages. Instead, Samza SystemConsumers delivers (i.e. pushes) the messages from
the physical streams to Samza tasks, and hence, the physical operators. How is this stream()
API used in Calcite? Can we leave it un-implemented?
> 
> Milinda Pathirage wrote:
>     AFAIK, this method is only useful in Calcite. We don't have to implement this. May
be I should change this to throw UnsupportedOperationException.

My intention was that if you had a relation that could act as both a stream and a table (consider
Orders - you can ask it for the history (Table) or future orders (Stream)) - then the catalog
would give you the table, then you call its stream() method to give you the streaming version.

Note how this relates to my previous comment. The Table object you get by calling stream()
is anonymous. It is not stored in the catalog directly.

Anyway. That method is experimental, and still evolving. Don't worry about it too much unless
you see errors. We will revisit it when we want to have streams that can also operate as tables.


- Julian


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33142/#review79941
-----------------------------------------------------------


On April 13, 2015, 9:04 p.m., Yi Pan (Data Infrastructure) wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/33142/
> -----------------------------------------------------------
> 
> (Updated April 13, 2015, 9:04 p.m.)
> 
> 
> Review request for samza and Milinda Pathirage.
> 
> 
> Bugs: SAMZA-561
>     https://issues.apache.org/jira/browse/SAMZA-561
> 
> 
> Repository: samza
> 
> 
> Description
> -------
> 
> [SAMZA-561] Review in progress
> 
> Post Milinda's patch for SAMZA-561 to ease the comment and discussion.
> 
> 
> Diffs
> -----
> 
>   build.gradle 97de3a28f6379e3862eec845da87587b1d4f742e 
>   gradle/dependency-versions.gradle ee6dfc411b7ab90b187df79f109884127953862e 
>   samza-sql/src/main/java/org/apache/samza/sql/Utils.java PRE-CREATION 
>   samza-sql/src/main/java/org/apache/samza/sql/api/operators/spec/OperatorSpec.java PRE-CREATION

>   samza-sql/src/main/java/org/apache/samza/sql/data/IntermediateMessageTuple.java PRE-CREATION

>   samza-sql/src/main/java/org/apache/samza/sql/data/serializers/SqlAvroSerdeFactory.java
PRE-CREATION 
>   samza-sql/src/main/java/org/apache/samza/sql/expressions/Expression.java PRE-CREATION

>   samza-sql/src/main/java/org/apache/samza/sql/expressions/RexToJavaCompiler.java PRE-CREATION

>   samza-sql/src/main/java/org/apache/samza/sql/expressions/RexToJavaUtils.java PRE-CREATION

>   samza-sql/src/main/java/org/apache/samza/sql/metadata/AvroSchemaConverter.java PRE-CREATION

>   samza-sql/src/main/java/org/apache/samza/sql/metadata/RelDataTypeToAvroSchemaConverter.java
PRE-CREATION 
>   samza-sql/src/main/java/org/apache/samza/sql/metadata/Stream.java PRE-CREATION 
>   samza-sql/src/main/java/org/apache/samza/sql/operators/factory/SimpleOperatorFactoryImpl.java
PRE-CREATION 
>   samza-sql/src/main/java/org/apache/samza/sql/operators/factory/SimpleOperatorSpec.java
PRE-CREATION 
>   samza-sql/src/main/java/org/apache/samza/sql/operators/factory/TypeAwareOperatorSpec.java
PRE-CREATION 
>   samza-sql/src/main/java/org/apache/samza/sql/operators/insert/InsertToStreamOp.java
PRE-CREATION 
>   samza-sql/src/main/java/org/apache/samza/sql/operators/insert/InsertToStreamSpec.java
PRE-CREATION 
>   samza-sql/src/main/java/org/apache/samza/sql/operators/project/ProjectOp.java PRE-CREATION

>   samza-sql/src/main/java/org/apache/samza/sql/operators/project/ProjectSpec.java PRE-CREATION

>   samza-sql/src/main/java/org/apache/samza/sql/operators/scan/ProjectableFilterableStreamScanOp.java
PRE-CREATION 
>   samza-sql/src/main/java/org/apache/samza/sql/operators/scan/ProjectableFilterableStreamScanSpec.java
PRE-CREATION 
>   samza-sql/src/main/java/org/apache/samza/sql/operators/scan/StreamScanSpec.java PRE-CREATION

>   samza-sql/src/main/java/org/apache/samza/sql/planner/ExecutionPlanner.java PRE-CREATION

>   samza-sql/src/main/java/org/apache/samza/sql/planner/QueryPlanner.java PRE-CREATION

>   samza-sql/src/main/java/org/apache/samza/sql/planner/rules/FilterableStreamScanRule.java
PRE-CREATION 
>   samza-sql/src/main/java/org/apache/samza/sql/planner/rules/ProjectableStreamScanRule.java
PRE-CREATION 
>   samza-sql/src/main/java/org/apache/samza/sql/planner/rules/RemoveIdentityProjectRule.java
PRE-CREATION 
>   samza-sql/src/main/java/org/apache/samza/sql/rel/ProjectableFilterableStreamScan.java
PRE-CREATION 
>   samza-sql/src/main/java/org/apache/samza/sql/rel/StreamScan.java PRE-CREATION 
>   samza-sql/src/main/java/org/apache/samza/task/sql/StreamSqlTask.java PRE-CREATION 
>   samza-sql/src/test/java/org/apache/samza/sql/data/serializers/SqlAvroSerdeTest.java
PRE-CREATION 
>   samza-sql/src/test/java/org/apache/samza/sql/planner/QueryPlannerTest.java PRE-CREATION

>   samza-sql/src/test/java/org/apache/samza/sql/planner/SamzaStreamTableFactory.java PRE-CREATION

>   samza-sql/src/test/java/org/apache/samza/sql/planner/TestExecutionPlanner.java PRE-CREATION

>   samza-sql/src/test/java/org/apache/samza/sql/planner/TestQueryPlanner.java PRE-CREATION

>   samza-sql/src/test/java/org/apache/samza/sql/planner/TestRexToJavaCompiler.java PRE-CREATION

>   samza-sql/src/test/java/org/apache/samza/sql/test/Constants.java PRE-CREATION 
>   samza-sql/src/test/java/org/apache/samza/sql/test/Utils.java PRE-CREATION 
>   samza-sql/src/test/java/org/apache/samza/sql/test/metadata/TestAvroSchemaConverter.java
PRE-CREATION 
>   samza-sql/src/test/java/org/apache/samza/task/sql/RandomOperatorTask.java PRE-CREATION

>   samza-sql/src/test/java/org/apache/samza/task/sql/StreamSqlTask.java PRE-CREATION 
>   samza-sql/src/test/resources/orders.avsc PRE-CREATION 
>   samza-sql/src/test/resources/orders.json PRE-CREATION 
>   samza-test/src/main/config/sql-filter.properties PRE-CREATION 
>   samza-test/src/main/java/org/apache/samza/test/integration/sql/OrdersStreamFactory.java
PRE-CREATION 
>   samza-test/src/main/java/org/apache/samza/test/integration/sql/SqlTask.java PRE-CREATION

>   samza-test/src/main/python/integration_tests.py df64e239a2e467c8e4429dbeb7039f1aa9965ecc

>   samza-test/src/main/python/requirements.txt 2ae95908248516b5b26e671f24fa680f7b801675

>   samza-test/src/main/python/samza_job_yarn_deployer.py 38635ca5899c43fb61d6b4042e8543f0508fd41b

>   samza-test/src/main/python/tests/sql_tests.py PRE-CREATION 
>   samza-test/src/main/resources/orders.avsc PRE-CREATION 
>   samza-test/src/main/resources/orders.json PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/33142/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Yi Pan (Data Infrastructure)
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message