beam-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Reuven Lax <re...@google.com>
Subject Re: [DISCUSS] Portability representation of schemas
Date Sat, 01 Jun 2019 17:44:05 GMT
Keep in mind that right now the SchemaRegistry is only assumed to exist at
graph-construction time, not at execution time; all information in the
schema registry is embedded in the SchemaCoder, which is the only thing we
keep around when the pipeline is actually running. We could look into
changing this, but it would potentially be a very big change, and I do
think we should start getting users actively using schemas soon.

On Fri, May 31, 2019 at 3:40 PM Brian Hulette <bhulette@google.com> wrote:

> > Can you propose what the protos would look like in this case? Right now
> LogicalType does not contain the to/from conversion functions in the proto.
> Do you think we'll need to add these in?
>
> Maybe. Right now the proposed LogicalType message is pretty simple/generic:
> message LogicalType {
>   FieldType representation = 1;
>   string logical_urn = 2;
>   bytes logical_payload = 3;
> }
>
> If we keep just logical_urn and logical_payload, the logical_payload could
> itself be a protobuf with attributes of 1) a serialized class and 2/3)
> to/from functions. Or, alternatively, we could have a generalization of the
> SchemaRegistry for logical types. Implementations for standard types and
> user-defined types would be registered by URN, and the SDK could look them
> up given just a URN. I put a brief section about this alternative in the
> doc last week [1]. What I suggested there included removing the
> logical_payload field, which is probably overkill. The critical piece is
> just relying on a registry in the SDK to look up types and to/from
> functions rather than storing them in the portable schema itself.
>
> I kind of like keeping the LogicalType message generic for now, since it
> gives us a way to try out these various approaches, but maybe that's just a
> cop out.
>
> [1]
> https://docs.google.com/document/d/1uu9pJktzT_O3DxGd1-Q2op4nRk4HekIZbzi-0oTAips/edit?ts=5cdf6a5b#heading=h.jlt5hdrolfy
>
> On Fri, May 31, 2019 at 12:36 PM Reuven Lax <relax@google.com> wrote:
>
>>
>>
>> On Tue, May 28, 2019 at 10:11 AM Brian Hulette <bhulette@google.com>
>> wrote:
>>
>>>
>>>
>>> On Sun, May 26, 2019 at 1:25 PM Reuven Lax <relax@google.com> wrote:
>>>
>>>>
>>>>
>>>> On Fri, May 24, 2019 at 11:42 AM Brian Hulette <bhulette@google.com>
>>>> wrote:
>>>>
>>>>> *tl;dr:* SchemaCoder represents a logical type with a base type of
>>>>> Row and we should think about that.
>>>>>
>>>>> I'm a little concerned that the current proposals for a portable
>>>>> representation don't actually fully represent Schemas. It seems to me
that
>>>>> the current java-only Schemas are made up three concepts that are
>>>>> intertwined:
>>>>> (a) The Java SDK specific code for schema inference, type coercion,
>>>>> and "schema-aware" transforms.
>>>>> (b) A RowCoder[1] that encodes Rows[2] which have a particular
>>>>> Schema[3].
>>>>> (c) A SchemaCoder[4] that has a RowCoder for a particular schema, and
>>>>> functions for converting Rows with that schema to/from a Java type T.
Those
>>>>> functions and the RowCoder are then composed to provider a Coder for
the
>>>>> type T.
>>>>>
>>>>
>>>> RowCoder is currently just an internal implementation detail, it can be
>>>> eliminated. SchemaCoder is the only thing that determines a schema today.
>>>>
>>> Why not keep it around? I think it would make sense to have a RowCoder
>>> implementation in every SDK, as well as something like SchemaCoder that
>>> defines a conversion from that SDK's "Row" to the language type.
>>>
>>
>> The point is that from a programmer's perspective, there is nothing much
>> special about Row. Any type can have a schema, and the only special thing
>> about Row is that it's always guaranteed to exist. From that standpoint,
>> Row is nearly an implementation detail. Today RowCoder is never set on
>> _any_ PCollection, it's literally just used as a helper library, so there's
>> no real need for it to exist as a "Coder."
>>
>>
>>>
>>>>
>>>>>
>>>>> We're not concerned with (a) at this time since that's specific to the
>>>>> SDK, not the interface between them. My understanding is we just want
to
>>>>> define a portable representation for (b) and/or (c).
>>>>>
>>>>> What has been discussed so far is really just a portable
>>>>> representation for (b), the RowCoder, since the discussion is only around
>>>>> how to represent the schema itself and not the to/from functions.
>>>>>
>>>>
>>>> Correct. The to/from functions are actually related to a). One of the
>>>> big goals of schemas was that users should not be forced to operate on rows
>>>> to get schemas. A user can create PCollection<MyRandomType> and as
long as
>>>> the SDK can infer a schema from MyRandomType, the user never needs to even
>>>> see a Row object. The to/fromRow functions are what make this work today.
>>>>
>>>>
>>>
>>> One of the points I'd like to make is that this type coercion is a
>>> useful concept on it's own, separate from schemas. It's especially useful
>>> for a type that has a schema and is encoded by RowCoder since that can
>>> represent many more types, but the type coercion doesn't have to be tied to
>>> just schemas and RowCoder. We could also do type coercion for types that
>>> are effectively wrappers around an integer or a string. It could just be a
>>> general way to map language types to base types (i.e. types that we have a
>>> coder for). Then it just becomes a general framework for extending coders
>>> to represent more language types.
>>>
>>
>> Let's not tie those conversations. Maybe a similar concept will hold true
>> for general coders (or we might decide to get rid of coders in favor of
>> schemas, in which case that becomes moot), but I don't think we should
>> prematurely generalize.
>>
>>
>>>
>>>
>>>
>>>> One of the outstanding questions for that schema representation is how
>>>>> to represent logical types, which may or may not have some language type
in
>>>>> each SDK (the canonical example being a timsetamp type with seconds and
>>>>> nanos and java.time.Instant). I think this question is critically
>>>>> important, because (c), the SchemaCoder, is actually *defining a logical
>>>>> type* with a language type T in the Java SDK. This becomes clear when
you
>>>>> compare SchemaCoder[4] to the Schema.LogicalType interface[5] - both
>>>>> essentially have three attributes: a base type, and two functions for
>>>>> converting to/from that base type. The only difference is for SchemaCoder
>>>>> that base type must be a Row so it can be represented by a Schema alone,
>>>>> while LogicalType can have any base type that can be represented by
>>>>> FieldType, including a Row.
>>>>>
>>>>
>>>> This is not true actually. SchemaCoder can have any base type, that's
>>>> why (in Java) it's SchemaCoder<T>. This is why PCollection<T>
can have a
>>>> schema, even if T is not Row.
>>>>
>>>>
>>> I'm not sure I effectively communicated what I meant - When I said
>>> SchemaCoder's "base type" I wasn't referring to T, I was referring to the
>>> base FieldType, whose coder we use for this type. I meant "base type" to be
>>> analogous to LogicalType's `getBaseType`, or what Kenn is suggesting we
>>> call "representation" in the portable beam schemas doc. To define some
>>> terms from my original message:
>>> base type = an instance of FieldType, crucially this is something that
>>> we have a coder for (be it VarIntCoder, Utf8Coder, RowCoder, ...)
>>> language type (or "T", "type T", "logical type") = Some Java class (or
>>> something analogous in the other SDKs) that we may or may not have a coder
>>> for. It's possible to define functions for converting instances of the
>>> language type to/from the base type.
>>>
>>> I was just trying to make the case that SchemaCoder is really a special
>>> case of LogicalType, where `getBaseType` always returns a Row with the
>>> stored Schema.
>>>
>>
>> Yeah, I think  I got that point.
>>
>> Can you propose what the protos would look like in this case? Right now
>> LogicalType does not contain the to/from conversion functions in the proto.
>> Do you think we'll need to add these in?
>>
>>
>>> To make the point with code: SchemaCoder<T> can be made to implement
>>> Schema.LogicalType<T,Row> with trivial implementations of getBaseType,
>>> toBaseType, and toInputType (I'm not trying to say we should or shouldn't
>>> do this, just using it illustrate my point):
>>>
>>> class SchemaCoder extends CustomCoder<T> implements
>>> Schema.LogicalType<T, Row> {
>>>   ...
>>>
>>>   @Override
>>>   FieldType getBaseType() {
>>>     return FieldType.row(getSchema());
>>>   }
>>>
>>>   @Override
>>>   public Row toBaseType() {
>>>     return this.toRowFunction.apply(input);
>>>   }
>>>
>>>   @Override
>>>   public T toInputType(Row base) {
>>>     return this.fromRowFunction.apply(base);
>>>   }
>>>   ...
>>> }
>>>
>>>
>>>>> I think it may make sense to fully embrace this duality, by letting
>>>>> SchemaCoder have a baseType other than just Row and renaming it to
>>>>> LogicalTypeCoder/LanguageTypeCoder. The current Java SDK schema-aware
>>>>> transforms (a) would operate only on LogicalTypeCoders with a Row base
>>>>> type. Perhaps some of the current schema logic could  alsobe applied
more
>>>>> generally to any logical type  - for example, to provide type coercion
for
>>>>> logical types with a base type other than Row, like int64 and a timestamp
>>>>> class backed by millis, or fixed size bytes and a UUID class. And having
a
>>>>> portable representation that represents those (non Row backed) logical
>>>>> types with some URN would also allow us to pass them to other languages
>>>>> without unnecessarily wrapping them in a Row in order to use SchemaCoder.
>>>>>
>>>>
>>>> I think the actual overlap here is between the to/from functions in
>>>> SchemaCoder (which is what allows SchemaCoder<T> where T != Row) and
the
>>>> equivalent functionality in LogicalType. However making all of schemas
>>>> simply just a logical type feels a bit awkward and circular to me. Maybe
we
>>>> should refactor that part out into a LogicalTypeConversion proto, and
>>>> reference that from both LogicalType and from SchemaCoder?
>>>>
>>>
>>> LogicalType is already potentially circular though. A schema can have a
>>> field with a logical type, and that logical type can have a base type of
>>> Row with a field with a logical type (and on and on...). To me it seems
>>> elegant, not awkward, to recognize that SchemaCoder is just a special case
>>> of this concept.
>>>
>>> Something like the LogicalTypeConversion proto would definitely be an
>>> improvement, but I would still prefer just using a top-level logical type :)
>>>
>>>>
>>>>
>>>> I've added a section to the doc [6] to propose this alternative in the
>>>>> context of the portable representation but I wanted to bring it up here
as
>>>>> well to solicit feedback.
>>>>>
>>>>> [1]
>>>>> https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/RowCoder.java#L41
>>>>> [2]
>>>>> https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/values/Row.java#L59
>>>>> [3]
>>>>> https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/Schema.java#L48
>>>>> [4]
>>>>> https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/SchemaCoder.java#L33
>>>>> [5]
>>>>> https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/Schema.java#L489
>>>>> [6]
>>>>> https://docs.google.com/document/d/1uu9pJktzT_O3DxGd1-Q2op4nRk4HekIZbzi-0oTAips/edit?ts=5cdf6a5b#heading=h.7570feur1qin
>>>>>
>>>>> On Fri, May 10, 2019 at 9:16 AM Brian Hulette <bhulette@google.com>
>>>>> wrote:
>>>>>
>>>>>> Ah thanks! I added some language there.
>>>>>>
>>>>>> *From: *Kenneth Knowles <kenn@apache.org>
>>>>>> *Date: *Thu, May 9, 2019 at 5:31 PM
>>>>>> *To: *dev
>>>>>>
>>>>>>
>>>>>>> *From: *Brian Hulette <bhulette@google.com>
>>>>>>> *Date: *Thu, May 9, 2019 at 2:02 PM
>>>>>>> *To: * <dev@beam.apache.org>
>>>>>>>
>>>>>>> We briefly discussed using arrow schemas in place of beam schemas
>>>>>>>> entirely in an arrow thread [1]. The biggest reason not to
this was that we
>>>>>>>> wanted to have a type for large iterables in beam schemas.
But given that
>>>>>>>> large iterables aren't currently implemented, beam schemas
look very
>>>>>>>> similar to arrow schemas.
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> I think it makes sense to take inspiration from arrow schemas
where
>>>>>>>> possible, and maybe even copy them outright. Arrow already
has a portable
>>>>>>>> (flatbuffers) schema representation [2], and implementations
for it in many
>>>>>>>> languages that we may be able to re-use as we bring schemas
to more SDKs
>>>>>>>> (the project has Python and Go implementations). There are
a couple of
>>>>>>>> concepts in Arrow schemas that are specific for the format
and wouldn't
>>>>>>>> make sense for us, (fields can indicate whether or not they
are dictionary
>>>>>>>> encoded, and the schema has an endianness field), but if
you drop those
>>>>>>>> concepts the arrow spec looks pretty similar to the beam
proto spec.
>>>>>>>>
>>>>>>>
>>>>>>> FWIW I left a blank section in the doc for filling out what the
>>>>>>> differences are and why, and conversely what the interop opportunities
may
>>>>>>> be. Such sections are some of my favorite sections of design
docs.
>>>>>>>
>>>>>>> Kenn
>>>>>>>
>>>>>>>
>>>>>>> Brian
>>>>>>>>
>>>>>>>> [1]
>>>>>>>> https://lists.apache.org/thread.html/6be7715e13b71c2d161e4378c5ca1c76ac40cfc5988a03ba87f1c434@%3Cdev.beam.apache.org%3E
>>>>>>>> [2]
>>>>>>>> https://github.com/apache/arrow/blob/master/format/Schema.fbs#L194
>>>>>>>>
>>>>>>>> *From: *Robert Bradshaw <robertwb@google.com>
>>>>>>>> *Date: *Thu, May 9, 2019 at 1:38 PM
>>>>>>>> *To: *dev
>>>>>>>>
>>>>>>>> From: Reuven Lax <relax@google.com>
>>>>>>>>> Date: Thu, May 9, 2019 at 7:29 PM
>>>>>>>>> To: dev
>>>>>>>>>
>>>>>>>>> > Also in the future we might be able to do optimizations
at the
>>>>>>>>> runner level if at the portability layer we understood
schemes instead of
>>>>>>>>> just raw coders. This could be things like only parsing
a subset of a row
>>>>>>>>> (if we know only a few fields are accessed) or using
a columnar data
>>>>>>>>> structure like Arrow to encode batches of rows across
portability. This
>>>>>>>>> doesn't affect data semantics of course, but having a
richer,
>>>>>>>>> more-expressive type system opens up other opportunities.
>>>>>>>>>
>>>>>>>>> But we could do all of that with a RowCoder we understood
to
>>>>>>>>> designate
>>>>>>>>> the type(s), right?
>>>>>>>>>
>>>>>>>>> > On Thu, May 9, 2019 at 10:16 AM Robert Bradshaw
<
>>>>>>>>> robertwb@google.com> wrote:
>>>>>>>>> >>
>>>>>>>>> >> On the flip side, Schemas are equivalent to
the space of Coders
>>>>>>>>> with
>>>>>>>>> >> the addition of a RowCoder and the ability to
materialize to
>>>>>>>>> something
>>>>>>>>> >> other than bytes, right? (Perhaps I'm missing
something big
>>>>>>>>> here...)
>>>>>>>>> >> This may make a backwards-compatible transition
easier.
>>>>>>>>> (SDK-side, the
>>>>>>>>> >> ability to reason about and operate on such
types is of course
>>>>>>>>> much
>>>>>>>>> >> richer than anything Coders offer right now.)
>>>>>>>>> >>
>>>>>>>>> >> From: Reuven Lax <relax@google.com>
>>>>>>>>> >> Date: Thu, May 9, 2019 at 4:52 PM
>>>>>>>>> >> To: dev
>>>>>>>>> >>
>>>>>>>>> >> > FYI I can imagine a world in which we have
no coders. We
>>>>>>>>> could define the entire model on top of schemas. Today's
"Coder" is
>>>>>>>>> completely equivalent to a single-field schema with a
logical-type field
>>>>>>>>> (actually the latter is slightly more expressive as you
aren't forced to
>>>>>>>>> serialize into bytes).
>>>>>>>>> >> >
>>>>>>>>> >> > Due to compatibility constraints and the
effort that would
>>>>>>>>> be  involved in such a change, I think the practical
decision should be for
>>>>>>>>> schemas and coders to coexist for the time being. However
when we start
>>>>>>>>> planning Beam 3.0, deprecating coders is something I
would like to suggest.
>>>>>>>>> >> >
>>>>>>>>> >> > On Thu, May 9, 2019 at 7:48 AM Robert Bradshaw
<
>>>>>>>>> robertwb@google.com> wrote:
>>>>>>>>> >> >>
>>>>>>>>> >> >> From: Kenneth Knowles <kenn@apache.org>
>>>>>>>>> >> >> Date: Thu, May 9, 2019 at 10:05 AM
>>>>>>>>> >> >> To: dev
>>>>>>>>> >> >>
>>>>>>>>> >> >> > This is a huge development. Top
posting because I can be
>>>>>>>>> more compact.
>>>>>>>>> >> >> >
>>>>>>>>> >> >> > I really think after the initial
idea converges this needs
>>>>>>>>> a design doc with goals and alternatives. It is an extraordinarily
>>>>>>>>> consequential model change. So in the spirit of doing
the work / bias
>>>>>>>>> towards action, I created a quick draft at
>>>>>>>>> https://s.apache.org/beam-schemas and added everyone
on this
>>>>>>>>> thread as editors. I am still in the process of writing
this to match the
>>>>>>>>> thread.
>>>>>>>>> >> >>
>>>>>>>>> >> >> Thanks! Added some comments there.
>>>>>>>>> >> >>
>>>>>>>>> >> >> > *Multiple timestamp resolutions*:
you can use logcial
>>>>>>>>> types to represent nanos the same way Java and proto
do.
>>>>>>>>> >> >>
>>>>>>>>> >> >> As per the other discussion, I'm unsure
the value in
>>>>>>>>> supporting
>>>>>>>>> >> >> multiple timestamp resolutions is high
enough to outweigh
>>>>>>>>> the cost.
>>>>>>>>> >> >>
>>>>>>>>> >> >> > *Why multiple int types?* The
domain of values for these
>>>>>>>>> types are different. For a language with one "int" or
"number" type, that's
>>>>>>>>> another domain of values.
>>>>>>>>> >> >>
>>>>>>>>> >> >> What is the value in having different
domains? If your data
>>>>>>>>> has a
>>>>>>>>> >> >> natural domain, chances are it doesn't
line up exactly with
>>>>>>>>> one of
>>>>>>>>> >> >> these. I guess it's for languages whose
types have specific
>>>>>>>>> domains?
>>>>>>>>> >> >> (There's also compactness in representation,
encoded and
>>>>>>>>> in-memory,
>>>>>>>>> >> >> though I'm not sure that's high.)
>>>>>>>>> >> >>
>>>>>>>>> >> >> > *Columnar/Arrow*: making sure
we unlock the ability to
>>>>>>>>> take this path is Paramount. So tying it directly to
a row-oriented coder
>>>>>>>>> seems counterproductive.
>>>>>>>>> >> >>
>>>>>>>>> >> >> I don't think Coders are necessarily
row-oriented. They are,
>>>>>>>>> however,
>>>>>>>>> >> >> bytes-oriented. (Perhaps they need
not be.) There seems to
>>>>>>>>> be a lot of
>>>>>>>>> >> >> overlap between what Coders express
in terms of element
>>>>>>>>> typing
>>>>>>>>> >> >> information and what Schemas express,
and I'd rather have
>>>>>>>>> one concept
>>>>>>>>> >> >> if possible. Or have a clear division
of responsibilities.
>>>>>>>>> >> >>
>>>>>>>>> >> >> > *Multimap*: what does it add over
an array-valued map or
>>>>>>>>> large-iterable-valued map? (honest question, not rhetorical)
>>>>>>>>> >> >>
>>>>>>>>> >> >> Multimap has a different notion of
what it means to contain
>>>>>>>>> a value,
>>>>>>>>> >> >> can handle (unordered) unions of non-disjoint
keys, etc.
>>>>>>>>> Maybe this
>>>>>>>>> >> >> isn't worth a new primitive type.
>>>>>>>>> >> >>
>>>>>>>>> >> >> > *URN/enum for type names*: I see
the case for both. The
>>>>>>>>> core types are fundamental enough they should never really
change - after
>>>>>>>>> all, proto, thrift, avro, arrow, have addressed this
(not to mention most
>>>>>>>>> programming languages). Maybe additions once every few
years. I prefer the
>>>>>>>>> smallest intersection of these schema languages. A oneof
is more clear,
>>>>>>>>> while URN emphasizes the similarity of built-in and logical
types.
>>>>>>>>> >> >>
>>>>>>>>> >> >> Hmm... Do we have any examples of the
multi-level
>>>>>>>>> primitive/logical
>>>>>>>>> >> >> type in any of these other systems?
I have a bias towards
>>>>>>>>> all types
>>>>>>>>> >> >> being on the same footing unless there
is compelling reason
>>>>>>>>> to divide
>>>>>>>>> >> >> things into primitive/use-defined ones.
>>>>>>>>> >> >>
>>>>>>>>> >> >> Here it seems like the most essential
value of the primitive
>>>>>>>>> type set
>>>>>>>>> >> >> is to describe the underlying representation,
for encoding
>>>>>>>>> elements in
>>>>>>>>> >> >> a variety of ways (notably columnar,
but also interfacing
>>>>>>>>> with other
>>>>>>>>> >> >> external systems like IOs). Perhaps,
rather than the previous
>>>>>>>>> >> >> suggestion of making everything a logical
of bytes, this
>>>>>>>>> could be made
>>>>>>>>> >> >> clear by still making everything a
logical type, but renaming
>>>>>>>>> >> >> "TypeName" to Representation. There
would be URNs (typically
>>>>>>>>> with
>>>>>>>>> >> >> empty payloads) for the various primitive
types (whose
>>>>>>>>> mapping to
>>>>>>>>> >> >> their representations would be the
identity).
>>>>>>>>> >> >>
>>>>>>>>> >> >> - Robert
>>>>>>>>>
>>>>>>>>

Mime
View raw message