arrow-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Wes McKinney <wesmck...@gmail.com>
Subject Re: Schema serialization and bindings generation?
Date Sat, 26 Oct 2019 18:26:59 GMT
hi Max,

Note that Flatbuffers and the Arrow columnar format are certainly not
competing with each other. While we use Flatbuffers to serialize
schemas, data headers, and other metadata, the data itself is not
stored inside a Flatbuffer or similar serialized structure. This is a
common misconception so worth restating.

You can see the serialization details in the columnar format documentation

http://arrow.apache.org/docs/format/Columnar.html#serialization-and-interprocess-communication-ipc

Each library generally has functions to serialize schemas into
language-independent protocol messages.

Regarding "binding generation tools", as I recall there has been some
discussion in the past of generating schema-specific accessor APIs
similar to how Protobuf/Thrift/Flatbuffers generates accessor APIs,
this seems like an interesting direction to explore for a volunteer in
the community.

- Wes

On Fri, Oct 25, 2019 at 1:49 PM Max Burke <max@urbanlogiq.com> wrote:
>
> Hello!
>
> I had a couple questions. I have a system that is currently using Flatbuffers because
they provide a couple features: 1) flatbuffers support generating flatbuffer-encoded representations
of schemas (ie: reflection) and 2) flatbuffers have a bindings generator for type-safe access
in client languages.
>
> That said, the ecosystem evolving around Arrow is really intriguing.
>
> Is there any equivalent now for serializing schemas/reflection?
>
> Are there any plans for supporting binding generation tools?
>
> --
> -Max

Mime
View raw message