arrow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Julien Le Dem (JIRA)" <>
Subject [jira] [Commented] (ARROW-474) Create an Arrow streaming file fomat
Date Wed, 11 Jan 2017 23:24:17 GMT


Julien Le Dem commented on ARROW-474:

[~nongli] I added you to the list of contributors (which controls who can be assigned a JIRA)
and assigned this jira to you.

> Create an Arrow streaming file fomat
> ------------------------------------
>                 Key: ARROW-474
>                 URL:
>             Project: Apache Arrow
>          Issue Type: Improvement
>            Reporter: Nong Li
>            Assignee: Nong Li
> It would be helpful to have a file/container layout that allows for streaming consumer
and producers of arrow data.
> As a motivating example, a reader could have this API:
> Iterator<Record> read( source). 
> Similar a writer can have:
> void write(RecordBatch, dest)
> The streams can be from a variety of sources: it be files, over the network, shared memory,
pipe, etc. The layout would make it possible to implement
> both APIs without requiring intermediate buffering. 
> Speaking with Julien, this involves creating a simple container format similar to ARROW-264
but suitable for streaming consumer and producers. The layout in ARROW-264 has the schema
at the end so can't be used in a streaming way. This format would likely be a rearranging
of the current File layout pieces, something similar to:
> MAGIC, schema, totalBatches, (RecordBatchHeader, RecordBatchBody)*, MAGIC (or some end
> totalBatches is optional for streaming producers that don't know.

This message was sent by Atlassian JIRA

View raw message