arrow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
Subject arrow git commit: ARROW-252: Add implementation guidelines to the documentation
Date Sat, 20 Aug 2016 17:40:03 GMT
Repository: arrow
Updated Branches:
  refs/heads/master 812201a2d -> 78619686f

ARROW-252: Add implementation guidelines to the documentation

Author: Julien Le Dem <>

Closes #120 from julienledem/arrow_252_impl_guidelines and squashes the following commits:

caf6994 [Julien Le Dem] ARROW-252: review feedback
6b68ce1 [Julien Le Dem] ARROW-252: Add implementation guidelines to the documentation


Branch: refs/heads/master
Commit: 78619686f44da5a28319032551b07ddfadc26468
Parents: 812201a
Author: Julien Le Dem <>
Authored: Sat Aug 20 10:39:50 2016 -0700
Committer: Julien Le Dem <>
Committed: Sat Aug 20 10:39:50 2016 -0700

 format/ | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)
diff --git a/format/ b/format/
new file mode 100644
index 0000000..14f1057
--- /dev/null
+++ b/format/
@@ -0,0 +1,17 @@
+# Implementation guidelines
+An execution engine (or framework, or UDF executor, or storage engine, etc) can implements
only a subset of the Arrow spec and/or extend it given the following constraints:
+## Implementing a subset the spec
+### If only producing (and not consuming) arrow vectors.
+Any subset of the vector spec and the corresponding metadata can be implemented.
+### If consuming and producing vectors
+There is a minimal subset of vectors to be supported.
+Production of a subset of vectors and their corresponding metadata is always fine.
+Consumption of vectors should at least convert the unsupported input vectors to the supported
subset (for example Timestamp.millis to timestamp.micros or int32 to int64)
+## Extensibility
+An execution engine implementor can also extend their memory representation with their own
vectors internally as long as they are never exposed. Before sending data to another system
expecting Arrow data these custom vectors should be converted to a type that exist in the
Arrow spec.
+An example of this is operating on compressed data.
+These custom vectors are not exchanged externaly and there is no support for custom metadata.

View raw message