arrow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From w...@apache.org
Subject [26/27] arrow-site git commit: Update Python documentation
Date Mon, 08 May 2017 04:53:11 GMT
http://git-wip-us.apache.org/repos/asf/arrow-site/blob/6360599f/docs/python/_sources/memory.rst.txt
----------------------------------------------------------------------
diff --git a/docs/python/_sources/memory.rst.txt b/docs/python/_sources/memory.rst.txt
new file mode 100644
index 0000000..d1020da
--- /dev/null
+++ b/docs/python/_sources/memory.rst.txt
@@ -0,0 +1,235 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements.  See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership.  The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License.  You may obtain a copy of the License at
+
+..   http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied.  See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. currentmodule:: pyarrow
+.. _io:
+
+Memory and IO Interfaces
+========================
+
+This section will introduce you to the major concepts in PyArrow's memory
+management and IO systems:
+
+* Buffers
+* File-like and stream-like objects
+* Memory pools
+
+pyarrow.Buffer
+--------------
+
+The :class:`~pyarrow.Buffer` object wraps the C++ ``arrow::Buffer`` type and is
+the primary tool for memory management in Apache Arrow in C++. It permits
+higher-level array classes to safely interact with memory which they may or may
+not own. ``arrow::Buffer`` can be zero-copy sliced to permit Buffers to cheaply
+reference other Buffers, while preserving memory lifetime and clean
+parent-child relationships.
+
+There are many implementations of ``arrow::Buffer``, but they all provide a
+standard interface: a data pointer and length. This is similar to Python's
+built-in `buffer protocol` and ``memoryview`` objects.
+
+A :class:`~pyarrow.Buffer` can be created from any Python object which
+implements the buffer protocol. Let's consider a bytes object:
+
+.. ipython:: python
+
+   import pyarrow as pa
+
+   data = b'abcdefghijklmnopqrstuvwxyz'
+   buf = pa.frombuffer(data)
+   buf
+   buf.size
+
+Creating a Buffer in this way does not allocate any memory; it is a zero-copy
+view on the memory exported from the ``data`` bytes object.
+
+The Buffer's ``to_pybytes`` method can convert to a Python byte string:
+
+.. ipython:: python
+
+   buf.to_pybytes()
+
+Buffers can be used in circumstances where a Python buffer or memoryview is
+required, and such conversions are also zero-copy:
+
+.. ipython:: python
+
+   memoryview(buf)
+
+.. _io.native_file:
+
+Native Files
+------------
+
+The Arrow C++ libraries have several abstract interfaces for different kinds of
+IO objects:
+
+* Read-only streams
+* Read-only files supporting random access
+* Write-only streams
+* Write-only files supporting random access
+* File supporting reads, writes, and random access
+
+In the the interest of making these objects behave more like Python's built-in
+``file`` objects, we have defined a :class:`~pyarrow.NativeFile` base class
+which is intended to mimic Python files and able to be used in functions where
+a Python file (such as ``file`` or ``BytesIO``) is expected.
+
+:class:`~pyarrow.NativeFile` has some important features which make it
+preferable to using Python files with PyArrow where possible:
+
+* Other Arrow classes can access the internal C++ IO objects natively, and do
+  not need to acquire the Python GIL
+* Native C++ IO may be able to do zero-copy IO, such as with memory maps
+
+There are several kinds of :class:`~pyarrow.NativeFile` options available:
+
+* :class:`~pyarrow.OSFile`, a native file that uses your operating system's
+  file descriptors
+* :class:`~pyarrow.MemoryMappedFile`, for reading (zero-copy) and writing with
+  memory maps
+* :class:`~pyarrow.BufferReader`, for reading :class:`~pyarrow.Buffer` objects
+  as a file
+* :class:`~pyarrow.InMemoryOutputStream`, for writing data in-memory, producing
+  a Buffer at the end
+* :class:`~pyarrow.HdfsFile`, for reading and writing data to the Hadoop Filesystem
+* :class:`~pyarrow.PythonFile`, for interfacing with Python file objects in C++
+
+We will discuss these in the following sections after explaining memory pools.
+
+Memory Pools
+------------
+
+All memory allocations and deallocations (like ``malloc`` and ``free`` in C)
+are tracked in an instance of ``arrow::MemoryPool``. This means that we can
+then precisely track amount of memory that has been allocated:
+
+.. ipython:: python
+
+   pa.total_allocated_bytes()
+
+PyArrow uses a default built-in memory pool, but in the future there may be
+additional memory pools (and subpools) to choose from. Let's consider an
+``InMemoryOutputStream``, which is like a ``BytesIO``:
+
+.. ipython:: python
+
+   stream = pa.InMemoryOutputStream()
+   stream.write(b'foo')
+   pa.total_allocated_bytes()
+   for i in range(1024): stream.write(b'foo')
+   pa.total_allocated_bytes()
+
+The default allocator requests memory in a minimum increment of 64 bytes. If
+the stream is garbaged-collected, all of the memory is freed:
+
+.. ipython:: python
+
+   stream = None
+   pa.total_allocated_bytes()
+
+Classes and functions that may allocate memory will often have an option to
+pass in a custom memory pool:
+
+.. ipython:: python
+
+   my_pool = pa.jemalloc_memory_pool()
+   my_pool
+   my_pool.bytes_allocated()
+   stream = pa.InMemoryOutputStream(my_pool)
+   stream.write(b'foo')
+   my_pool.bytes_allocated()
+
+On-Disk and Memory Mapped Files
+-------------------------------
+
+PyArrow includes two ways to interact with data on disk: standard operating
+system-level file APIs, and memory-mapped files. In regular Python we can
+write:
+
+.. ipython:: python
+
+   with open('example.dat', 'wb') as f:
+       f.write(b'some example data')
+
+Using pyarrow's :class:`~pyarrow.OSFile` class, you can write:
+
+.. ipython:: python
+
+   with pa.OSFile('example2.dat', 'wb') as f:
+       f.write(b'some example data')
+
+For reading files, you can use ``OSFile`` or
+:class:`~pyarrow.MemoryMappedFile`. The difference between these is that
+:class:`~pyarrow.OSFile` allocates new memory on each read, like Python file
+objects. In reads from memory maps, the library constructs a buffer referencing
+the mapped memory without any memory allocation or copying:
+
+.. ipython:: python
+
+   file_obj = pa.OSFile('example.dat')
+   mmap = pa.memory_map('example.dat')
+   file_obj.read(4)
+   mmap.read(4)
+
+The ``read`` method implements the standard Python file ``read`` API. To read
+into Arrow Buffer objects, use ``read_buffer``:
+
+.. ipython:: python
+
+   mmap.seek(0)
+   buf = mmap.read_buffer(4)
+   print(buf)
+   buf.to_pybytes()
+
+Many tools in PyArrow, particular the Apache Parquet interface and the file and
+stream messaging tools, are more efficient when used with these ``NativeFile``
+types than with normal Python file objects.
+
+.. ipython:: python
+   :suppress:
+
+   buf = mmap = file_obj = None
+   !rm example.dat
+   !rm example2.dat
+
+In-Memory Reading and Writing
+-----------------------------
+
+To assist with serialization and deserialization of in-memory data, we have
+file interfaces that can read and write to Arrow Buffers.
+
+.. ipython:: python
+
+   writer = pa.InMemoryOutputStream()
+   writer.write(b'hello, friends')
+
+   buf = writer.get_result()
+   buf
+   buf.size
+   reader = pa.BufferReader(buf)
+   reader.seek(7)
+   reader.read(7)
+
+These have similar semantics to Python's built-in ``io.BytesIO``.
+
+Hadoop Filesystem
+-----------------
+
+:class:`~pyarrow.HdfsFile` is an implementation of :class:`~pyarrow.NativeFile`
+that can read and write to the Hadoop filesytem. Read more in the
+:ref:`Filesystems Section <hdfs>`.

http://git-wip-us.apache.org/repos/asf/arrow-site/blob/6360599f/docs/python/_sources/pandas.rst.txt
----------------------------------------------------------------------
diff --git a/docs/python/_sources/pandas.rst.txt b/docs/python/_sources/pandas.rst.txt
index 34445ae..cb7a56d 100644
--- a/docs/python/_sources/pandas.rst.txt
+++ b/docs/python/_sources/pandas.rst.txt
@@ -15,17 +15,17 @@
 .. specific language governing permissions and limitations
 .. under the License.
 
-Pandas Interface
-================
+Using PyArrow with pandas
+=========================
 
-To interface with Pandas, PyArrow provides various conversion routines to
-consume Pandas structures and convert back to them.
+To interface with pandas, PyArrow provides various conversion routines to
+consume pandas structures and convert back to them.
 
 DataFrames
 ----------
 
-The equivalent to a Pandas DataFrame in Arrow is a :class:`pyarrow.table.Table`.
-Both consist of a set of named columns of equal length. While Pandas only
+The equivalent to a pandas DataFrame in Arrow is a :class:`pyarrow.table.Table`.
+Both consist of a set of named columns of equal length. While pandas only
 supports flat columns, the Table also provides nested columns, thus it can
 represent more data than a DataFrame, so a full conversion is not always possible.
 
@@ -33,9 +33,9 @@ Conversion from a Table to a DataFrame is done by calling
 :meth:`pyarrow.table.Table.to_pandas`. The inverse is then achieved by using
 :meth:`pyarrow.Table.from_pandas`. This conversion routine provides the
 convience parameter ``timestamps_to_ms``. Although Arrow supports timestamps of
-different resolutions, Pandas only supports nanosecond timestamps and most
+different resolutions, pandas only supports nanosecond timestamps and most
 other systems (e.g. Parquet) only work on millisecond timestamps. This parameter
-can be used to already do the time conversion during the Pandas to Arrow
+can be used to already do the time conversion during the pandas to Arrow
 conversion.
 
 .. code-block:: python
@@ -44,35 +44,35 @@ conversion.
     import pandas as pd
 
     df = pd.DataFrame({"a": [1, 2, 3]})
-    # Convert from Pandas to Arrow
+    # Convert from pandas to Arrow
     table = pa.Table.from_pandas(df)
-    # Convert back to Pandas
+    # Convert back to pandas
     df_new = table.to_pandas()
 
 
 Series
 ------
 
-In Arrow, the most similar structure to a Pandas Series is an Array.
+In Arrow, the most similar structure to a pandas Series is an Array.
 It is a vector that contains data of the same type as linear memory. You can
-convert a Pandas Series to an Arrow Array using :meth:`pyarrow.array.from_pandas_series`.
+convert a pandas Series to an Arrow Array using :meth:`pyarrow.array.from_pandas_series`.
 As Arrow Arrays are always nullable, you can supply an optional mask using
 the ``mask`` parameter to mark all null-entries.
 
 Type differences
 ----------------
 
-With the current design of Pandas and Arrow, it is not possible to convert all
-column types unmodified. One of the main issues here is that Pandas has no
+With the current design of pandas and Arrow, it is not possible to convert all
+column types unmodified. One of the main issues here is that pandas has no
 support for nullable columns of arbitrary type. Also ``datetime64`` is currently
 fixed to nanosecond resolution. On the other side, Arrow might be still missing
 support for some types.
 
-Pandas -> Arrow Conversion
+pandas -> Arrow Conversion
 ~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 +------------------------+--------------------------+
-| Source Type (Pandas)   | Destination Type (Arrow) |
+| Source Type (pandas)   | Destination Type (Arrow) |
 +========================+==========================+
 | ``bool``               | ``BOOL``                 |
 +------------------------+--------------------------+
@@ -91,11 +91,11 @@ Pandas -> Arrow Conversion
 | ``datetime.date``      | ``DATE``                 |
 +------------------------+--------------------------+
 
-Arrow -> Pandas Conversion
+Arrow -> pandas Conversion
 ~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 +-------------------------------------+--------------------------------------------------------+
-| Source Type (Arrow)                 | Destination Type (Pandas)                              |
+| Source Type (Arrow)                 | Destination Type (pandas)                              |
 +=====================================+========================================================+
 | ``BOOL``                            | ``bool``                                               |
 +-------------------------------------+--------------------------------------------------------+

http://git-wip-us.apache.org/repos/asf/arrow-site/blob/6360599f/docs/python/_sources/parquet.rst.txt
----------------------------------------------------------------------
diff --git a/docs/python/_sources/parquet.rst.txt b/docs/python/_sources/parquet.rst.txt
index 8e011e4..3317b99 100644
--- a/docs/python/_sources/parquet.rst.txt
+++ b/docs/python/_sources/parquet.rst.txt
@@ -15,77 +15,218 @@
 .. specific language governing permissions and limitations
 .. under the License.
 
-Reading/Writing Parquet files
-=============================
+.. currentmodule:: pyarrow
+.. _parquet:
 
-If you have built ``pyarrow`` with Parquet support, i.e. ``parquet-cpp`` was
-found during the build, you can read files in the Parquet format to/from Arrow
-memory structures. The Parquet support code is located in the
-:mod:`pyarrow.parquet` module and your package needs to be built with the
-``--with-parquet`` flag for ``build_ext``.
+Reading and Writing the Apache Parquet Format
+=============================================
 
-Reading Parquet
----------------
+The `Apache Parquet <http://parquet.apache.org/>`_ project provides a
+standardized open-source columnar storage format for use in data analysis
+systems. It was created originally for use in `Apache Hadoop
+<http://hadoop.apache.org/>`_ with systems like `Apache Drill
+<http://drill.apache.org>`_, `Apache Hive <http://hive.apache.org>`_, `Apache
+Impala (incubating) <http://impala.apache.org>`_, and `Apache Spark
+<http://spark.apache.org>`_ adopting it as a shared standard for high
+performance data IO.
 
-To read a Parquet file into Arrow memory, you can use the following code
-snippet. It will read the whole Parquet file into memory as an
-:class:`~pyarrow.table.Table`.
+Apache Arrow is an ideal in-memory transport layer for data that is being read
+or written with Parquet files. We have been concurrently developing the `C++
+implementation of Apache Parquet <http://github.com/apache/parquet-cpp>`_,
+which includes a native, multithreaded C++ adapter to and from in-memory Arrow
+data. PyArrow includes Python bindings to this code, which thus enables reading
+and writing Parquet files with pandas as well.
 
-.. code-block:: python
+Obtaining PyArrow with Parquet Support
+--------------------------------------
+
+If you installed ``pyarrow`` with pip or conda, it should be built with Parquet
+support bundled:
+
+.. ipython:: python
+
+   import pyarrow.parquet as pq
+
+If you are building ``pyarrow`` from source, you must also build `parquet-cpp
+<http://github.com/apache/parquet-cpp>`_ and enable the Parquet extensions when
+building ``pyarrow``. See the :ref:`Development <development>` page for more
+details.
+
+Reading and Writing Single Files
+--------------------------------
+
+The functions :func:`~.parquet.read_table` and :func:`~.parquet.write_table`
+read and write the :ref:`pyarrow.Table <data.table>` objects, respectively.
+
+Let's look at a simple table:
+
+.. ipython:: python
+
+   import numpy as np
+   import pandas as pd
+   import pyarrow as pa
+
+   df = pd.DataFrame({'one': [-1, np.nan, 2.5],
+                      'two': ['foo', 'bar', 'baz'],
+                      'three': [True, False, True]})
+   table = pa.Table.from_pandas(df)
+
+We write this to Parquet format with ``write_table``:
+
+.. ipython:: python
+
+   import pyarrow.parquet as pq
+   pq.write_table(table, 'example.parquet')
+
+This creates a single Parquet file. In practice, a Parquet dataset may consist
+of many files in many directories. We can read a single file back with
+``read_table``:
+
+.. ipython:: python
+
+   table2 = pq.read_table('example.parquet')
+   table2.to_pandas()
+
+You can pass a subset of columns to read, which can be much faster than reading
+the whole file (due to the columnar layout):
+
+.. ipython:: python
+
+   pq.read_table('example.parquet', columns=['one', 'three'])
+
+We need not use a string to specify the origin of the file. It can be any of:
+
+* A file path as a string
+* A :ref:`NativeFile <io.native_file>` from PyArrow
+* A Python file object
+
+In general, a Python file object will have the worst read performance, while a
+string file path or an instance of :class:`~.NativeFIle` (especially memory
+maps) will perform the best.
 
-    import pyarrow.parquet as pq
+Finer-grained Reading and Writing
+---------------------------------
 
-    table = pq.read_table('<filename>')
+``read_table`` uses the :class:`~.ParquetFile` class, which has other features:
 
-As DataFrames stored as Parquet are often stored in multiple files, a
-convenience method :meth:`~pyarrow.parquet.read_multiple_files` is provided.
+.. ipython:: python
 
-If you already have the Parquet available in memory or get it via non-file
-source, you can utilize :class:`pyarrow.io.BufferReader` to read it from
-memory. As input to the :class:`~pyarrow.io.BufferReader` you can either supply
-a Python ``bytes`` object or a :class:`pyarrow.io.Buffer`.
+   parquet_file = pq.ParquetFile('example.parquet')
+   parquet_file.metadata
+   parquet_file.schema
 
-.. code:: python
+As you can learn more in the `Apache Parquet format
+<https://github.com/apache/parquet-format>`_, a Parquet file consists of
+multiple row groups. ``read_table`` will read all of the row groups and
+concatenate them into a single table. You can read individual row groups with
+``read_row_group``:
 
-    import pyarrow.io as paio
-    import pyarrow.parquet as pq
+.. ipython:: python
 
-    buf = ... # either bytes or paio.Buffer
-    reader = paio.BufferReader(buf)
-    table = pq.read_table(reader)
+   parquet_file.num_row_groups
+   parquet_file.read_row_group(0)
 
-Writing Parquet
----------------
+We can similarly write a Parquet file with multiple row groups by using
+``ParquetWriter``:
 
-Given an instance of :class:`pyarrow.table.Table`, the most simple way to
-persist it to Parquet is by using the :meth:`pyarrow.parquet.write_table`
-method.
+.. ipython:: python
+
+   writer = pq.ParquetWriter('example2.parquet', table.schema)
+   for i in range(3):
+       writer.write_table(table)
+   writer.close()
+
+   pf2 = pq.ParquetFile('example2.parquet')
+   pf2.num_row_groups
+
+.. ipython:: python
+   :suppress:
+
+   !rm example.parquet
+   !rm example2.parquet
+
+Compression, Encoding, and File Compatibility
+---------------------------------------------
+
+The most commonly used Parquet implementations use dictionary encoding when
+writing files; if the dictionaries grow too large, then they "fall back" to
+plain encoding. Whether dictionary encoding is used can be toggled using the
+``use_dictionary`` option:
 
 .. code-block:: python
 
-    import pyarrow as pa
-    import pyarrow.parquet as pq
+   pq.write_table(table, where, use_dictionary=False)
 
-    table = pa.Table(..)
-    pq.write_table(table, '<filename>')
+The data pages within a column in a row group can be compressed after the
+encoding passes (dictionary, RLE encoding). In PyArrow we use Snappy
+compression by default, but Brotli, Gzip, and uncompressed are also supported:
 
-By default this will write the Table as a single RowGroup using ``DICTIONARY``
-encoding. To increase the potential of parallelism a query engine can process
-a Parquet file, set the ``chunk_size`` to a fraction of the total number of rows.
+.. code-block:: python
+
+   pq.write_table(table, where, compression='snappy')
+   pq.write_table(table, where, compression='gzip')
+   pq.write_table(table, where, compression='brotli')
+   pq.write_table(table, where, compression='none')
+
+Snappy generally results in better performance, while Gzip may yield smaller
+files.
+
+These settings can also be set on a per-column basis:
+
+.. code-block:: python
 
-If you also want to compress the columns, you can select a compression
-method using the ``compression`` argument. Typically, ``GZIP`` is the choice if
-you want to minimize size and ``SNAPPY`` for performance.
+   pa.write_table(table, where, compression={'foo': 'snappy', 'bar': 'gzip'},
+                  use_dictionary=['foo', 'bar'])
 
-Instead of writing to a file, you can also write to Python ``bytes`` by
-utilizing an :class:`pyarrow.io.InMemoryOutputStream()`:
+Reading Multiples Files and Partitioned Datasets
+------------------------------------------------
 
-.. code:: python
+Multiple Parquet files constitute a Parquet *dataset*. These may present in a
+number of ways:
 
-    import pyarrow.io as paio
-    import pyarrow.parquet as pq
+* A list of Parquet absolute file paths
+* A directory name containing nested directories defining a partitioned dataset
+
+A dataset partitioned by year and month may look like on disk:
+
+.. code-block:: text
+
+   dataset_name/
+     year=2007/
+       month=01/
+          0.parq
+          1.parq
+          ...
+       month=02/
+          0.parq
+          1.parq
+          ...
+       month=03/
+       ...
+     year=2008/
+       month=01/
+       ...
+     ...
+
+The :class:`~.ParquetDataset` class accepts either a directory name or a list
+or file paths, and can discover and infer some common partition structures,
+such as those produced by Hive:
+
+.. code-block:: python
+
+   dataset = pq.ParquetDataset('dataset_name/')
+   table = dataset.read()
+
+Multithreaded Reads
+-------------------
+
+Each of the reading functions have an ``nthreads`` argument which will read
+columns with the indicated level of parallelism. Depending on the speed of IO
+and how expensive it is to decode the columns in a particular file
+(particularly with GZIP compression), this can yield significantly higher data
+throughput:
+
+.. code-block:: python
 
-    table = ...
-    output = paio.InMemoryOutputStream()
-    pq.write_table(table, output)
-    pybytes = output.get_result().to_pybytes()
+   pq.read_table(where, nthreads=4)
+   pq.ParquetDataset(where).read(nthreads=4)

http://git-wip-us.apache.org/repos/asf/arrow-site/blob/6360599f/docs/python/_static/contents.png
----------------------------------------------------------------------
diff --git a/docs/python/_static/contents.png b/docs/python/_static/contents.png
new file mode 100644
index 0000000..6c59aa1
Binary files /dev/null and b/docs/python/_static/contents.png differ

http://git-wip-us.apache.org/repos/asf/arrow-site/blob/6360599f/docs/python/_static/navigation.png
----------------------------------------------------------------------
diff --git a/docs/python/_static/navigation.png b/docs/python/_static/navigation.png
new file mode 100644
index 0000000..fda6cd2
Binary files /dev/null and b/docs/python/_static/navigation.png differ

http://git-wip-us.apache.org/repos/asf/arrow-site/blob/6360599f/docs/python/_static/pygments.css
----------------------------------------------------------------------
diff --git a/docs/python/_static/pygments.css b/docs/python/_static/pygments.css
index 8213e90..20c4814 100644
--- a/docs/python/_static/pygments.css
+++ b/docs/python/_static/pygments.css
@@ -47,8 +47,10 @@
 .highlight .mh { color: #208050 } /* Literal.Number.Hex */
 .highlight .mi { color: #208050 } /* Literal.Number.Integer */
 .highlight .mo { color: #208050 } /* Literal.Number.Oct */
+.highlight .sa { color: #4070a0 } /* Literal.String.Affix */
 .highlight .sb { color: #4070a0 } /* Literal.String.Backtick */
 .highlight .sc { color: #4070a0 } /* Literal.String.Char */
+.highlight .dl { color: #4070a0 } /* Literal.String.Delimiter */
 .highlight .sd { color: #4070a0; font-style: italic } /* Literal.String.Doc */
 .highlight .s2 { color: #4070a0 } /* Literal.String.Double */
 .highlight .se { color: #4070a0; font-weight: bold } /* Literal.String.Escape */
@@ -59,7 +61,9 @@
 .highlight .s1 { color: #4070a0 } /* Literal.String.Single */
 .highlight .ss { color: #517918 } /* Literal.String.Symbol */
 .highlight .bp { color: #007020 } /* Name.Builtin.Pseudo */
+.highlight .fm { color: #06287e } /* Name.Function.Magic */
 .highlight .vc { color: #bb60d5 } /* Name.Variable.Class */
 .highlight .vg { color: #bb60d5 } /* Name.Variable.Global */
 .highlight .vi { color: #bb60d5 } /* Name.Variable.Instance */
+.highlight .vm { color: #bb60d5 } /* Name.Variable.Magic */
 .highlight .il { color: #208050 } /* Literal.Number.Integer.Long */
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/arrow-site/blob/6360599f/docs/python/_static/sphinxdoc.css
----------------------------------------------------------------------
diff --git a/docs/python/_static/sphinxdoc.css b/docs/python/_static/sphinxdoc.css
new file mode 100644
index 0000000..8876773
--- /dev/null
+++ b/docs/python/_static/sphinxdoc.css
@@ -0,0 +1,345 @@
+/*
+ * sphinxdoc.css_t
+ * ~~~~~~~~~~~~~~~
+ *
+ * Sphinx stylesheet -- sphinxdoc theme.  Originally created by
+ * Armin Ronacher for Werkzeug.
+ *
+ * :copyright: Copyright 2007-2017 by the Sphinx team, see AUTHORS.
+ * :license: BSD, see LICENSE for details.
+ *
+ */
+
+@import url("basic.css");
+
+/* -- page layout ----------------------------------------------------------- */
+
+body {
+    font-family: 'Lucida Grande', 'Lucida Sans Unicode', 'Geneva',
+                 'Verdana', sans-serif;
+    font-size: 14px;
+    letter-spacing: -0.01em;
+    line-height: 150%;
+    text-align: center;
+    background-color: #BFD1D4;
+    color: black;
+    padding: 0;
+    border: 1px solid #aaa;
+
+    margin: 0px 80px 0px 80px;
+    min-width: 740px;
+}
+
+div.document {
+    background-color: white;
+    text-align: left;
+    background-image: url(contents.png);
+    background-repeat: repeat-x;
+}
+
+div.bodywrapper {
+    margin: 0 240px 0 0;
+    border-right: 1px solid #ccc;
+}
+
+div.body {
+    margin: 0;
+    padding: 0.5em 20px 20px 20px;
+}
+
+div.related {
+    font-size: 1em;
+}
+
+div.related ul {
+    background-image: url(navigation.png);
+    height: 2em;
+    border-top: 1px solid #ddd;
+    border-bottom: 1px solid #ddd;
+}
+
+div.related ul li {
+    margin: 0;
+    padding: 0;
+    height: 2em;
+    float: left;
+}
+
+div.related ul li.right {
+    float: right;
+    margin-right: 5px;
+}
+
+div.related ul li a {
+    margin: 0;
+    padding: 0 5px 0 5px;
+    line-height: 1.75em;
+    color: #EE9816;
+}
+
+div.related ul li a:hover {
+    color: #3CA8E7;
+}
+
+div.sphinxsidebarwrapper {
+    padding: 0;
+}
+
+div.sphinxsidebar {
+    margin: 0;
+    padding: 0.5em 15px 15px 0;
+    width: 210px;
+    float: right;
+    font-size: 1em;
+    text-align: left;
+}
+
+div.sphinxsidebar h3, div.sphinxsidebar h4 {
+    margin: 1em 0 0.5em 0;
+    font-size: 1em;
+    padding: 0.1em 0 0.1em 0.5em;
+    color: white;
+    border: 1px solid #86989B;
+    background-color: #AFC1C4;
+}
+
+div.sphinxsidebar h3 a {
+    color: white;
+}
+
+div.sphinxsidebar ul {
+    padding-left: 1.5em;
+    margin-top: 7px;
+    padding: 0;
+    line-height: 130%;
+}
+
+div.sphinxsidebar ul ul {
+    margin-left: 20px;
+}
+
+div.footer {
+    background-color: #E3EFF1;
+    color: #86989B;
+    padding: 3px 8px 3px 0;
+    clear: both;
+    font-size: 0.8em;
+    text-align: right;
+}
+
+div.footer a {
+    color: #86989B;
+    text-decoration: underline;
+}
+
+/* -- body styles ----------------------------------------------------------- */
+
+p {    
+    margin: 0.8em 0 0.5em 0;
+}
+
+a {
+    color: #CA7900;
+    text-decoration: none;
+}
+
+a:hover {
+    color: #2491CF;
+}
+
+div.body a {
+    text-decoration: underline;
+}
+
+h1 {
+    margin: 0;
+    padding: 0.7em 0 0.3em 0;
+    font-size: 1.5em;
+    color: #11557C;
+}
+
+h2 {
+    margin: 1.3em 0 0.2em 0;
+    font-size: 1.35em;
+    padding: 0;
+}
+
+h3 {
+    margin: 1em 0 -0.3em 0;
+    font-size: 1.2em;
+}
+
+div.body h1 a, div.body h2 a, div.body h3 a, div.body h4 a, div.body h5 a, div.body h6 a {
+    color: black!important;
+}
+
+h1 a.anchor, h2 a.anchor, h3 a.anchor, h4 a.anchor, h5 a.anchor, h6 a.anchor {
+    display: none;
+    margin: 0 0 0 0.3em;
+    padding: 0 0.2em 0 0.2em;
+    color: #aaa!important;
+}
+
+h1:hover a.anchor, h2:hover a.anchor, h3:hover a.anchor, h4:hover a.anchor,
+h5:hover a.anchor, h6:hover a.anchor {
+    display: inline;
+}
+
+h1 a.anchor:hover, h2 a.anchor:hover, h3 a.anchor:hover, h4 a.anchor:hover,
+h5 a.anchor:hover, h6 a.anchor:hover {
+    color: #777;
+    background-color: #eee;
+}
+
+a.headerlink {
+    color: #c60f0f!important;
+    font-size: 1em;
+    margin-left: 6px;
+    padding: 0 4px 0 4px;
+    text-decoration: none!important;
+}
+
+a.headerlink:hover {
+    background-color: #ccc;
+    color: white!important;
+}
+
+cite, code, code {
+    font-family: 'Consolas', 'Deja Vu Sans Mono',
+                 'Bitstream Vera Sans Mono', monospace;
+    font-size: 0.95em;
+    letter-spacing: 0.01em;
+}
+
+code {
+    background-color: #f2f2f2;
+    border-bottom: 1px solid #ddd;
+    color: #333;
+}
+
+code.descname, code.descclassname, code.xref {
+    border: 0;
+}
+
+hr {
+    border: 1px solid #abc;
+    margin: 2em;
+}
+
+a code {
+    border: 0;
+    color: #CA7900;
+}
+
+a code:hover {
+    color: #2491CF;
+}
+
+pre {
+    font-family: 'Consolas', 'Deja Vu Sans Mono',
+                 'Bitstream Vera Sans Mono', monospace;
+    font-size: 0.95em;
+    letter-spacing: 0.015em;
+    line-height: 120%;
+    padding: 0.5em;
+    border: 1px solid #ccc;
+    background-color: #f8f8f8;
+}
+
+pre a {
+    color: inherit;
+    text-decoration: underline;
+}
+
+td.linenos pre {
+    padding: 0.5em 0;
+}
+
+div.quotebar {
+    background-color: #f8f8f8;
+    max-width: 250px;
+    float: right;
+    padding: 2px 7px;
+    border: 1px solid #ccc;
+}
+
+div.topic {
+    background-color: #f8f8f8;
+}
+
+table {
+    border-collapse: collapse;
+    margin: 0 -0.5em 0 -0.5em;
+}
+
+table td, table th {
+    padding: 0.2em 0.5em 0.2em 0.5em;
+}
+
+div.admonition, div.warning {
+    font-size: 0.9em;
+    margin: 1em 0 1em 0;
+    border: 1px solid #86989B;
+    background-color: #f7f7f7;
+    padding: 0;
+}
+
+div.admonition p, div.warning p {
+    margin: 0.5em 1em 0.5em 1em;
+    padding: 0;
+}
+
+div.admonition pre, div.warning pre {
+    margin: 0.4em 1em 0.4em 1em;
+}
+
+div.admonition p.admonition-title,
+div.warning p.admonition-title {
+    margin: 0;
+    padding: 0.1em 0 0.1em 0.5em;
+    color: white;
+    border-bottom: 1px solid #86989B;
+    font-weight: bold;
+    background-color: #AFC1C4;
+}
+
+div.warning {
+    border: 1px solid #940000;
+}
+
+div.warning p.admonition-title {
+    background-color: #CF0000;
+    border-bottom-color: #940000;
+}
+
+div.admonition ul, div.admonition ol,
+div.warning ul, div.warning ol {
+    margin: 0.1em 0.5em 0.5em 3em;
+    padding: 0;
+}
+
+div.versioninfo {
+    margin: 1em 0 0 0;
+    border: 1px solid #ccc;
+    background-color: #DDEAF0;
+    padding: 8px;
+    line-height: 1.3em;
+    font-size: 0.9em;
+}
+
+.viewcode-back {
+    font-family: 'Lucida Grande', 'Lucida Sans Unicode', 'Geneva',
+                 'Verdana', sans-serif;
+}
+
+div.viewcode-block:target {
+    background-color: #f4debf;
+    border-top: 1px solid #ac9;
+    border-bottom: 1px solid #ac9;
+}
+
+div.code-block-caption {
+    background-color: #ddd;
+    color: #222;
+    border: 1px solid #ccc;
+}
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/arrow-site/blob/6360599f/docs/python/api.html
----------------------------------------------------------------------
diff --git a/docs/python/api.html b/docs/python/api.html
index cb0f390..9043e13 100644
--- a/docs/python/api.html
+++ b/docs/python/api.html
@@ -1,318 +1,105 @@
+<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
+  "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
 
 
-<!DOCTYPE html>
-<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
-<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
-<head>
-  <meta charset="utf-8">
-  
-  <meta name="viewport" content="width=device-width, initial-scale=1.0">
-  
-  <title>API Reference &mdash; pyarrow  documentation</title>
-  
-
-  
-  
-  
-  
-
-  
-
-  
-  
+<html xmlns="http://www.w3.org/1999/xhtml">
+  <head>
+    <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
     
-
-  
-
-  
-  
-    <link rel="stylesheet" href="_static/css/theme.css" type="text/css" />
-  
-
-  
-
-  
-        <link rel="index" title="Index"
-              href="genindex.html"/>
-        <link rel="search" title="Search" href="search.html"/>
-    <link rel="top" title="pyarrow  documentation" href="index.html"/>
-        <link rel="next" title="pyarrow.null" href="generated/pyarrow.null.html"/>
-        <link rel="prev" title="Reading/Writing Parquet files" href="parquet.html"/> 
-
-  
-  <script src="_static/js/modernizr.min.js"></script>
-
-</head>
-
-<body class="wy-body-for-nav" role="document">
-
-   
-  <div class="wy-grid-for-nav">
-
+    <title>API Reference &#8212; pyarrow  documentation</title>
     
-    <nav data-toggle="wy-nav-shift" class="wy-nav-side">
-      <div class="wy-side-scroll">
-        <div class="wy-side-nav-search">
-          
-
-          
-            <a href="index.html" class="icon icon-home"> pyarrow
-          
-
-          
-          </a>
-
-          
-            
-            
-          
-
-          
-<div role="search">
-  <form id="rtd-search-form" class="wy-form" action="search.html" method="get">
-    <input type="text" name="q" placeholder="Search docs" />
-    <input type="hidden" name="check_keywords" value="yes" />
-    <input type="hidden" name="area" value="default" />
-  </form>
-</div>
-
-          
-        </div>
-
-        <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
-          
-            
-            
-              
-            
-            
-              <p class="caption"><span class="caption-text">Getting Started</span></p>
-<ul class="current">
-<li class="toctree-l1"><a class="reference internal" href="install.html">Install PyArrow</a></li>
-<li class="toctree-l1"><a class="reference internal" href="pandas.html">Pandas Interface</a></li>
-<li class="toctree-l1"><a class="reference internal" href="filesystems.html">File interfaces and Memory Maps</a></li>
-<li class="toctree-l1"><a class="reference internal" href="parquet.html">Reading/Writing Parquet files</a></li>
-<li class="toctree-l1 current"><a class="current reference internal" href="#">API Reference</a><ul>
-<li class="toctree-l2"><a class="reference internal" href="#type-and-schema-factory-functions">Type and Schema Factory Functions</a><ul>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.null.html">pyarrow.null</a></li>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.bool_.html">pyarrow.bool_</a></li>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.int8.html">pyarrow.int8</a></li>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.int16.html">pyarrow.int16</a></li>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.int32.html">pyarrow.int32</a></li>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.int64.html">pyarrow.int64</a></li>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.uint8.html">pyarrow.uint8</a></li>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.uint16.html">pyarrow.uint16</a></li>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.uint32.html">pyarrow.uint32</a></li>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.uint64.html">pyarrow.uint64</a></li>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.float16.html">pyarrow.float16</a></li>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.float32.html">pyarrow.float32</a></li>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.float64.html">pyarrow.float64</a></li>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.time32.html">pyarrow.time32</a></li>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.time64.html">pyarrow.time64</a></li>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.timestamp.html">pyarrow.timestamp</a></li>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.date32.html">pyarrow.date32</a></li>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.date64.html">pyarrow.date64</a></li>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.binary.html">pyarrow.binary</a></li>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.string.html">pyarrow.string</a></li>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.decimal.html">pyarrow.decimal</a></li>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.list_.html">pyarrow.list_</a></li>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.struct.html">pyarrow.struct</a></li>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.dictionary.html">pyarrow.dictionary</a></li>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.field.html">pyarrow.field</a></li>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.schema.html">pyarrow.schema</a></li>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.from_numpy_dtype.html">pyarrow.from_numpy_dtype</a></li>
-</ul>
-</li>
-<li class="toctree-l2"><a class="reference internal" href="#scalar-value-types">Scalar Value Types</a><ul>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.NA.html">pyarrow.NA</a></li>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.NAType.html">pyarrow.NAType</a></li>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.Scalar.html">pyarrow.Scalar</a></li>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.ArrayValue.html">pyarrow.ArrayValue</a></li>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.BooleanValue.html">pyarrow.BooleanValue</a></li>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.Int8Value.html">pyarrow.Int8Value</a></li>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.Int16Value.html">pyarrow.Int16Value</a></li>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.Int32Value.html">pyarrow.Int32Value</a></li>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.Int64Value.html">pyarrow.Int64Value</a></li>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.UInt8Value.html">pyarrow.UInt8Value</a></li>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.UInt16Value.html">pyarrow.UInt16Value</a></li>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.UInt32Value.html">pyarrow.UInt32Value</a></li>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.UInt64Value.html">pyarrow.UInt64Value</a></li>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.FloatValue.html">pyarrow.FloatValue</a></li>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.DoubleValue.html">pyarrow.DoubleValue</a></li>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.ListValue.html">pyarrow.ListValue</a></li>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.BinaryValue.html">pyarrow.BinaryValue</a></li>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.StringValue.html">pyarrow.StringValue</a></li>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.FixedSizeBinaryValue.html">pyarrow.FixedSizeBinaryValue</a></li>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.Date32Value.html">pyarrow.Date32Value</a></li>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.Date64Value.html">pyarrow.Date64Value</a></li>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.TimestampValue.html">pyarrow.TimestampValue</a></li>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.DecimalValue.html">pyarrow.DecimalValue</a></li>
-</ul>
-</li>
-<li class="toctree-l2"><a class="reference internal" href="#array-types-and-constructors">Array Types and Constructors</a><ul>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.array.html">pyarrow.array</a></li>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.Array.html">pyarrow.Array</a></li>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.BooleanArray.html">pyarrow.BooleanArray</a></li>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.DictionaryArray.html">pyarrow.DictionaryArray</a></li>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.FloatingPointArray.html">pyarrow.FloatingPointArray</a></li>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.IntegerArray.html">pyarrow.IntegerArray</a></li>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.Int8Array.html">pyarrow.Int8Array</a></li>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.Int16Array.html">pyarrow.Int16Array</a></li>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.Int32Array.html">pyarrow.Int32Array</a></li>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.Int64Array.html">pyarrow.Int64Array</a></li>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.NullArray.html">pyarrow.NullArray</a></li>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.NumericArray.html">pyarrow.NumericArray</a></li>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.UInt8Array.html">pyarrow.UInt8Array</a></li>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.UInt16Array.html">pyarrow.UInt16Array</a></li>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.UInt32Array.html">pyarrow.UInt32Array</a></li>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.UInt64Array.html">pyarrow.UInt64Array</a></li>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.BinaryArray.html">pyarrow.BinaryArray</a></li>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.FixedSizeBinaryArray.html">pyarrow.FixedSizeBinaryArray</a></li>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.StringArray.html">pyarrow.StringArray</a></li>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.Time32Array.html">pyarrow.Time32Array</a></li>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.Time64Array.html">pyarrow.Time64Array</a></li>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.Date32Array.html">pyarrow.Date32Array</a></li>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.Date64Array.html">pyarrow.Date64Array</a></li>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.TimestampArray.html">pyarrow.TimestampArray</a></li>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.DecimalArray.html">pyarrow.DecimalArray</a></li>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.ListArray.html">pyarrow.ListArray</a></li>
-</ul>
-</li>
-<li class="toctree-l2"><a class="reference internal" href="#tables-and-record-batches">Tables and Record Batches</a><ul>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.ChunkedArray.html">pyarrow.ChunkedArray</a></li>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.Column.html">pyarrow.Column</a></li>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.RecordBatch.html">pyarrow.RecordBatch</a></li>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.Table.html">pyarrow.Table</a></li>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.get_record_batch_size.html">pyarrow.get_record_batch_size</a></li>
-</ul>
-</li>
-<li class="toctree-l2"><a class="reference internal" href="#tensor-type-and-functions">Tensor type and Functions</a><ul>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.Tensor.html">pyarrow.Tensor</a></li>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.write_tensor.html">pyarrow.write_tensor</a></li>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.get_tensor_size.html">pyarrow.get_tensor_size</a></li>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.read_tensor.html">pyarrow.read_tensor</a></li>
-</ul>
-</li>
-<li class="toctree-l2"><a class="reference internal" href="#input-output-and-shared-memory">Input / Output and Shared Memory</a><ul>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.Buffer.html">pyarrow.Buffer</a></li>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.BufferReader.html">pyarrow.BufferReader</a></li>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.InMemoryOutputStream.html">pyarrow.InMemoryOutputStream</a></li>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.NativeFile.html">pyarrow.NativeFile</a></li>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.MemoryMappedFile.html">pyarrow.MemoryMappedFile</a></li>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.memory_map.html">pyarrow.memory_map</a></li>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.create_memory_map.html">pyarrow.create_memory_map</a></li>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.PythonFile.html">pyarrow.PythonFile</a></li>
-</ul>
-</li>
-<li class="toctree-l2"><a class="reference internal" href="#interprocess-communication-and-messaging">Interprocess Communication and Messaging</a><ul>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.FileReader.html">pyarrow.FileReader</a></li>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.FileWriter.html">pyarrow.FileWriter</a></li>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.StreamReader.html">pyarrow.StreamReader</a></li>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.StreamWriter.html">pyarrow.StreamWriter</a></li>
-</ul>
-</li>
-<li class="toctree-l2"><a class="reference internal" href="#memory-pools">Memory Pools</a><ul>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.MemoryPool.html">pyarrow.MemoryPool</a></li>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.default_memory_pool.html">pyarrow.default_memory_pool</a></li>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.jemalloc_memory_pool.html">pyarrow.jemalloc_memory_pool</a></li>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.total_allocated_bytes.html">pyarrow.total_allocated_bytes</a></li>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.set_memory_pool.html">pyarrow.set_memory_pool</a></li>
-</ul>
-</li>
-<li class="toctree-l2"><a class="reference internal" href="#type-classes">Type Classes</a><ul>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.DataType.html">pyarrow.DataType</a></li>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.DecimalType.html">pyarrow.DecimalType</a></li>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.DictionaryType.html">pyarrow.DictionaryType</a></li>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.FixedSizeBinaryType.html">pyarrow.FixedSizeBinaryType</a></li>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.Time32Type.html">pyarrow.Time32Type</a></li>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.Time64Type.html">pyarrow.Time64Type</a></li>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.TimestampType.html">pyarrow.TimestampType</a></li>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.Field.html">pyarrow.Field</a></li>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.Schema.html">pyarrow.Schema</a></li>
-</ul>
-</li>
-<li class="toctree-l2"><a class="reference internal" href="#apache-parquet">Apache Parquet</a><ul>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.parquet.ParquetDataset.html">pyarrow.parquet.ParquetDataset</a></li>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.parquet.ParquetFile.html">pyarrow.parquet.ParquetFile</a></li>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.parquet.read_table.html">pyarrow.parquet.read_table</a></li>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.parquet.write_metadata.html">pyarrow.parquet.write_metadata</a></li>
-<li class="toctree-l3"><a class="reference internal" href="generated/pyarrow.parquet.write_table.html">pyarrow.parquet.write_table</a></li>
-</ul>
-</li>
+    <link rel="stylesheet" href="_static/sphinxdoc.css" type="text/css" />
+    <link rel="stylesheet" href="_static/pygments.css" type="text/css" />
+    
+    <script type="text/javascript">
+      var DOCUMENTATION_OPTIONS = {
+        URL_ROOT:    './',
+        VERSION:     '',
+        COLLAPSE_INDEX: false,
+        FILE_SUFFIX: '.html',
+        HAS_SOURCE:  true,
+        SOURCELINK_SUFFIX: '.txt'
+      };
+    </script>
+    <script type="text/javascript" src="_static/jquery.js"></script>
+    <script type="text/javascript" src="_static/underscore.js"></script>
+    <script type="text/javascript" src="_static/doctools.js"></script>
+    <script type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.0/MathJax.js?config=TeX-AMS-MML_HTMLorMML"></script>
+    <link rel="index" title="Index" href="genindex.html" />
+    <link rel="search" title="Search" href="search.html" />
+    <link rel="next" title="pyarrow.null" href="generated/pyarrow.null.html" />
+    <link rel="prev" title="Reading and Writing the Apache Parquet Format" href="parquet.html" /> 
+  </head>
+  <body role="document">
+    <div class="related" role="navigation" aria-label="related navigation">
+      <h3>Navigation</h3>
+      <ul>
+        <li class="right" style="margin-right: 10px">
+          <a href="genindex.html" title="General Index"
+             accesskey="I">index</a></li>
+        <li class="right" >
+          <a href="generated/pyarrow.null.html" title="pyarrow.null"
+             accesskey="N">next</a> |</li>
+        <li class="right" >
+          <a href="parquet.html" title="Reading and Writing the Apache Parquet Format"
+             accesskey="P">previous</a> |</li>
+        <li class="nav-item nav-item-0"><a href="index.html">pyarrow  documentation</a> &#187;</li> 
+      </ul>
+    </div>
+      <div class="sphinxsidebar" role="navigation" aria-label="main navigation">
+        <div class="sphinxsidebarwrapper">
+  <h3><a href="index.html">Table Of Contents</a></h3>
+  <ul>
+<li><a class="reference internal" href="#">API Reference</a><ul>
+<li><a class="reference internal" href="#type-and-schema-factory-functions">Type and Schema Factory Functions</a></li>
+<li><a class="reference internal" href="#scalar-value-types">Scalar Value Types</a></li>
+<li><a class="reference internal" href="#array-types-and-constructors">Array Types and Constructors</a></li>
+<li><a class="reference internal" href="#tables-and-record-batches">Tables and Record Batches</a></li>
+<li><a class="reference internal" href="#tensor-type-and-functions">Tensor type and Functions</a></li>
+<li><a class="reference internal" href="#input-output-and-shared-memory">Input / Output and Shared Memory</a></li>
+<li><a class="reference internal" href="#interprocess-communication-and-messaging">Interprocess Communication and Messaging</a></li>
+<li><a class="reference internal" href="#memory-pools">Memory Pools</a></li>
+<li><a class="reference internal" href="#type-classes">Type Classes</a></li>
+<li><a class="reference internal" href="#apache-parquet">Apache Parquet</a></li>
 </ul>
 </li>
-<li class="toctree-l1"><a class="reference internal" href="getting_involved.html">Getting Involved</a></li>
-</ul>
-<p class="caption"><span class="caption-text">Additional Features</span></p>
-<ul>
-<li class="toctree-l1"><a class="reference internal" href="jemalloc.html">jemalloc MemoryPool</a></li>
 </ul>
 
-            
-          
+  <h4>Previous topic</h4>
+  <p class="topless"><a href="parquet.html"
+                        title="previous chapter">Reading and Writing the Apache Parquet Format</a></p>
+  <h4>Next topic</h4>
+  <p class="topless"><a href="generated/pyarrow.null.html"
+                        title="next chapter">pyarrow.null</a></p>
+  <div role="note" aria-label="source link">
+    <h3>This Page</h3>
+    <ul class="this-page-menu">
+      <li><a href="_sources/api.rst.txt"
+            rel="nofollow">Show Source</a></li>
+    </ul>
+   </div>
+<div id="searchbox" style="display: none" role="search">
+  <h3>Quick search</h3>
+    <form class="search" action="search.html" method="get">
+      <div><input type="text" name="q" /></div>
+      <div><input type="submit" value="Go" /></div>
+      <input type="hidden" name="check_keywords" value="yes" />
+      <input type="hidden" name="area" value="default" />
+    </form>
+</div>
+<script type="text/javascript">$('#searchbox').show(0);</script>
         </div>
       </div>
-    </nav>
-
-    <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
-
-      
-      <nav class="wy-nav-top" role="navigation" aria-label="top navigation">
-        
-          <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
-          <a href="index.html">pyarrow</a>
-        
-      </nav>
-
-
-      
-      <div class="wy-nav-content">
-        <div class="rst-content">
-          
-
-
-
-
-
-
-
-
 
-
-
-
-
-
-
-<div role="navigation" aria-label="breadcrumbs navigation">
-
-  <ul class="wy-breadcrumbs">
-    
-      <li><a href="index.html">Docs</a> &raquo;</li>
-        
-      <li>API Reference</li>
-    
-    
-      <li class="wy-breadcrumbs-aside">
-        
-            
-            <a href="_sources/api.rst.txt" rel="nofollow"> View page source</a>
-          
-        
-      </li>
-    
-  </ul>
-
-  
-  <hr/>
-</div>
-          <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
-           <div itemprop="articleBody">
+    <div class="document">
+      <div class="documentwrapper">
+        <div class="bodywrapper">
+          <div class="body" role="main">
             
   <div class="section" id="api-reference">
 <span id="api"></span><h1>API Reference<a class="headerlink" href="#api-reference" title="Permalink to this headline">¶</a></h1>
 <div class="section" id="type-and-schema-factory-functions">
-<span id="api-functions"></span><h2>Type and Schema Factory Functions<a class="headerlink" href="#type-and-schema-factory-functions" title="Permalink to this headline">¶</a></h2>
+<span id="api-types"></span><h2>Type and Schema Factory Functions<a class="headerlink" href="#type-and-schema-factory-functions" title="Permalink to this headline">¶</a></h2>
 <table border="1" class="longtable docutils">
 <colgroup>
 <col width="10%" />
@@ -391,11 +178,11 @@
 <tr class="row-even"><td><a class="reference internal" href="generated/pyarrow.dictionary.html#pyarrow.dictionary" title="pyarrow.dictionary"><code class="xref py py-obj docutils literal"><span class="pre">dictionary</span></code></a>(DataType&nbsp;index_type,&nbsp;Array&nbsp;dictionary)</td>
 <td>Dictionary (categorical, or simply encoded) type</td>
 </tr>
-<tr class="row-odd"><td><a class="reference internal" href="generated/pyarrow.field.html#pyarrow.field" title="pyarrow.field"><code class="xref py py-obj docutils literal"><span class="pre">field</span></code></a>(name,&nbsp;type,&nbsp;bool&nbsp;nullable=True)</td>
-<td></td>
+<tr class="row-odd"><td><a class="reference internal" href="generated/pyarrow.field.html#pyarrow.field" title="pyarrow.field"><code class="xref py py-obj docutils literal"><span class="pre">field</span></code></a>(name,&nbsp;DataType&nbsp;type,&nbsp;...)</td>
+<td>Create a pyarrow.Field instance</td>
 </tr>
 <tr class="row-even"><td><a class="reference internal" href="generated/pyarrow.schema.html#pyarrow.schema" title="pyarrow.schema"><code class="xref py py-obj docutils literal"><span class="pre">schema</span></code></a>(fields)</td>
-<td></td>
+<td>Construct pyarrow.Schema from collection of fields</td>
 </tr>
 <tr class="row-odd"><td><a class="reference internal" href="generated/pyarrow.from_numpy_dtype.html#pyarrow.from_numpy_dtype" title="pyarrow.from_numpy_dtype"><code class="xref py py-obj docutils literal"><span class="pre">from_numpy_dtype</span></code></a>(dtype)</td>
 <td>Convert NumPy dtype to pyarrow.DataType</td>
@@ -404,7 +191,7 @@
 </table>
 </div>
 <div class="section" id="scalar-value-types">
-<h2>Scalar Value Types<a class="headerlink" href="#scalar-value-types" title="Permalink to this headline">¶</a></h2>
+<span id="api-value"></span><h2>Scalar Value Types<a class="headerlink" href="#scalar-value-types" title="Permalink to this headline">¶</a></h2>
 <table border="1" class="longtable docutils">
 <colgroup>
 <col width="10%" />
@@ -484,7 +271,7 @@
 </table>
 </div>
 <div class="section" id="array-types-and-constructors">
-<h2>Array Types and Constructors<a class="headerlink" href="#array-types-and-constructors" title="Permalink to this headline">¶</a></h2>
+<span id="api-array"></span><h2>Array Types and Constructors<a class="headerlink" href="#array-types-and-constructors" title="Permalink to this headline">¶</a></h2>
 <table border="1" class="longtable docutils">
 <colgroup>
 <col width="10%" />
@@ -573,7 +360,7 @@
 </table>
 </div>
 <div class="section" id="tables-and-record-batches">
-<h2>Tables and Record Batches<a class="headerlink" href="#tables-and-record-batches" title="Permalink to this headline">¶</a></h2>
+<span id="api-table"></span><h2>Tables and Record Batches<a class="headerlink" href="#tables-and-record-batches" title="Permalink to this headline">¶</a></h2>
 <table border="1" class="longtable docutils">
 <colgroup>
 <col width="10%" />
@@ -599,7 +386,7 @@
 </table>
 </div>
 <div class="section" id="tensor-type-and-functions">
-<h2>Tensor type and Functions<a class="headerlink" href="#tensor-type-and-functions" title="Permalink to this headline">¶</a></h2>
+<span id="api-tensor"></span><h2>Tensor type and Functions<a class="headerlink" href="#tensor-type-and-functions" title="Permalink to this headline">¶</a></h2>
 <table border="1" class="longtable docutils">
 <colgroup>
 <col width="10%" />
@@ -622,7 +409,7 @@
 </table>
 </div>
 <div class="section" id="input-output-and-shared-memory">
-<h2>Input / Output and Shared Memory<a class="headerlink" href="#input-output-and-shared-memory" title="Permalink to this headline">¶</a></h2>
+<span id="api-io"></span><h2>Input / Output and Shared Memory<a class="headerlink" href="#input-output-and-shared-memory" title="Permalink to this headline">¶</a></h2>
 <table border="1" class="longtable docutils">
 <colgroup>
 <col width="10%" />
@@ -657,7 +444,7 @@
 </table>
 </div>
 <div class="section" id="interprocess-communication-and-messaging">
-<h2>Interprocess Communication and Messaging<a class="headerlink" href="#interprocess-communication-and-messaging" title="Permalink to this headline">¶</a></h2>
+<span id="api-ipc"></span><h2>Interprocess Communication and Messaging<a class="headerlink" href="#interprocess-communication-and-messaging" title="Permalink to this headline">¶</a></h2>
 <table border="1" class="longtable docutils">
 <colgroup>
 <col width="10%" />
@@ -680,7 +467,7 @@
 </table>
 </div>
 <div class="section" id="memory-pools">
-<h2>Memory Pools<a class="headerlink" href="#memory-pools" title="Permalink to this headline">¶</a></h2>
+<span id="api-memory-pool"></span><h2>Memory Pools<a class="headerlink" href="#memory-pools" title="Permalink to this headline">¶</a></h2>
 <table border="1" class="longtable docutils">
 <colgroup>
 <col width="10%" />
@@ -706,7 +493,7 @@
 </table>
 </div>
 <div class="section" id="type-classes">
-<h2>Type Classes<a class="headerlink" href="#type-classes" title="Permalink to this headline">¶</a></h2>
+<span id="api-type-classes"></span><h2>Type Classes<a class="headerlink" href="#type-classes" title="Permalink to this headline">¶</a></h2>
 <table border="1" class="longtable docutils">
 <colgroup>
 <col width="10%" />
@@ -735,7 +522,7 @@
 <td></td>
 </tr>
 <tr class="row-even"><td><a class="reference internal" href="generated/pyarrow.Field.html#pyarrow.Field" title="pyarrow.Field"><code class="xref py py-obj docutils literal"><span class="pre">Field</span></code></a></td>
-<td></td>
+<td>Represents a named field, with a data type, nullability, and optional</td>
 </tr>
 <tr class="row-odd"><td><a class="reference internal" href="generated/pyarrow.Schema.html#pyarrow.Schema" title="pyarrow.Schema"><code class="xref py py-obj docutils literal"><span class="pre">Schema</span></code></a></td>
 <td></td>
@@ -744,7 +531,7 @@
 </table>
 </div>
 <div class="section" id="apache-parquet">
-<h2>Apache Parquet<a class="headerlink" href="#apache-parquet" title="Permalink to this headline">¶</a></h2>
+<span id="api-parquet"></span><h2>Apache Parquet<a class="headerlink" href="#apache-parquet" title="Permalink to this headline">¶</a></h2>
 <table border="1" class="longtable docutils">
 <colgroup>
 <col width="10%" />
@@ -772,76 +559,29 @@
 </div>
 
 
-           </div>
-           <div class="articleComments">
-            
-           </div>
           </div>
-          <footer>
-  
-    <div class="rst-footer-buttons" role="navigation" aria-label="footer navigation">
-      
-        <a href="generated/pyarrow.null.html" class="btn btn-neutral float-right" title="pyarrow.null" accesskey="n" rel="next">Next <span class="fa fa-arrow-circle-right"></span></a>
-      
-      
-        <a href="parquet.html" class="btn btn-neutral" title="Reading/Writing Parquet files" accesskey="p" rel="prev"><span class="fa fa-arrow-circle-left"></span> Previous</a>
-      
-    </div>
-  
-
-  <hr/>
-
-  <div role="contentinfo">
-    <p>
-        &copy; Copyright 2016 Apache Software Foundation.
-
-    </p>
-  </div>
-  Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>. 
-
-</footer>
-
         </div>
       </div>
-
-    </section>
-
-  </div>
-  
-
-
-  
-
-    <script type="text/javascript">
-        var DOCUMENTATION_OPTIONS = {
-            URL_ROOT:'./',
-            VERSION:'',
-            COLLAPSE_INDEX:false,
-            FILE_SUFFIX:'.html',
-            HAS_SOURCE:  true,
-            SOURCELINK_SUFFIX: '.txt'
-        };
-    </script>
-      <script type="text/javascript" src="_static/jquery.js"></script>
-      <script type="text/javascript" src="_static/underscore.js"></script>
-      <script type="text/javascript" src="_static/doctools.js"></script>
-      <script type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.0/MathJax.js?config=TeX-AMS-MML_HTMLorMML"></script>
-
-  
-
-  
-  
-    <script type="text/javascript" src="_static/js/theme.js"></script>
-  
-
-  
-  
-  <script type="text/javascript">
-      jQuery(function () {
-          SphinxRtdTheme.StickyNav.enable();
-      });
-  </script>
-   
-
-</body>
+      <div class="clearer"></div>
+    </div>
+    <div class="related" role="navigation" aria-label="related navigation">
+      <h3>Navigation</h3>
+      <ul>
+        <li class="right" style="margin-right: 10px">
+          <a href="genindex.html" title="General Index"
+             >index</a></li>
+        <li class="right" >
+          <a href="generated/pyarrow.null.html" title="pyarrow.null"
+             >next</a> |</li>
+        <li class="right" >
+          <a href="parquet.html" title="Reading and Writing the Apache Parquet Format"
+             >previous</a> |</li>
+        <li class="nav-item nav-item-0"><a href="index.html">pyarrow  documentation</a> &#187;</li> 
+      </ul>
+    </div>
+    <div class="footer" role="contentinfo">
+        &#169; Copyright 2016-2017 Apache Software Foundation.
+      Created using <a href="http://sphinx-doc.org/">Sphinx</a> 1.5.5.
+    </div>
+  </body>
 </html>
\ No newline at end of file


Mime
View raw message