arrow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From w...@apache.org
Subject arrow git commit: ARROW-1338: [Python] Do not close RecordBatchWriter on dealloc in case sink is no longer valid
Date Tue, 08 Aug 2017 13:30:21 GMT
Repository: arrow
Updated Branches:
  refs/heads/master 5281a8264 -> 20cee707c


ARROW-1338: [Python] Do not close RecordBatchWriter on dealloc in case sink is no longer valid

Also add missing close() statements to test_mock_output_stream to fix invalid writes causing
core dump on OS X.

Author: Wes McKinney <wes.mckinney@twosigma.com>

Closes #952 from wesm/ARROW-1338 and squashes the following commits:

88e8cefe [Wes McKinney] Do not close RecordBatchWriter on dealloc in case sink is no longer
valid. Add missing close() statements to test_mock_output_stream


Project: http://git-wip-us.apache.org/repos/asf/arrow/repo
Commit: http://git-wip-us.apache.org/repos/asf/arrow/commit/20cee707
Tree: http://git-wip-us.apache.org/repos/asf/arrow/tree/20cee707
Diff: http://git-wip-us.apache.org/repos/asf/arrow/diff/20cee707

Branch: refs/heads/master
Commit: 20cee707cbfdaa5dc4f2b7dea09619f34a1f9f71
Parents: 5281a82
Author: Wes McKinney <wes.mckinney@twosigma.com>
Authored: Tue Aug 8 09:30:16 2017 -0400
Committer: Wes McKinney <wes.mckinney@twosigma.com>
Committed: Tue Aug 8 09:30:16 2017 -0400

----------------------------------------------------------------------
 python/pyarrow/ipc.pxi          | 16 ++++++++++++++--
 python/pyarrow/tests/test_io.py |  2 ++
 2 files changed, 16 insertions(+), 2 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/arrow/blob/20cee707/python/pyarrow/ipc.pxi
----------------------------------------------------------------------
diff --git a/python/pyarrow/ipc.pxi b/python/pyarrow/ipc.pxi
index d6f62aa..ceed4b0 100644
--- a/python/pyarrow/ipc.pxi
+++ b/python/pyarrow/ipc.pxi
@@ -163,8 +163,7 @@ cdef class _RecordBatchWriter:
         self.closed = True
 
     def __dealloc__(self):
-        if not self.closed:
-            self.close()
+        pass
 
     def _open(self, sink, Schema schema):
         cdef:
@@ -182,11 +181,24 @@ cdef class _RecordBatchWriter:
         self.closed = False
 
     def write_batch(self, RecordBatch batch):
+        """
+        Write RecordBatch to stream
+
+        Parameters
+        ----------
+        batch : RecordBatch
+        """
         with nogil:
             check_status(self.writer.get()
                          .WriteRecordBatch(deref(batch.batch)))
 
     def close(self):
+        """
+        Close stream and write end-of-stream 0 marker
+        """
+        if self.closed:
+            return
+
         with nogil:
             check_status(self.writer.get().Close())
         self.closed = True

http://git-wip-us.apache.org/repos/asf/arrow/blob/20cee707/python/pyarrow/tests/test_io.py
----------------------------------------------------------------------
diff --git a/python/pyarrow/tests/test_io.py b/python/pyarrow/tests/test_io.py
index c81a048..d503ea2 100644
--- a/python/pyarrow/tests/test_io.py
+++ b/python/pyarrow/tests/test_io.py
@@ -277,6 +277,8 @@ def test_mock_output_stream():
 
     stream_writer1.write_batch(record_batch)
     stream_writer2.write_batch(record_batch)
+    stream_writer1.close()
+    stream_writer2.close()
 
     assert f1.size() == len(f2.get_result())
 


Mime
View raw message