arrow-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Antoine Pitrou <anto...@python.org>
Subject Re: [Python] testing custom pyarrow.fs filesystems
Date Mon, 21 Jun 2021 09:43:30 GMT

Hello Bruce,

On Fri, 18 Jun 2021 12:08:39 +0000
"Badger, Bruce" <bruce.badger@bofa.com> wrote:
> Dear pyarrow.fs team,
> 
> We are implementing a custom pyarrow.fs filesystem to map the contents of internal file
stores as filesystems for use in Arrow.
> 
> We have a suite of unit tests which exercise the internal parts of our implementation,
and we can run pyarrow.tests.test_fs to ensure that the supplied pyarrow.fs filesystems work
as we have them installed.
> 
> I would like to include our custom filesystem as a sibling of the included pyarrow.fs
filesystems in the pyarrow.tests.test_fs tests in order to ensure that our filesystem conforms
to the expectations of the pyarrow.fs implementation, and continues to conform as pyarrow.fs
and our internal systems evolve.
> 
> Are the pyarrow.tests.test_fs tests extensible to allow the testing of custom filesystems
in addition to, and as peer of, the supplied filesystems?  If so, how is this intended to
work?  ... and if not, may I suggest that this be added as a feature for the (hopefully not
too distant) future?

I don't think we intend to make the PyArrow test suite extensible.  It
is a test suite for PyArrow, not for third-party libraries.

That said, it's probably easy to take those tests and copy/adapt them
inside your own project.

Be aware, however, that most tests for the PyArrow filesystems are
written in C++.  The Python tests do not intend to cover all
functionality in detail.

Regards

Antoine.



Mime
View raw message