arrow-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Teh, Kenneth M." <...@anl.gov>
Subject [python] not an arrow file
Date Sun, 24 Jan 2021 14:10:42 GMT
Just started with arrow...

I wrote a record batch to a file using ipc::MakeFileWriter to create a writer and writer->WriteRecordBatch
in a C++ program and tried to read it in python with:

[] import pyarrow as pa
[] reader = pa.ipc.open_file("myfile")


It raises the ArrowInvalid with the message "not an arrow file".

If I write it out as a Table in feather format, I can read it in python. But I want to write
large files on the order of 100GB or more and then read them back into python as pandas dataframes
or something similar.

So, I switched to using an ipc writer.

Can something point me in the right direction?  Thanks.

Ken

Mime
View raw message