arrow-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tanveer Ahmad - EWI <>
Subject plasma client output buffer conversion to table through parquet routines issue.
Date Wed, 26 Sep 2018 13:06:36 GMT
Hi all,

I can convert my arrow table to buffer and can read back/convert that buffer to table thorough
below parquet routines. 
But getting corrupt footer error when trying to convert plasma stored output buffer to table
using  parquet routines.

I have an arrow table as "my_arrow_table",

auto sink = std::make_shared<parquet::InMemoryOutputStream>();
const std::shared_ptr<parquet::arrow::ArrowWriterProperties>& arrow_properties =
WriteTable(*my_arrow_table, ::arrow::default_memory_pool(), sink, arrow_alignments_table->num_columns(),
parquet::default_writer_properties(), arrow_properties);
std::shared_ptr<arrow::Buffer> table_data = sink->GetBuffer();

std::string metadata = "Table";
client.Create(object_id, table_data->size(), (uint8_t*), metadata.size(),

ObjectBuffer object_buffer;
ARROW_CHECK_OK(client1.Get(&object_id1, 1, -1, &object_buffer));

// Retrieve object data.
auto outbuffer =;
std::shared_ptr<arrow::Buffer> plasmaData = outbuffer;

auto reader = parquet::ParquetFileReader::Open(std::make_shared<::arrow::io::BufferReader>(plasmaData));
parquet::arrow::FileReader filereader(::arrow::default_memory_pool(), std::move(reader));
std::shared_ptr<::arrow::Table> table;

I'm getting following error:

terminate called after throwing an instance of 'parquet::ParquetException'
  what():  Invalid parquet file. Corrupt footer.
Aborted (core dumped)

Even the outbuffer size is being shown same as without using Plasma object store/get.

Any help in this regard will be highly appreciated. Thanks.

Tanveer Ahmad

View raw message