arrow-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Micah Kornfield <emkornfi...@gmail.com>
Subject Re: Arrow-C++ "Zero-Copy-Append" to BinaryArray
Date Fri, 19 Jun 2020 03:39:50 GMT
Hi Simon,
I don't think there is a public API for this in C++.  You would have to
presize a values buffer to the size expected for the compressed data, have
the compressor output directly to that buffer while recording the necessary
offsets.  You could then construct the BinaryArray directly with these
buffers (I would need to double check, but you might need to construct an
intermediate ArrayData object).

Hope this helps.

Micah



On Thursday, June 18, 2020, Simon Dumke <simon.dumke@ipp.mpg.de> wrote:

> Hi all,
>
> I would like build RecordBatches with (besides others) a BinaryArray
> column containing compressed data. when filling the BinaryArray, i would
> like to allow the compresseor to immediately output into the Arrow Buffer
> instead of allocating an output buffer and then copying the data into Arrow
> Buffers.
>
> Is such an approach possible? And if so - how do I achieve this?
>
> I'd be thankfull for any insights!
>
> Best regards,
>
> Simon
>
>

Mime
View raw message