arrow-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Wes McKinney <wesmck...@gmail.com>
Subject Re: Compressing parquet metadata?
Date Wed, 04 Nov 2020 16:41:00 GMT
 You mean the key-value metadata at the schema/field-level? That can
be binary (it gets base64-encoded when written to Parquet)

On Wed, Nov 4, 2020 at 10:22 AM Jason Sachs <jmsachs@gmail.com> wrote:
>
> OK. If I take the manual approach, do parquet / arrow care whether metadata is binary
or not?
>
> On 2020/11/04 14:16:37, Wes McKinney <wesmckinn@gmail.com> wrote:
> > There is not to my knowledge.
> >
> > On Tue, Nov 3, 2020 at 5:55 PM Jason Sachs <jmsachs@gmail.com> wrote:
> > >
> > > Is there any built-in method to compress parquet metadata? From what I can
tell, the main table columns are compressed, but not the metadata.
> > >
> > > I have metadata which includes 100-200KB of text (JSON format) that is easily
compressible... is there any alternative to doing it myself?
> >

Mime
View raw message