arrow-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sutou Kouhei <...@clear-code.com>
Subject Re: Snappy Compression with red-parquet Ruby Gem
Date Thu, 23 Apr 2020 21:48:16 GMT
Hi,

Oh, we forgot to integrate saver interface with the Parquet
compression option.

You can use the feature by the following code with 0.17.0:

--
require "parquet"

table = Arrow::Table.new({"count" => [1, 2, 3]})
Arrow::FileOutputStream.open("test.parquet", false) do |output|
  properties = Parquet::WriterProperties.new
  properties.set_compression(:snappy)
  Parquet::ArrowFileWriter.open(table.schema, output, properties) do |writer|
    chunk_size = 1024
    writer.write_table(table, chunk_size)
  end
end
--

You'll be able to write the following code with the next release:

--
require "parquet"

table = Arrow::Table.new({"count" => [1, 2, 3]})
table.save("test.parquet", compression: :snappy)
--


Thanks,
--
kou

In <78B1B196-4217-4526-B848-FE126EDB27B2@contoso.com>
  "Snappy Compression with red-parquet Ruby Gem" on Thu, 23 Apr 2020 20:13:25 +0000,
  David Lahn <david.lahn@forwardpmx.com> wrote:

> Hi,
> 
> Does anyone have any examples of how to output a Parquet file with Snappy compression
using the Ruby gem?
> 
> We have tested trying to set compression to “snappy” on the TableSaver, but we get
the following:
> 
> [compressed-output-stream][new]: NotImplemented: Streaming compression unsupported with
Snappy (Arrow::Error::NotImplemented)
> 
> Example:
> 
> Arrow::TableSaver.new(table, 'test.parquet', {compression: 'snappy'}).save
> 
> Or are we completely turned around on how to accomplish this?
> 
> Dave
>     
> David Lahn
> DevOps Lead
> Development
>        
> ForwardPMX 
> Privacy Policy
> 
>  
>   
> 
> This e-mail is confidential to ForwardPMX intended for use by the recipient. If you received
this in error or are not the intended recipient, you are hereby notified that any review,
retransmission, copying or other use of, or taking of any action in reliance upon this information
is strictly prohibited.
> 

Mime
View raw message