arrow-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yakov Galka <>
Subject How to create a Dictionary encoded string column with the Javascript interface?
Date Fri, 17 Jul 2020 01:47:01 GMT
Hi All,

I have code that creates a table with string columns as follows:

for(/* each column */) {
    // ...
    column_vectors.push( Utf8(), 0, element_count,
null_count, nullmap_buffer, offsets_buffer, data_buffer)));
const arrow_table =, column_names);
const data = arrow_table.serialize('binary', false).buffer;
const arrow_table2 = Table.from([new Uint8Array(data)]);

Here offsets_buffer is a Int32Array with the offsets and data_buffer is a
Uint8Array with the strings, in accordance to the Arrow format described in

I am trying to change this to use a dictionary encoding instead. I change
the producer of the data to return only the unique strings in data_buffer
and offsets_buffer, and additionally produce an interned_buffer
(Int32Array) with the indices of the strings. However I couldn't find how
to initialize the column in Javascript.

Shooting in the dark, I tried:

for(/* each column */) {
    // ...
    const dictionary = Utf8(), 0,
offsets_buffer.length - 1, 0, 0, offsets_buffer, data_buffer));
    column_vectors.push( Dictionary(new
Utf8(), new Int32()), 0, element_count, null_count, nullmap_buffer, 0,
interned_buffer, dictionary)));
// ...

However, this causes the deserialization (Table.from) to fail with:

TypeError: undefined has no properties

What's the correct way of creating a dictionary encoded column?

Yakov Galka

View raw message