arrow-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Patrick Kalita <patr...@axiomalaska.com>
Subject [JS] Error while loading 6GB arrow file
Date Mon, 12 Apr 2021 20:59:35 GMT
Hi, I'm running into an error while attempting to load an arrow file. `ls`
lists the size of the file as 6347031002 bytes. Most examples I see online
use `fs.readFileSync` and pass the result to `Table.from`. Since you can't
`readFileSync` (or `readFile` for that matter) a file larger than 2GB, I'm
doing something like this instead:

const stream = fs.createReadStream('data.arrow')
const table = Table.from(stream)

That seems to work for smaller files, but with the 6GB file, I hit the
following error:

(node:17465) UnhandledPromiseRejectionWarning: RangeError: Invalid typed
array length: 6347031002
    at new Uint8Array (<anonymous>)
    at joinUint8Arrays
(..../node_modules/@apache-arrow/es5-esm/util/buffer.js:79:36)
    at byteRange
(..../node_modules/@apache-arrow/es5-esm/io/adapters.js:437:85)
    at ..../node_modules/@apache-arrow/es5-esm/io/adapters.js:537:73
    at step (..../node_modules/tslib/tslib.js:141:27)
    at Object.next (..../node_modules/tslib/tslib.js:122:57)
    at resume (..../node_modules/tslib/tslib.js:208:48)
    at fulfill (..../node_modules/tslib/tslib.js:210:35)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:93:5)

Does that imply that the largest arrow file I can load is limited by the
largest Uint8Array that can be made (2^32, per
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Errors/Invalid_array_length)?
Or is there a better way that I should be loading this file?

Thanks,
Patrick

Mime
View raw message