arrow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wes McKinney (JIRA)" <>
Subject [jira] [Commented] (ARROW-243) Use generic HDFS component instead of libhdfs
Date Tue, 26 Jul 2016 22:23:21 GMT


Wes McKinney commented on ARROW-243:

My preference would be to add an option to {{HdfsConnectionConfig}} to choose the client library
used -- to be loaded at runtime with dlopen -- since libhdfs and libhdfs3 are designed to
be roughly ABI-compatible. One reason for this is that binary distributions of Arrow may not
know which libraries may be present on the target system.

This is more work than switching between the libraries at compile time but will be more flexible.

> Use generic HDFS component instead of libhdfs
> ---------------------------------------------
>                 Key: ARROW-243
>                 URL:
>             Project: Apache Arrow
>          Issue Type: New Feature
>            Reporter: Ryan Lewis
> I would like to use for example libhdfs3 from pivotal to read apache parquet files. This
would be a small change to the hdfs layer of apache arrow to support this.

This message was sent by Atlassian JIRA

View raw message