spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Reynold Xin <>
Subject Re: Ability to auto-detect input data for datasources (by file extension).
Date Fri, 19 Feb 2016 02:27:32 GMT
Thanks for the email.

Don't make it that complicated. We just want to simplify the common cases
(e.g. csv/parquet), and don't need this to work for everything out there.

On Thu, Feb 18, 2016 at 9:25 PM, Hyukjin Kwon <> wrote:

> Hi all,
> I am planning to submit a PR for
> Currently, file format is not detected by the file extension unlike
> compression codecs are being detected.
> I am thinking of introducing another interface (a function) at
> DataSourceRegister just like shortName() at in order to specify possible
> file exceptions so that we can detect datasources by file extensions just
> like Hadoop does for compression codecs.
> Since adding an interface should be carefully done, I want to first ask if
> this approach looks appropriate.
> Could you please give me some feedback for this?
> Thanks!

View raw message