flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marco Villalobos <mvillalo...@kineteque.com>
Subject Re: Does Flink support reading files or CSV files from java.io.InputStream instead of file paths?
Date Tue, 16 Jun 2020 16:55:10 GMT
Okay, it is not supported.  

I thought about this more and I disagree that this would break "distributability".

Currently, the API accepts a String which is a path, whether it be a path to a remote URL
or a local file.
However, after the URL is parsed, ultimately what ends up happening is that an InputStream
will serve as the abstraction that reads input from some source.

An InputStream can be remote, it can be a local file, it can be a connection to a server,
or another client, and that situation, the system remains distributed.

Also, such an enhancement promotes "Interoperability" because now the user can decide the
source of that data, rather forcing it to be a URL or physical file path.

I think this feature would make testing and demos more portable. I was writing a demo, and
I wanted it to run without command-line arguments, which would have been very handy. I want
the user to simply checkout the code and run it without having to supply a command line parameter
declaring where the input file resides.

Thank you.

> On Jun 16, 2020, at 4:57 AM, Aljoscha Krettek <aljoscha@apache.org> wrote:
> 
> Hi Marco,
> 
> this is not possible since Flink is designed mostly to read files from a distributed
filesystem, where paths are used to refer to those files. If you read from files on the classpath
you could just use plain old Java code and won't need a distributed processing system such
as Flink.
> 
> Best,
> Aljoscha
> 
> On 16.06.20 06:46, Marco Villalobos wrote:
>> Does Flink support reading files or CSV files from java.io.InputStream instead of
file paths?
>> I'd rather just store my file on the class path and load it with java.lang.ClassLoader#getResourceAsStream(String).
>> If there is a way, I'd appreciate an example.
> 


Mime
View raw message