flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vishwas Siravara <vsirav...@gmail.com>
Subject Re: Non parallel file sources
Date Tue, 23 Jun 2020 19:57:29 GMT
Thanks that makes sense.

On Tue, Jun 23, 2020 at 2:13 PM Laurent Exsteens <
laurent.exsteens@euranova.eu> wrote:

> Hi Nick,
>
> On a project I worked on, we simply made the file accessible on a shared
> NFS drive.
> Our source was custom, and we forced it to parallelism 1 inside the job,
> so the file wouldn't be read multiple times. The rest of the job was
> distributed.
> This was also on a standalone cluster. On a resource managed cluster I
> guess the resource manager could take care of copying the file for us.
>
> Hope this can help. If there would have been a better solution, I'm also
> happy to hear it :).
>
> Regards,
>
> Laurent.
>
>
> On Tue, Jun 23, 2020, 20:51 Nick Bendtner <buggie89@gmail.com> wrote:
>
>> Hi guys,
>> What is the best way to process a file from a unix file system since
>> there is no guarantee as to which task manager will be assigned to process
>> the file. We run flink in standalone mode. We currently follow the brute
>> force way in which we copy the file to every task manager, is there a
>> better way to do this ?
>>
>>
>> Best,
>> Nick.
>>
>
> ♻ Be green, keep it on the screen

Mime
View raw message