Mailing-List: contact user-help@flink.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@flink.apache.org
MIME-Version: 1.0
In-Reply-To: <3D759DEF-A2BA-4E90-8ADD-8DED85BE612E@alibaba-inc.com>
References: <CA+gCnNOSQt3UEBc1_sd8HDvRE4K5L00SWd0zWTb_f6m+xnRtGw@mail.gmail.com>
 <E700B7DE-6D38-40B6-983C-376981CF0211@data-artisans.com> <CA+gCnNN+q=MTR_dwnp2rTRk6545CPc=uN57_4x7X_n2Q6C-+VQ@mail.gmail.com>
 <iicb2ya5vkdr4i2cuhew5821.1471959942981@email.android.com> <3D759DEF-A2BA-4E90-8ADD-8DED85BE612E@alibaba-inc.com>
From: Maximilian Michels <mxm@apache.org>
Date: Wed, 24 Aug 2016 13:47:27 +0200
Message-ID: <CAGco--asdSh5k6ib5=A5__jso9n-BaXTjZNUYu9+W4YJCXadSw@mail.gmail.com>
Subject: Re: How to share text file across tasks at run time in flink.
To: "user@flink.apache.org" <user@flink.apache.org>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
archived-at: Wed, 24 Aug 2016 11:47:51 -0000

Hi!

1. The community is working on adding side inputs to the DataStream
API. That will allow you to easily distribute data to all of your
workers.

2. In the meantime, you could use `.broadcast()` on a DataSet to
broadcast data to all workers. You still have to join that data with
another stream though.

3. The easiest method of all is to simply load your file in the
RichMapFunction's open() method. The file can reside in a distributed
file system which is accessible by all workers.

Cheers,
Max

On Wed, Aug 24, 2016 at 6:45 AM, Jark Wu <wuchong.wc@alibaba-inc.com> wrote=
:
> Hi,
>
> I think what Bswaraj want is excatly something like Storm Distributed Cac=
he
> API[1] (if I=E2=80=99m not misunderstanding).
>
> The distributed cache feature in storm is used to efficiently distribute
> files (or blobs, which is the equivalent terminology for a file in the
> distributed cache and is used interchangeably in this document) that are
> large and can change during the lifetime of a topology, such as geo-locat=
ion
> data, dictionaries, etc. Typical use cases include phrase recognition,
> entity extraction, document classification, URL re-writing, location/addr=
ess
> detection and so forth. Such files may be several KB to several GB in siz=
e.
> For small datasets that don't need dynamic updates, including them in the
> topology jar could be fine. But for large files, the startup times could
> become very large. In these cases, the distributed cache feature can prov=
ide
> fast topology startup, especially if the files were previously downloaded
> for the same submitter and are still in the cache. This is useful with
> frequent deployments, sometimes few times a day with updated jars, becaus=
e
> the large cached files will remain available without changes. The large
> cached blobs that do not change frequently will remain available in the
> distributed cache.
>
>
> We can look into this whether it is a common use case and how to implemen=
t
> it in Flink.
>
> [1] http://storm.apache.org/releases/2.0.0-SNAPSHOT/distcache-blobstore.h=
tml
>
>
> - Jark Wu
>
> =E5=9C=A8 2016=E5=B9=B48=E6=9C=8823=E6=97=A5=EF=BC=8C=E4=B8=8B=E5=8D=889:=
45=EF=BC=8CLohith Samaga M <Lohith.Samaga@mphasis.com> =E5=86=99=E9=81=93=
=EF=BC=9A
>
> Hi
> May be you could use Cassandra to store and fetch all such reference data=
.
> This way the reference data can be updated without restarting your
> application.
>
> Lohith
>
> Sent from my Sony Xperia=E2=84=A2 smartphone
>
>
>
> ---- Baswaraj Kasture wrote ----
>
> Thanks Kostas !
> I am using DataStream API.
>
> I have few config/property files (key vale text file) and also have busin=
ess
> rule files (json).
> These rules and configurations are needed when we process incoming event.
> Is there any way to share them to task nodes from driver program ?
> I think this is very common use case and am sure other users may face
> similar issues.
>
> +Baswaraj
>
> On Mon, Aug 22, 2016 at 4:56 PM, Kostas Kloudas
> <k.kloudas@data-artisans.com> wrote:
>>
>> Hello Baswaraj,
>>
>> Are you using the DataSet (batch) or the DataStream API?
>>
>> If you are in the first, you can use a broadcast variable for your task.
>> If you are using the DataStream one, then there is no proper support for
>> that.
>>
>> Thanks,
>> Kostas
>>
>> On Aug 20, 2016, at 12:33 PM, Baswaraj Kasture <kbaswaraj1@gmail.com>
>> wrote:
>>
>> Am running Flink standalone cluster.
>>
>> I have text file that need to be shared across tasks when i submit my
>> application.
>> in other words , put this text file in class path of running tasks.
>>
>> How can we achieve this with flink ?
>>
>> In spark, spark-submit has --jars option that puts all the files specifi=
ed
>> in class path of executors (executors run in separate JVM and spawned
>> dynamically, so it is possible).
>>
>> Flink's task managers run tasks in separate thread under taskmanager JVM
>> (?) , how can we make this text file to be accessible on all tasks spawn=
ed
>> by current application ?
>>
>> Using HDFS, NFS or including file in program jar is one way that i know,
>> but am looking for solution that can allows me to provide text file at r=
un
>> time and still accessible in all tasks.
>> Thanks.
>>
>>
>
>
> Information transmitted by this e-mail is proprietary to Mphasis, its
> associated companies and/ or its customers and is intended
> for use only by the individual or entity to which it is addressed, and ma=
y
> contain information that is privileged, confidential or
> exempt from disclosure under applicable law. If you are not the intended
> recipient or it appears that this mail has been forwarded
> to you without proper authority, you are notified that any use or
> dissemination of this information in any manner is strictly
> prohibited. In such cases, please notify us immediately at
> mailmaster@mphasis.com and delete this mail from your records.
>
>