flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bauss, Julian" <Julian.Ba...@bonprix.net>
Subject Storing JPMML-Model Object as a Variable Closure?
Date Fri, 02 Sep 2016 09:20:05 GMT
Hello Everybody,

I’m currently refactoring some code and am looking for a better alternative to handle
JPMML-Models in data streams. At the moment the flink job I’m working on references a model-object
as a Singleton which I want to change because static references tend to cause problems in
distributed systems.

I thought about handing the model-object to the function that uses it as a variable closure.
The object
can be between 16MB and 250MB in size (depending on the depth of the decision tree).

According to https://cwiki.apache.org/confluence/display/FLINK/Variables+Closures+vs.+Broadcast+Variables
that’s way too large though.
Are there any viable alternatives or would this be the „right way“ to handle this situation?

Best Regards,

Julian


**************************************************************************************************************

bonprix Handelsgesellschaft mbH
Sitz der Gesellschaft: Hamburg

Geschäftsführung:
Dr. Marcus Ackermann (Vorsitzender)
Dr. Kai Heck
Rien Jansen
Markus Fuchshofen
Beiratsvorsitzender: Alexander Birken

Handelsregister AG Hamburg HR B 36 455

Adresse:

bonprix Handelsgesellschaft mbH

Haldesdorfer Str. 61
22179 Hamburg

Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen.
Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben,
informieren Sie bitte sofort den Absender und vernichten Sie diese Mail.
Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser E-Mail ist nicht gestattet.

This e-mail may contain confidential and/or privileged information.
If you are not the intended recipient (or have received the e-mail in error)
please notify the sender immediately and delete this e-mail. Any unauthorized copying,
disclosure or distribution of the material in this e-mail is strictly forbidden.

**************************************************************************************************************

Mime
View raw message