flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From KirstiLaurila <kirsti.laur...@rovio.com>
Subject Re: Flink ML 1.0.0 - Saving and Loading Models to Score a Single Feature Vector
Date Tue, 12 Apr 2016 08:29:28 GMT
How should this be done for the recommendation engine (that is ALS, example
here 
https://ci.apache.org/projects/flink/flink-docs-release-1.0/apis/batch/libs/ml/als.html
<https://ci.apache.org/projects/flink/flink-docs-release-1.0/apis/batch/libs/ml/als.html>

).

 I am able to run the example with my example data but cannot get anything
written to any file (user or item matrices). 

Basically, I have tried something like this


    

Tried also to apply similar approach than this 



but with no success. Could someone help me with this to get my model saved?


Best,
Kirsti



Trevor Grant wrote
> I'm just about to open an issue / PR solution for 'warm-starts'
> 
> Once this is in, we could just add a setter for the weight vector (and
> what
> ever iteration you're on if you're going to do more partial fits).
> 
> Then all you need to save if your weight vector (and iter number).
> 
> 
> 
> Trevor Grant
> Data Scientist
> https://github.com/rawkintrevo
> http://stackexchange.com/users/3002022/rawkintrevo
> http://trevorgrant.org
> 
> *"Fortunate is he, who is able to know the causes of things."  -Virgil*
> 
> 
> On Fri, Apr 8, 2016 at 9:04 AM, Behrouz Derakhshan <

> behrouz.derakhshan@

>> wrote:
> 
>> Is there a reasons the Predictor or Estimator class don't have read and
>> write methods for saving and retrieving the model? I couldn't find Jira
>> issues for it. Does it make sense to create one ?
>>
>> BR,
>> Behrouz
>>
>> On Wed, Mar 30, 2016 at 4:40 PM, Till Rohrmann &lt;

> trohrmann@

> &gt;
>> wrote:
>>
>>> Yes Suneel is completely wright. If the data does not implement
>>> IOReadableWritable it is probably easier to use the
>>> TypeSerializerOutputFormat. What you need here to seralize the data is a
>>> TypeSerializer. You can obtain it the following way:
>>>
>>> val model = mlr.weightsOption.get
>>>
>>> val weightVectorTypeInfo = TypeInformation.of(classOf[WeightVector])
>>> val weightVectorSerializer = weightVectorTypeInfo.createSerializer(new
>>> ExecutionConfig())
>>> val outputFormat = new TypeSerializerOutputFormat[WeightVector]
>>> outputFormat.setSerializer(weightVectorSerializer)
>>>
>>> model.write(outputFormat, "path")
>>>
>>> Cheers,
>>> Till
>>> ​
>>>
>>> On Tue, Mar 29, 2016 at 8:22 PM, Suneel Marthi &lt;

> smarthi@

> &gt;
>>> wrote:
>>>
>>>> U may want to use FlinkMLTools.persist() methods which use
>>>> TypeSerializerFormat and don't enforce IOReadableWritable.
>>>>
>>>>
>>>>
>>>> On Tue, Mar 29, 2016 at 2:12 PM, Sourigna Phetsarath <
>>>> 

> gna.phetsarath@

>> wrote:
>>>>
>>>>> Till,
>>>>>
>>>>> Thank you for your reply.
>>>>>
>>>>> Having this issue though, WeightVector does not extend
>>>>> IOReadWriteable:
>>>>>
>>>>> *public* *class* SerializedOutputFormat<*T* *extends*
>>>>> IOReadableWritable>
>>>>>
>>>>> *case* *class* WeightVector(weights: Vector, intercept: Double)
>>>>> *extends* Serializable {}
>>>>>
>>>>>
>>>>> However, I will use the approach to write out the weights as text.
>>>>>
>>>>>
>>>>> On Tue, Mar 29, 2016 at 5:01 AM, Till Rohrmann &lt;

> trohrmann@

> &gt;
>>>>> wrote:
>>>>>
>>>>>> Hi Gna,
>>>>>>
>>>>>> there are no utilities yet to do that but you can do it manually.
In
>>>>>> the end, a model is simply a Flink DataSet which you can serialize
to
>>>>>> some file. Upon reading this DataSet you simply have to give it to
>>>>>> your algorithm to be used as the model. The following code snippet
>>>>>> illustrates this approach:
>>>>>>
>>>>>> mlr.fit(inputDS, parameters)
>>>>>>
>>>>>> // write model to disk using the SerializedOutputFormat
>>>>>> mlr.weightsOption.get.write(new SerializedOutputFormat[WeightVector],
>>>>>> "path")
>>>>>>
>>>>>> // read the serialized model from disk
>>>>>> val model = env.readFile(new SerializedInputFormat[WeightVector],
>>>>>> "path")
>>>>>>
>>>>>> // set the read model for the MLR algorithm
>>>>>> mlr.weightsOption = model
>>>>>>
>>>>>> Cheers,
>>>>>> Till
>>>>>> ​
>>>>>>
>>>>>> On Tue, Mar 29, 2016 at 10:46 AM, Simone Robutti <
>>>>>> 

> simone.robutti@

>> wrote:
>>>>>>
>>>>>>> To my knowledge there is nothing like that. PMML is not supported
in
>>>>>>> any form and there's no custom saving format yet. If you really
need
>>>>>>> a
>>>>>>> quick and dirty solution, it's not that hard to serialize the
model
>>>>>>> into a
>>>>>>> file.
>>>>>>>
>>>>>>> 2016-03-28 17:59 GMT+02:00 Sourigna Phetsarath <
>>>>>>> 

> gna.phetsarath@

>>:
>>>>>>>
>>>>>>>> Flinksters,
>>>>>>>>
>>>>>>>> Is there an example of saving a Trained Model, loading a
Trained
>>>>>>>> Model and then scoring one or more feature vectors using
Flink ML?
>>>>>>>>
>>>>>>>> All of the examples I've seen have shown only sequential
fit and
>>>>>>>> predict.
>>>>>>>>
>>>>>>>> Thank you.
>>>>>>>>
>>>>>>>> -Gna
>>>>>>>> --
>>>>>>>>
>>>>>>>>
>>>>>>>> *Gna Phetsarath*System Architect // AOL Platforms // Data
Services
>>>>>>>> // Applied Research Chapter
>>>>>>>> 770 Broadway, 5th Floor, New York, NY 10003
>>>>>>>> o: 212.402.4871 // m: 917.373.7363
>>>>>>>> vvmr: 8890237 aim: sphetsarath20 t: @sourigna
>>>>>>>>
>>>>>>>> * &lt;http://www.aolplatforms.com&gt;*
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>>
>>>>> *Gna Phetsarath*System Architect // AOL Platforms // Data Services //
>>>>> Applied Research Chapter
>>>>> 770 Broadway, 5th Floor, New York, NY 10003
>>>>> o: 212.402.4871 // m: 917.373.7363
>>>>> vvmr: 8890237 aim: sphetsarath20 t: @sourigna
>>>>>
>>>>> * &lt;http://www.aolplatforms.com&gt;*
>>>>>
>>>>
>>>>
>>>
>>





--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-ML-1-0-0-Saving-and-Loading-Models-to-Score-a-Single-Feature-Vector-tp5766p6056.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

Mime
View raw message