hivemall-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Makoto Yui <m...@apache.org>
Subject Re: Early Stopping / FFMs performance
Date Thu, 12 Oct 2017 10:59:45 GMT
In-memory representable of FFM models can be more than 2~3G bytes and
it does not fit in memory of Hadoop workers (memory space for each
worker is limited).
Instead of loading entire FFM prediction model into memory, Hivemall's
FFM prediction uses UDAF.

To do realtime prediction, it's better to load FFM prediction model
into memory using a memory-rich prediction (API) server.

If you want a realtime prediction, you need load FFM prediction model
from a Hive relation <i, W_i, Vf_i> into memory and the do in-memory
prediction there.
https://github.com/apache/incubator-hivemall/blob/master/core/src/main/java/hivemall/fm/FieldAwareFactorizationMachineModel.java#L84

For example, you can export ffm_model table as TSV file on S3, then
you can load it into memory.
It requires some coding though.

Makoto

2017-10-12 17:34 GMT+09:00 Shadi Mari <shadimari@gmail.com>:
> That was very helpful Makoto. Thanks indeed.
>
> One thing still am not sure of, is real-time prediction. I understand your
> example tackles batch prediction; however suppose i have only one vector of
> features to be predicted during a single AdRequest. how does ffm_predict
> perform on a single prediction request given a huge # of rows in the model.
> This is going to take seconds, where am aware that others handle such issues
> (using FFM) in milliseconds.
>
> Regards
>
> On Thu, Oct 12, 2017 at 2:09 AM, Makoto Yui <myui@apache.org> wrote:
>>
>> Hi,
>>
>> > Is there a way such models can be reduced into one model as a step
>> > before prediction, so that at least the # of rows can be reduced?
>>
>> See model ensemble in
>>
>> https://gist.github.com/myui/aaeef548a17eb90c4e88f824c3ca1bcd#model-ensemble
>>
>> The implementation is still beta and I might need to revise V
>> initialization scheme by using common Random seed for each V entries
>> of FFM on each node to get a better prediction result. Currently,
>> initial value of V is different for each node and thus I'm not sure
>> that model ensemble works properly.
>>
>> In my experience, extensive feature engineering as in [1] and
>> hyper-parameter tuning is mandatory to get a good result by FM.
>> [1] https://github.com/guestwalk/kaggle-2014-criteo/tree/master/converters
>>
>> Without them, It's hard to get a better prediction score than plain
>> logistic regression especially when applying feature hashing.
>>
>> Moreover, libffm's FFM implementation does some built-in feature
>> engineering such as instance-wise L2 normalization.
>>
>> I need more work to stabilize the current FFM implementation.
>>
>> > Given such a huge # of rows in the model table, i wonder how Real-Time
>> > prediction is feasible on HiveMALL?
>>
>> The prediction scheme in my example is batch prediction that has tons
>> of examples in test dataset.
>> When doing large batch prediction, realtime prediction is infeasible
>> in FFM that is not limited to Hivemall.
>>
>> Thanks,
>> Makoto
>>
>> 2017-10-11 23:22 GMT+09:00 Shadi Mari <shadimari@gmail.com>:
>> > Many thanks Makoto.
>> >
>> > I noticed that ffm_train produces ~1.8 Billion raws in the ffm_model
>> > table.
>> > In fact, each Tez Task has its own separate Model identified by the
>> > Model
>> > ID. Is there a way such models can be reduced into one model as a step
>> > before prediction, so that at least the # of rows can be reduced?
>> >
>> > Given such a huge # of rows in the model table, i wonder how Real-Time
>> > prediction is feasible on HiveMALL?
>> >
>> > Regards
>> >
>> > On Wed, Oct 11, 2017 at 9:17 AM, Makoto Yui <myui@apache.org> wrote:
>> >>
>> >> Shandi,
>> >>
>> >> - First release (v0.5.0) Nov, 2017
>> >> We plan to release the first Apache release in beginning of Nov.
>> >> Currently, feature freeze phase except minor patches.
>> >>
>> >> FFM is included but still in beta.
>> >>
>> >> - 2nd release (v0.5.1) Dec, 2017
>> >> word2vec and FFM are skipped in the first release and to be included
>> >> in the 2nd release in late Dec.
>> >>
>> >> - 3rd release (v0.6) Q1, 2018
>> >> xgboost and Multi-nominal logistic regression will be introduced in
>> >> the 3rd release in Q1, 2018.
>> >> https://github.com/apache/incubator-hivemall/pull/93
>> >>
>> >> Thanks,
>> >> Makoto
>> >>
>> >> 2017-10-11 2:02 GMT+09:00 Shadi Mari <shadimari@gmail.com>:
>> >> > Do you have an anticipated timeframe in order to move from Beta to
>> >> > GA.
>> >> > My
>> >> > observation is that hivemall releases are not so often, and so i
>> >> > would
>> >> > like
>> >> > to get a clue of your next cycle timeframe.
>> >> >
>> >> > Many thanks
>> >> >
>> >> > ________________________________
>> >> > From: Makoto Yui <myui@apache.org>
>> >> > Sent: Tuesday, October 10, 2017 7:33:37 PM
>> >> > To: user@hivemall.incubator.apache.org
>> >> > Cc: shadimari@gmail.com
>> >> > Subject: Re: Early Stopping / FFMs performance
>> >> >
>> >> > Hi,
>> >> >
>> >> > 2017-10-11 1:10 GMT+09:00 Shadi Mari <shadimari@gmail.com>:
>> >> >> I am using Criteo 2014 dataset for CTR prediction, which is 45M
>> >> >> examples
>> >> >> in
>> >> >> total. Do you think 8 hours is still resonable training duration
>> >> >> given
>> >> >> than
>> >> >> I am using your  EMR configurations? I never assumed this can be
>> >> >> such
>> >> >> time
>> >> >> consuming.
>> >> >
>> >> > I don't remember exact number but I took 5 hours or so for my
>> >> > previous
>> >> > setting:
>> >> > "-iters 10 -factors 4 -feature_hashing 20"
>> >> >
>> >> > FFM is very computation heavy algorithm and training of FFM takes
>> >> > time.
>> >> >
>> >> >> I already built a version from the master branch. As per your
>> >> >> feedback,
>> >> >> i
>> >> >> assume FFM implementation can not yet be used in production!
>> >> >> correct?
>> >> >
>> >> > Yes, it's still in beta. Use it at your own risks.
>> >> >
>> >> > FM implementation is stable and ready for production uses.
>> >> >
>> >> > Hivemall's FFM support linear term and global bias as in plain FM
>> >> > that
>> >> > are not supported in libffm.
>> >> > I'm not yet get a satisfied prediction accuracy in the current FFM
>> >> > implementation for the Criteo 2014 task.
>> >> >
>> >> > It may due to the default hyperparameter setting such as learning
>> >> > rate
>> >> > and l1/l2 params.
>> >> > https://www.kaggle.com/c/criteo-display-ad-challenge/discussion/10555
>> >> >
>> >> > Thanks,
>> >> > Makoto
>> >> >
>> >> > --
>> >> > Makoto YUI <myui AT apache.org>
>> >> > Research Engineer, Treasure Data, Inc.
>> >> > http://myui.github.io/
>> >>
>> >>
>> >>
>> >> --
>> >> Makoto YUI <myui AT apache.org>
>> >> Research Engineer, Treasure Data, Inc.
>> >> http://myui.github.io/
>> >
>> >
>>
>>
>>
>> --
>> Makoto YUI <myui AT apache.org>
>> Research Engineer, Treasure Data, Inc.
>> http://myui.github.io/
>
>



-- 
Makoto YUI <myui AT apache.org>
Research Engineer, Treasure Data, Inc.
http://myui.github.io/

Mime
View raw message