spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daniel Siegmann <>
Subject Re: Spark ML - Scaling logistic regression for many features
Date Thu, 28 Apr 2016 19:06:42 GMT

I have submitted a PR as well.

On Fri, Mar 18, 2016 at 7:15 AM, Nick Pentreath <>

> No, I didn't yet - feel free to create a JIRA.
> On Thu, 17 Mar 2016 at 22:55 Daniel Siegmann <>
> wrote:
>> Hi Nick,
>> Thanks again for your help with this. Did you create a ticket in JIRA for
>> investigating sparse models in LR and / or multivariate summariser? If so,
>> can you give me the issue key(s)? If not, would you like me to create these
>> tickets?
>> I'm going to look into this some more and see if I can figure out how to
>> implement these fixes.
>> ~Daniel Siegmann
>> On Sat, Mar 12, 2016 at 5:53 AM, Nick Pentreath <
>> > wrote:
>>> Also adding dev list in case anyone else has ideas / views.
>>> On Sat, 12 Mar 2016 at 12:52, Nick Pentreath <>
>>> wrote:
>>>> Thanks for the feedback.
>>>> I think Spark can certainly meet your use case when your data size
>>>> scales up, as the actual model dimension is very small - you will need to
>>>> use those indexers or some other mapping mechanism.
>>>> There is ongoing work for Spark 2.0 to make it easier to use models
>>>> outside of Spark - also see PMML export (I think mllib logistic regression
>>>> is supported but I have to check that). That will help use spark models in
>>>> serving environments.
>>>> Finally, I will add a JIRA to investigate sparse models for LR - maybe
>>>> also a ticket for multivariate summariser (though I don't think in practice
>>>> there will be much to gain).
>>>> On Fri, 11 Mar 2016 at 21:35, Daniel Siegmann <
>>>>> wrote:
>>>>> Thanks for the pointer to those indexers, those are some good
>>>>> examples. A good way to go for the trainer and any scoring done in Spark.
>>>>> will definitely have to deal with scoring in non-Spark systems though.
>>>>> I think I will need to scale up beyond what single-node liblinear can
>>>>> practically provide. The system will need to handle much larger sub-samples
>>>>> of this data (and other projects might be larger still). Additionally,
>>>>> system needs to train many models in parallel (hyper-parameter optimization
>>>>> with n-fold cross-validation, multiple algorithms, different sets of
>>>>> features).
>>>>> Still, I suppose we'll have to consider whether Spark is the best
>>>>> system for this. For now though, my job is to see what can be achieved
>>>>> Spark.
>>>>> On Fri, Mar 11, 2016 at 12:45 PM, Nick Pentreath <
>>>>>> wrote:
>>>>>> Ok, I think I understand things better now.
>>>>>> For Spark's current implementation, you would need to map those
>>>>>> features as you mention. You could also use say StringIndexer ->
>>>>>> OneHotEncoder or VectorIndexer. You could create a Pipeline to deal
>>>>>> the mapping and training (e.g.
>>>>>> Pipeline supports persistence.
>>>>>> But it depends on your scoring use case too - a Spark pipeline can
>>>>>> saved and then reloaded, but you need all of Spark dependencies in
>>>>>> serving app which is often not ideal. If you're doing bulk scoring
>>>>>> then it may suit.
>>>>>> Honestly though, for that data size I'd certainly go with something
>>>>>> like Liblinear :) Spark will ultimately scale better with # training
>>>>>> examples for very large scale problems. However there are definitely
>>>>>> limitations on model dimension and sparse weight vectors currently.
>>>>>> are potential solutions to these but they haven't been implemented
as yet.
>>>>>> On Fri, 11 Mar 2016 at 18:35 Daniel Siegmann <
>>>>>>> wrote:
>>>>>>> On Fri, Mar 11, 2016 at 5:29 AM, Nick Pentreath <
>>>>>>>> wrote:
>>>>>>>> Would you mind letting us know the # training examples in
>>>>>>>> datasets? Also, what do your features look like? Are they
text, categorical
>>>>>>>> etc? You mention that most rows only have a few features,
and all rows
>>>>>>>> together have a few 10,000s features, yet your max feature
value is 20
>>>>>>>> million. How are your constructing your feature vectors to
get a 20 million
>>>>>>>> size? The only realistic way I can see this situation occurring
in practice
>>>>>>>> is with feature hashing (HashingTF).
>>>>>>> The sub-sample I'm currently training on is about 50K rows, so
>>>>>>> small.
>>>>>>> The features causing this issue are numeric (int) IDs for ...
>>>>>>> call it "Thing". For each Thing in the record, we set the feature
>>>>>>> to a value of 1.0 in our vector (which is of course
>>>>>>> SparseVector). I'm not sure how IDs are generated for Things,
>>>>>>> they can be large numbers.
>>>>>>> The largest Thing ID is around 20 million, so that ends up being
>>>>>>> size of the vector. But in fact there are fewer than 10,000 unique
>>>>>>> IDs in this data. The mean number of features per record in what
>>>>>>> currently training against is 41, while the maximum for any given
>>>>>>> was 1754.
>>>>>>> It is possible to map the features into a small set (just need
>>>>>>> zipWithIndex), but this is undesirable because of the added complexity
>>>>>>> just for the training, but also anything wanting to score against
>>>>>>> model). It might be a little easier if this could be encapsulated
>>>>>>> the model object itself (perhaps via composition), though I'm
not sure how
>>>>>>> feasible that is.
>>>>>>> But I'd rather not bother with dimensionality reduction at all
>>>>>>> since we can train using liblinear in just a few minutes, it
doesn't seem
>>>>>>> necessary.
>>>>>>>> MultivariateOnlineSummarizer uses dense arrays, but it should
>>>>>>>> possible to enable sparse data. Though in theory, the result
will tend to
>>>>>>>> be dense anyway, unless you have very many entries in the
input feature
>>>>>>>> vector that never occur and are actually zero throughout
the data set
>>>>>>>> (which it seems is the case with your data?). So I doubt
whether using
>>>>>>>> sparse vectors for the summarizer would improve performance
in general.
>>>>>>> Yes, that is exactly my case - the vast majority of entries in
>>>>>>> input feature vector will *never* occur. Presumably that means
>>>>>>> of the values in the aggregators' arrays will be zero.
>>>>>>>> LR doesn't accept a sparse weight vector, as it uses dense
>>>>>>>> for coefficients and gradients currently. When using L1 regularization,
>>>>>>>> could support sparse weight vectors, but the current implementation
>>>>>>>> do that yet.
>>>>>>> Good to know it is theoretically possible to implement. I'll
have to
>>>>>>> give it some thought. In the meantime I guess I'll experiment
>>>>>>> coalescing the data to minimize the communication overhead.
>>>>>>> Thanks again.

View raw message