spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Apache Spark (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-20114) spark.ml parity for sequential pattern mining - PrefixSpan
Date Tue, 13 Mar 2018 13:22:00 GMT

    [ https://issues.apache.org/jira/browse/SPARK-20114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16396926#comment-16396926
] 

Apache Spark commented on SPARK-20114:
--------------------------------------

User 'WeichenXu123' has created a pull request for this issue:
https://github.com/apache/spark/pull/20810

> spark.ml parity for sequential pattern mining - PrefixSpan
> ----------------------------------------------------------
>
>                 Key: SPARK-20114
>                 URL: https://issues.apache.org/jira/browse/SPARK-20114
>             Project: Spark
>          Issue Type: Sub-task
>          Components: ML
>    Affects Versions: 2.2.0
>            Reporter: yuhao yang
>            Priority: Major
>
> Creating this jira to track the feature parity for PrefixSpan and sequential pattern
mining in Spark ml with DataFrame API. 
> First list a few design issues to be discussed, then subtasks like Scala, Python and
R API will be created.
> # Wrapping the MLlib PrefixSpan and provide a generic fit() should be straightforward.
Yet PrefixSpan only extracts frequent sequential patterns, which is not good to be used directly
for predicting on new records. Please read  http://data-mining.philippe-fournier-viger.com/introduction-to-sequential-rule-mining/
for some background knowledge. Thanks Philippe Fournier-Viger for providing insights. If we
want to keep using the Estimator/Transformer pattern, options are:
>      #*  Implement a dummy transform for PrefixSpanModel, which will not add new column
to the input DataSet. The PrefixSpanModel is only used to provide access for frequent sequential
patterns.
>      #*  Adding the feature to extract sequential rules from sequential patterns. Then
use the sequential rules in the transform as FPGrowthModel.  The rules extracted are of the
form X–> Y where X and Y are sequential patterns. But in practice, these rules are not
very good as they are too precise and thus not noise tolerant.
> #  Different from association rules and frequent itemsets, sequential rules can be extracted
from the original dataset more efficiently using algorithms like RuleGrowth, ERMiner. The
rules are X–> Y where X is unordered and Y is unordered, but X must appear before Y,
which is more general and can work better in practice for prediction. 
> I'd like to hear more from the users to see which kind of Sequential rules are more practical.




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message