spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ruslan Dautkhanov (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-22658) SPIP: TeansorFlowOnSpark as a Scalable Deep Learning Lib of Apache Spark
Date Tue, 10 Jul 2018 21:48:00 GMT

    [ https://issues.apache.org/jira/browse/SPARK-22658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16539266#comment-16539266
] 

Ruslan Dautkhanov commented on SPARK-22658:
-------------------------------------------

Intel's BigDL is somewhat a TensorFlowOnSpark solution?


 [https://github.com/intel-analytics/BigDL]

> SPIP: TeansorFlowOnSpark as a Scalable Deep Learning Lib of Apache Spark
> ------------------------------------------------------------------------
>
>                 Key: SPARK-22658
>                 URL: https://issues.apache.org/jira/browse/SPARK-22658
>             Project: Spark
>          Issue Type: New Feature
>          Components: ML
>    Affects Versions: 2.2.0
>            Reporter: Andy Feng
>            Priority: Major
>         Attachments: SPIP_ TensorFlowOnSpark.pdf
>
>   Original Estimate: 336h
>  Remaining Estimate: 336h
>
> TensorFlowOnSpark (TFoS) was released at github for distributed TensorFlow training and
inference on Apache Spark clusters. TFoS is designed to:
> * Easily migrate all existing TensorFlow programs with minimum code change;
> * Support all TensorFlow functionalities: synchronous/asynchronous training, model/data
parallelism, inference and TensorBoard;
> * Easily integrate with your existing data processing pipelines (ex. Spark SQL) and machine
learning algorithms (ex. MLlib);
> * Be easily deployed on cloud or on-premise: CPU & GPU, Ethernet and Infiniband.
> We propose to merge TFoS into Apache Spark as a scalable deep learning library to:
> * Make deep learning easy for Apache Spark community: Familiar pipeline API for training
and inference; Enable TensorFlow training/inference on existing Spark clusters.
> * Further simplify data scientist experience: Ensure compatibility b/w Apache Spark and
TFoS; Reduce steps for installation.
> * Help Apache Spark evolutions on deep learning: Establish a design pattern for additional
frameworks (ex. Caffe, CNTK); Structured streaming for DL training/inference.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message