spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kazuaki Ishizaki (JIRA)" <>
Subject [jira] [Closed] (SPARK-12620) Proposal of GPU exploitation for Spark
Date Tue, 05 Jan 2016 07:46:39 GMT


Kazuaki Ishizaki closed SPARK-12620.

Move to SPARK-3875.

> Proposal of GPU exploitation for Spark
> --------------------------------------
>                 Key: SPARK-12620
>                 URL:
>             Project: Spark
>          Issue Type: New Feature
>          Components: Spark Core
>            Reporter: Kazuaki Ishizaki
> I created a new JIRA entry to move from SPARK-3875
> Exploiting GPUs can allow us to shorten the execution time of a Spark job and to reduce
the number of machines in a cluster. We are working to effectively and easily exploit GPUs
on Spark at  []. Our project page is [].
A design document is [here|]
> Our ideas for exploiting GPUs are
> # adding a new format for a partition in an RDD, which is a column-based structure in
an array format, in addition to the current Iterator\[T\] format with Seq\[T\]
> # generating parallelized GPU native code to access data in the new format from a Spark
application program by using an optimizer and code generator (this is similar to [Project
and pre-compiled library
> The motivation of idea 1 is to reduce the overhead of serializing/deserializing partition
data for copy between CPU and GPU. The motivation of idea 2 is to avoid writing hardware-dependent
code by application programmers. At first, we are working for idea A (For idea B, we need
to write [CUDA|] code for now). 
> This prototype achieved [3.15x performance improvement|]
of logistic regression ([SparkGPULR|])
in examples on a 16-thread IvyBridge box with an NVIDIA K40 GPU card over that with no GPU
> You can download the pre-build binary for x86_64 and ppc64le from [here|].
You can run this on Amazon EC2 by [the procedure|],

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message