spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shivaram Venkataraman (JIRA)" <>
Subject [jira] [Created] (SPARK-19486) Investigate using multiple threads for task serialization
Date Tue, 07 Feb 2017 06:01:41 GMT
Shivaram Venkataraman created SPARK-19486:

             Summary: Investigate using multiple threads for task serialization
                 Key: SPARK-19486
             Project: Spark
          Issue Type: Improvement
          Components: Scheduler
    Affects Versions: 2.1.0
            Reporter: Shivaram Venkataraman

This is related to SPARK-18890, where all the serialization logic is moved into the Scheduler
backend thread. As a follow on to this we can investigate using a thread pool to serialize
a number of tasks together instead of using a single thread to serialize all of them.

Note that this may not yield sufficient benefits unless the driver has enough cores and we
don't run into contention across threads. We can first investigate potential benefits and
if there are sufficient benefits we can create a PR for this.

cc [~kayousterhout]

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message