spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shixiong Zhu (JIRA)" <>
Subject [jira] [Commented] (SPARK-20952) TaskContext should be an InheritableThreadLocal
Date Fri, 09 Jun 2017 00:37:18 GMT


Shixiong Zhu commented on SPARK-20952:

For `ParquetFileFormat#readFootersInParallel`, I would suggest that you just set the TaskContext
in "parFiles.flatMap". 

    val taskContext = TaskContext.get

    val parFiles = partFiles.par
    parFiles.tasksupport = new ForkJoinTaskSupport(new ForkJoinPool(8))
    parFiles.flatMap { currentFile =>

In this special case, it's safe since this is a local one-time thread pool.

> TaskContext should be an InheritableThreadLocal
> -----------------------------------------------
>                 Key: SPARK-20952
>                 URL:
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core
>    Affects Versions: 2.1.1
>            Reporter: Robert Kruszewski
>            Priority: Minor
> TaskContext is a ThreadLocal as a result when you fork a thread inside your executor
task you lose the handle on the original context set by the executor. We should change it
to InheritableThreadLocal so we can access it inside thread pools on executors. 
> See ParquetFileFormat#readFootersInParallel for example of code that uses thread pools
inside the tasks.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message