spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Robert Kruszewski (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-20952) ParquetFileFormat should forward TaskContext to its forkjoinpool
Date Wed, 16 Aug 2017 20:00:05 GMT

    [ https://issues.apache.org/jira/browse/SPARK-20952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16129327#comment-16129327
] 

Robert Kruszewski commented on SPARK-20952:
-------------------------------------------

I don't care about what actual filesystem impls are doing. This is because I have a wrapper
that I want to have a handle on actual taskcontext that extracts the right parameters and
delegates to the underlying system. As such anything underneath my wrapper doesn't have to
deal with spark. We have been relying on taskcontext being available everywhere on the executor
during execution.

> ParquetFileFormat should forward TaskContext to its forkjoinpool
> ----------------------------------------------------------------
>
>                 Key: SPARK-20952
>                 URL: https://issues.apache.org/jira/browse/SPARK-20952
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core
>    Affects Versions: 2.1.1
>            Reporter: Robert Kruszewski
>            Priority: Minor
>
> TaskContext is a ThreadLocal as a result when you fork a thread inside your executor
task you lose the handle on the original context set by the executor. We should change it
to InheritableThreadLocal so we can access it inside thread pools on executors. 
> See ParquetFileFormat#readFootersInParallel for example of code that uses thread pools
inside the tasks.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message