hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kevin Risden (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-14251) BlockPoolSlice ForkJoinPool introduced by HDFS-13768 breaks under Java SecurityManager
Date Fri, 01 Feb 2019 13:12:00 GMT

    [ https://issues.apache.org/jira/browse/HDFS-14251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16758281#comment-16758281
] 

Kevin Risden commented on HDFS-14251:
-------------------------------------

JDK9+ changed the behavior of ForkJoinPool so that threads by default get no permissions under
the Java SecurityManager with the default thread factory. This wasn't the case in JDK8. 

[https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/ForkJoinPool.html]
{code:java}
If a SecurityManager is present and no factory is specified, then the default pool uses
a factory supplying threads that have no Permissions enabled. The system class loader is
used to load these classes. Upon any error in establishing these settings, default parameters
are used. It is possible to disable or limit the use of threads in the common pool by setting
the parallelism property to zero, and/or using a factory that may return null. However doing
so may cause unjoined tasks to never be executed.
{code}
[https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/util/concurrent/ForkJoinPool.html]
{code:java}
If no thread factory is supplied via a system property, then the common pool uses a factory
that uses the system class loader as the thread context class loader. In addition, if a SecurityManager is
present, then the common pool uses a factory supplying threads that have no Permissions enabled.
Upon any error in establishing these settings, default parameters are used. It is possible
to disable or limit the use of threads in the common pool by setting the parallelism property
to zero, and/or using a factory that may return null. However doing so may cause unjoined
tasks to never be executed.{code}
From [https://github.com/apache/hadoop/blob/branch-3.2/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/BlockPoolSlice.java#L213]

 

Based on discussion in SOLR-9515, doing any IO within the ForkJoinPool can cause issues and
should most likely not be used for this purpose. If the ForkJoinPool must be used, then a
way to set the custom thread factory would be useful so the security policy can be passed
in.

> BlockPoolSlice  ForkJoinPool introduced by HDFS-13768 breaks under Java SecurityManager
> ---------------------------------------------------------------------------------------
>
>                 Key: HDFS-14251
>                 URL: https://issues.apache.org/jira/browse/HDFS-14251
>             Project: Hadoop HDFS
>          Issue Type: Bug
>    Affects Versions: 2.10.0, 3.2.0, 3.1.2
>            Reporter: Kevin Risden
>            Priority: Major
>
> HDFS-13768 introduced ForkJoinPool to improve performance of BlockPoolSlice. The created
ForkJoinPool doesn't pass in a custom thread factory and by default the thread factory prevents
all privileges under the Java SecurityManager.
> This broke when integrating Hadoop 3.2.0 with Apache Solr in SOLR-9515.
> It isn't clear that the ForkJoinPool is necessary when a regular normal executor could
work instead?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message