hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "James Clampffer (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-12628) libhdfs crashes on thread exit for JNI+libhdfs applications
Date Mon, 16 Oct 2017 18:09:00 GMT

    [ https://issues.apache.org/jira/browse/HDFS-12628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16206312#comment-16206312

James Clampffer commented on HDFS-12628:

[~aw], I agree.  Going to send out a [discuss] email out to hdfs-dev in the next couple days
to see what everyone thinks about merging (I'm aware of some small blockers related to doc
and packaging).

> libhdfs crashes on thread exit for JNI+libhdfs applications
> -----------------------------------------------------------
>                 Key: HDFS-12628
>                 URL: https://issues.apache.org/jira/browse/HDFS-12628
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: native
>    Affects Versions: 3.0.0-alpha3
>            Reporter: Joe McDonnell
>            Priority: Critical
>         Attachments: jni-util-test2.cc
> Impala uses libhdfs to access HDFS while also using JNI to run other Java code. Impala
currently relies on HDFS's getJNIEnv to get a JNIEnv to interact with the process JVM (which
is created by HDFS code). It uses this JNIEnv even for code that is not related to HDFS.
> In recent versions of HDFS, getJNIEnv is no longer visible in libhdfs due to HDFS-7879.
In HDFS-8474, the proposed solution was for Impala to write its own equivalent (tracked by
IMPALA-2029). After implementing an equivalent of getJNIEnv (heavily based on HDFS code, but
with distinct names), we are seeing crashes in hdfsThreadDestructor() in threads that use
both HDFS and JNI codepaths. The crash shows up under concurrency and does not reproduce in
serial execution.
> I have distilled it down to a simple testcase that reproduces the issue. It creates a
JVM in the main thread (which Impala does at startup), then spawns multiple threads that do
basic HDFS and JNI work. I have removed all but the essential steps. 
> This blocks running Impala on any hadoop version past 2.7 (when HDFS-7879 was merged).
Note that exposing getJNIEnv should unblock Impala development if a fix is not forthcoming.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message