hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Allen Wittenauer (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-12628) libhdfs crashes on thread exit for JNI+libhdfs applications
Date Thu, 12 Oct 2017 23:49:00 GMT

    [ https://issues.apache.org/jira/browse/HDFS-12628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16202824#comment-16202824

Allen Wittenauer commented on HDFS-12628:

Honestly, it's probably time HDFS-8707 got merged into a release and then we can kill off
libhdfs.  cc: [~James C], [~bobhansen], [~anatoli.shein], ...

> libhdfs crashes on thread exit for JNI+libhdfs applications
> -----------------------------------------------------------
>                 Key: HDFS-12628
>                 URL: https://issues.apache.org/jira/browse/HDFS-12628
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: native
>    Affects Versions: 3.0.0-alpha3
>            Reporter: Joe McDonnell
>            Priority: Critical
>         Attachments: jni-util-test2.cc
> Impala uses libhdfs to access HDFS while also using JNI to run other Java code. Impala
currently relies on HDFS's getJNIEnv to get a JNIEnv to interact with the process JVM (which
is created by HDFS code). It uses this JNIEnv even for code that is not related to HDFS.
> In recent versions of HDFS, getJNIEnv is no longer visible in libhdfs due to HDFS-7879.
In HDFS-8474, the proposed solution was for Impala to write its own equivalent (tracked by
IMPALA-2029). After implementing an equivalent of getJNIEnv (heavily based on HDFS code, but
with distinct names), we are seeing crashes in hdfsThreadDestructor() in threads that use
both HDFS and JNI codepaths. The crash shows up under concurrency and does not reproduce in
serial execution.
> I have distilled it down to a simple testcase that reproduces the issue. It creates a
JVM in the main thread (which Impala does at startup), then spawns multiple threads that do
basic HDFS and JNI work. I have removed all but the essential steps. 
> This blocks running Impala on any hadoop version past 2.7 (when HDFS-7879 was merged).
Note that exposing getJNIEnv should unblock Impala development if a fix is not forthcoming.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message