hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Nauroth (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-9724) WebHDFS listing is too slow after HDFS-6565
Date Tue, 02 Feb 2016 16:59:39 GMT

    [ https://issues.apache.org/jira/browse/HDFS-9724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15128552#comment-15128552

Chris Nauroth commented on HDFS-9724:

bq. ...yes, as long as you always configure instance before use...

Most of the {{ObjectMapper}} API is thread-safe after initialization.  That means that if
we want to call methods like {{ObjectMapper#enable}} and {{ObjectMapper#disable}} to control
how the serialization/deserialization works, then we'd need to do it during some single-threaded
initialization phase, such as process startup.  After that, multiple threads may call the
serialization/deserialization methods concurrently.  I've used Jackson successfully this way
in other multi-threaded codebases.

bq. I'm thinking the method is not reentrant. ObjectMapper uses thread local buffer for parsers
and generators to achieve thread-safety.

I'm curious why the question was brought up as re-entrancy instead of thread-safety.  If we
have a re-entrant call pattern, where the same thread ends up with more than one stack frame
inside the serialization/deserialization methods, then the later call would stomp on the earlier
call's usage of thread-local storage.  AFAIK, we don't have a re-entrant call pattern in WebHDFS
though, and we just need to worry about thread-safety.  Am I missing something?

In addition to the changes in the current patch, I also see one more {{new ObjectMapper()}}
call inside {{WebHdfsFileSystem}}.  We probably need a comprehensive review across the whole
codebase to look for this pattern, but we can keep the current JIRA's scope focused on WebHDFS,
since we know it is causing a performance problem there.

> WebHDFS listing is too slow after HDFS-6565
> -------------------------------------------
>                 Key: HDFS-9724
>                 URL: https://issues.apache.org/jira/browse/HDFS-9724
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: performance
>    Affects Versions: 2.7.0
>            Reporter: Akira AJISAKA
>            Assignee: Akira AJISAKA
>            Priority: Critical
>         Attachments: HDFS-9724-branch-2.7.01.patch, HDFS-9724.01.patch
> Eremikhin Alexey reported at user ML that WebHDFS listing is too slow because of HDFS-6565.
> https://mail-archives.apache.org/mod_mbox/hadoop-hdfs-user/201601.mbox/%3C56AB857B.6020105%40corp.badoo.com%3E

This message was sent by Atlassian JIRA

View raw message