hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sean Busbey (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HADOOP-11708) CryptoOutputStream synchronization differences from DFSOutputStream break HBase
Date Thu, 12 Mar 2015 15:09:38 GMT
Sean Busbey created HADOOP-11708:
------------------------------------

             Summary: CryptoOutputStream synchronization differences from DFSOutputStream
break HBase
                 Key: HADOOP-11708
                 URL: https://issues.apache.org/jira/browse/HADOOP-11708
             Project: Hadoop Common
          Issue Type: Bug
          Components: fs
    Affects Versions: 2.6.0
            Reporter: Sean Busbey
            Assignee: Sean Busbey
            Priority: Critical


For the write-ahead-log, HBase writes to DFS from a single thread and sends sync/flush/hflush
from a configurable number of other threads (default 5).

FSDataOutputStream does not document anything about being thread safe, and it is not thread
safe for concurrent writes.

However, DFSOutputStream is thread safe for concurrent writes + syncs. When it is the stream
FSDataOutputStream wraps, the combination is threadsafe for 1 writer and multiple syncs (the
exact behavior HBase relies on).

When HDFS Transparent Encryption is turned on, CryptoOutputStream is inserted between FSDataOutputStream
and DFSOutputStream. It is proactively labeled as not thread safe, and this composition is
not thread safe for any operations.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message