Return-Path: X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 29B00175CA for ; Thu, 12 Mar 2015 16:43:47 +0000 (UTC) Received: (qmail 48205 invoked by uid 500); 12 Mar 2015 16:43:39 -0000 Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org Received: (qmail 48159 invoked by uid 500); 12 Mar 2015 16:43:38 -0000 Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: common-issues@hadoop.apache.org Delivered-To: mailing list common-issues@hadoop.apache.org Received: (qmail 48145 invoked by uid 99); 12 Mar 2015 16:43:38 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 Mar 2015 16:43:38 +0000 Date: Thu, 12 Mar 2015 16:43:38 +0000 (UTC) From: "Xiaoyu Yao (JIRA)" To: common-issues@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (HADOOP-11708) CryptoOutputStream synchronization differences from DFSOutputStream break HBase MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HADOOP-11708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14358943#comment-14358943 ] Xiaoyu Yao commented on HADOOP-11708: ------------------------------------- [~busbey], we found the same issue a couple of days ago and [~hitliuyi] is working on a fix under [HDFS-7911|https://issues.apache.org/jira/browse/HDFS-7911] . > CryptoOutputStream synchronization differences from DFSOutputStream break HBase > ------------------------------------------------------------------------------- > > Key: HADOOP-11708 > URL: https://issues.apache.org/jira/browse/HADOOP-11708 > Project: Hadoop Common > Issue Type: Bug > Components: fs > Affects Versions: 2.6.0 > Reporter: Sean Busbey > Assignee: Sean Busbey > Priority: Critical > > For the write-ahead-log, HBase writes to DFS from a single thread and sends sync/flush/hflush from a configurable number of other threads (default 5). > FSDataOutputStream does not document anything about being thread safe, and it is not thread safe for concurrent writes. > However, DFSOutputStream is thread safe for concurrent writes + syncs. When it is the stream FSDataOutputStream wraps, the combination is threadsafe for 1 writer and multiple syncs (the exact behavior HBase relies on). > When HDFS Transparent Encryption is turned on, CryptoOutputStream is inserted between FSDataOutputStream and DFSOutputStream. It is proactively labeled as not thread safe, and this composition is not thread safe for any operations. -- This message was sent by Atlassian JIRA (v6.3.4#6332)