hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yi Liu (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-7911) Buffer Overflow when running HBase on HDFS Encryption Zone
Date Fri, 13 Mar 2015 01:04:39 GMT

    [ https://issues.apache.org/jira/browse/HDFS-7911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14359720#comment-14359720
] 

Yi Liu commented on HDFS-7911:
------------------------------

As discussed in HADOOP-11708, I +1 for changing CryptoOutputStream to behave the same as HDFS.
We could not make DFSOutputStream or CryptOutputStream synchronized for all methods, that
would affect performance, in most cases, applications should handle the synchronization, so
it's enough we keep the same behave as HDFS.

Sorry that I could not get time working on this JIRA in the past two days for personal reason.
Since Sean Busbey has a patch in HADOOP-11710, I would mark this as duplicated.

> Buffer Overflow when running HBase on HDFS Encryption Zone
> ----------------------------------------------------------
>
>                 Key: HDFS-7911
>                 URL: https://issues.apache.org/jira/browse/HDFS-7911
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: encryption
>    Affects Versions: 2.6.0
>            Reporter: Xiaoyu Yao
>            Assignee: Yi Liu
>            Priority: Blocker
>
> Create an HDFS EZ for HBase under /apps/hbase with some basic testing passed, including
creating tables, listing, adding a few rows, scanning them, etc. However, when doing bulk
load 100's k rows. After 10 minutes or so, we get the following error on the Region Server
that owns the table.
> {code}
> 2015-03-02 10:25:47,784 FATAL [regionserver60020-WAL.AsyncSyncer0] wal.FSHLog: Error
while AsyncSyncer sync, request close of hlog 
> java.io.IOException: java.nio.BufferOverflowException 
> at org.apache.hadoop.crypto.JceAesCtrCryptoCodec$JceAesCtrCipher.process(JceAesCtrCryptoCodec.java:156)
> at org.apache.hadoop.crypto.JceAesCtrCryptoCodec$JceAesCtrCipher.encrypt(JceAesCtrCryptoCodec.java:127)
> at org.apache.hadoop.crypto.CryptoOutputStream.encrypt(CryptoOutputStream.java:162) 
> at org.apache.hadoop.crypto.CryptoOutputStream.flush(CryptoOutputStream.java:232) 
> at org.apache.hadoop.crypto.CryptoOutputStream.hflush(CryptoOutputStream.java:267) 
> at org.apache.hadoop.crypto.CryptoOutputStream.sync(CryptoOutputStream.java:262) 
> at org.apache.hadoop.fs.FSDataOutputStream.sync(FSDataOutputStream.java:123) 
> at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter.sync(ProtobufLogWriter.java:165)

> at org.apache.hadoop.hbase.regionserver.wal.FSHLog$AsyncSyncer.run(FSHLog.java:1241)

> at java.lang.Thread.run(Thread.java:744) 
> Caused by: java.nio.BufferOverflowException 
> at java.nio.DirectByteBuffer.put(DirectByteBuffer.java:357) 
> at javax.crypto.CipherSpi.bufferCrypt(CipherSpi.java:823) 
> at javax.crypto.CipherSpi.engineUpdate(CipherSpi.java:546) 
> at javax.crypto.Cipher.update(Cipher.java:1760) 
> at org.apache.hadoop.crypto.JceAesCtrCryptoCodec$JceAesCtrCipher.process(JceAesCtrCryptoCodec.java:145)
> ... 9 more 
> {code}
> It looks like the HBase WAL  (Write Ahead Log) use case is broken on the CryptoOutputStream().
The use case has one flusher thread that keeps calling the hflush() on WAL file while other
roller threads are trying to write concurrently to that same file handle.
> As the class comments mentioned. *""CryptoOutputStream encrypts data. It is not thread-safe."*
I check the code and it seems the buffer overflow is related to the race between the CryptoOutputStream#write()
and CryptoOutputStream#flush() as both can call CryptoOutputStream#encrypt(). The inBuffer/outBuffer
of the CryptoOutputStream is not thread safe. They can be changed during encrypt for flush()
when write() is coming from other threads. 
> I have validated this with multi-threaded unit tests that mimic the HBase WAL use case.
For file not under encryption zone (*DFSOutputStream*), multi-threaded flusher/writer works
fine. For file under encryption zone (*CryptoOutputStream*), multi-threaded flusher/writer
randomly fails with Buffer Overflow/Underflow.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message