hawq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vineet Goel <vvin...@apache.org>
Subject HAWQ and Azure blob storage
Date Thu, 18 Aug 2016 03:30:31 GMT
Recently, a question came up whether HAWQ can support Azure with it's blob
storage (WASB), given that WASB is HDFS compatible. I wanted to get the
developer community's thoughts on HAWQ compatibility. It seems to me that
HAWQ will not work as-is with WASB, right?

Some investigation and discussion:
Windows Azure Storage Blob (WASB) is an extension built on top of the HDFS
APIs. Upon further investigation, I was told that WASB added support for
append relatively recently in HADOOP-12635
<https://issues.apache.org/jira/browse/HADOOP-12635>, but with limitations
in its semantics as compared to HDFS.  Unlike HDFS, it does not enforce a
single-writer guarantee.  Instead, responsibility is pushed to the
application to guarantee mutually exclusive access to the file being
appended.  Failure to do so can result in data loss or corruption.  If HAWQ
relies on the traditional HDFS single-writer semantics, then WASB’s append
implementation won’t be suitable. Also, WASB has no support for HDFS
Truncate.  Attempts to call truncate will fail with an exception.

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message