hawq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Lei Chang <lei_ch...@apache.org>
Subject Re: HAWQ and Azure blob storage
Date Fri, 19 Aug 2016 01:30:28 GMT
I think this is quite related to "[HAWQ-786
<https://issues.apache.org/jira/browse/HAWQ-786>] - Framework to support
pluggable formats and file systems)".

Append should not be a problem since when we do update & delete feature,
anyway we need to do "file merge".

And for single-writer guarantee, currently, hawq enforce this on master
side. we only need to handle some corner cases for example avoiding split
brain issues. Then we can support WASB feature.

Cheers
Lei







On Thu, Aug 18, 2016 at 11:30 AM, Vineet Goel <vvineet@apache.org> wrote:

> Recently, a question came up whether HAWQ can support Azure with it's blob
> storage (WASB), given that WASB is HDFS compatible. I wanted to get the
> developer community's thoughts on HAWQ compatibility. It seems to me that
> HAWQ will not work as-is with WASB, right?
>
> Some investigation and discussion:
> Windows Azure Storage Blob (WASB) is an extension built on top of the HDFS
> APIs. Upon further investigation, I was told that WASB added support for
> append relatively recently in HADOOP-12635
> <https://issues.apache.org/jira/browse/HADOOP-12635>, but with limitations
> in its semantics as compared to HDFS.  Unlike HDFS, it does not enforce a
> single-writer guarantee.  Instead, responsibility is pushed to the
> application to guarantee mutually exclusive access to the file being
> appended.  Failure to do so can result in data loss or corruption.  If HAWQ
> relies on the traditional HDFS single-writer semantics, then WASB’s append
> implementation won’t be suitable. Also, WASB has no support for HDFS
> Truncate.  Attempts to call truncate will fail with an exception.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message