hawq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dmitry Buzolin <Dmitry.Buzo...@theice.com>
Subject RE: Different storage backends in HAWQ?
Date Thu, 12 Jan 2017 14:19:25 GMT
Also, besides supporting 3 different storage interfaces Ceph is more sophisticated storage
backend compare to Hadoop at this time.
For example: in addition to replicated pools, Ceph supports erasure coded pools  (kind of
host based RAID), which has requires lot less storage compare to the former.
Other great features of Ceph is an algorytmic approach to map data to the nodes rather than
having centrally managed namenodes and snapshots. I don't think HDFS offers any of these features.
In terms of performance, Ceph should be faster than HFDS since it is written on C++ and because
it doesn't have scalability limitations when mapping data to storage pools, compare to Hadoop,
where name node is such point of contention.


-----Original Message-----
From: Paul Guo [mailto:paulguo@gmail.com]
Sent: Wednesday, January 11, 2017 9:15 PM
To: dev@hawq.incubator.apache.org
Subject: Re: Different storage backends in HAWQ?

WARNING - External email; exercise caution


HAWQ supports row oriented format also (AO Table). For block storage &
posix fs, I suspect you could use gpdb since gpdb uses local storage though
probably a bit additional work is needed. For object storage (e.g. S3
interface), this is an interesting topic. There have been a JIRA for this
though there are some debates about better solutions.

HAWQ-823 Amazon S3 External Table Support
https://issues.apache.org/jira/browse/HAWQ-823


2017-01-11 23:14 GMT+08:00 Dmitry Buzolin <Dmitry.Buzolin@theice.com>:

> Since HAWQ only depends on Hadoop and Parquet as a columnar format. I
> would like to propose having Hawq having pluggable storage backends.
> Hadoop is already supported but there is Ceph storage backend which offers
> standard Posix compliant file system, object and a block storage. Ceph is
> also is location aware and written in C++. I believe this is more important
> than porting HAWQ to windows. Your thoughts?
>
> Thanks,
> Dmitry.
>
> ________________________________
>
> This message may contain confidential information and is intended for
> specific recipients unless explicitly noted otherwise. If you have reason
> to believe you are not an intended recipient of this message, please delete
> it and notify the sender. This message may not represent the opinion of
> Intercontinental Exchange, Inc. (ICE), its subsidiaries or affiliates, and
> does not constitute a contract or guarantee. Unencrypted electronic mail is
> not secure and the recipient of this message is expected to provide
> safeguards from viruses and pursue alternate means of communication where
> privacy or a binding message is desired.
>

________________________________

This message may contain confidential information and is intended for specific recipients
unless explicitly noted otherwise. If you have reason to believe you are not an intended recipient
of this message, please delete it and notify the sender. This message may not represent the
opinion of Intercontinental Exchange, Inc. (ICE), its subsidiaries or affiliates, and does
not constitute a contract or guarantee. Unencrypted electronic mail is not secure and the
recipient of this message is expected to provide safeguards from viruses and pursue alternate
means of communication where privacy or a binding message is desired.
Mime
View raw message