hawq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yandong Yao <y...@pivotal.io>
Subject Re: Different storage backends in HAWQ?
Date Fri, 13 Jan 2017 01:53:32 GMT
Supporting Ceph will be very interesting. Could you please create a JIRA?
Would be great if there is any patch.

On Thu, Jan 12, 2017 at 10:19 PM, Dmitry Buzolin <Dmitry.Buzolin@theice.com>
wrote:

> Also, besides supporting 3 different storage interfaces Ceph is more
> sophisticated storage backend compare to Hadoop at this time.
> For example: in addition to replicated pools, Ceph supports erasure coded
> pools  (kind of host based RAID), which has requires lot less storage
> compare to the former.
> Other great features of Ceph is an algorytmic approach to map data to the
> nodes rather than having centrally managed namenodes and snapshots. I don't
> think HDFS offers any of these features. In terms of performance, Ceph
> should be faster than HFDS since it is written on C++ and because it
> doesn't have scalability limitations when mapping data to storage pools,
> compare to Hadoop, where name node is such point of contention.
>
>
> -----Original Message-----
> From: Paul Guo [mailto:paulguo@gmail.com]
> Sent: Wednesday, January 11, 2017 9:15 PM
> To: dev@hawq.incubator.apache.org
> Subject: Re: Different storage backends in HAWQ?
>
> WARNING - External email; exercise caution
>
>
> HAWQ supports row oriented format also (AO Table). For block storage &
> posix fs, I suspect you could use gpdb since gpdb uses local storage though
> probably a bit additional work is needed. For object storage (e.g. S3
> interface), this is an interesting topic. There have been a JIRA for this
> though there are some debates about better solutions.
>
> HAWQ-823 Amazon S3 External Table Support
> https://issues.apache.org/jira/browse/HAWQ-823
>
>
> 2017-01-11 23:14 GMT+08:00 Dmitry Buzolin <Dmitry.Buzolin@theice.com>:
>
> > Since HAWQ only depends on Hadoop and Parquet as a columnar format. I
> > would like to propose having Hawq having pluggable storage backends.
> > Hadoop is already supported but there is Ceph storage backend which
> offers
> > standard Posix compliant file system, object and a block storage. Ceph is
> > also is location aware and written in C++. I believe this is more
> important
> > than porting HAWQ to windows. Your thoughts?
> >
> > Thanks,
> > Dmitry.
> >
> > ________________________________
> >
> > This message may contain confidential information and is intended for
> > specific recipients unless explicitly noted otherwise. If you have reason
> > to believe you are not an intended recipient of this message, please
> delete
> > it and notify the sender. This message may not represent the opinion of
> > Intercontinental Exchange, Inc. (ICE), its subsidiaries or affiliates,
> and
> > does not constitute a contract or guarantee. Unencrypted electronic mail
> is
> > not secure and the recipient of this message is expected to provide
> > safeguards from viruses and pursue alternate means of communication where
> > privacy or a binding message is desired.
> >
>
> ________________________________
>
> This message may contain confidential information and is intended for
> specific recipients unless explicitly noted otherwise. If you have reason
> to believe you are not an intended recipient of this message, please delete
> it and notify the sender. This message may not represent the opinion of
> Intercontinental Exchange, Inc. (ICE), its subsidiaries or affiliates, and
> does not constitute a contract or guarantee. Unencrypted electronic mail is
> not secure and the recipient of this message is expected to provide
> safeguards from viruses and pursue alternate means of communication where
> privacy or a binding message is desired.
>



-- 
Best Regards,
Yandong

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message