apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Thomas Weise <tho...@datatorrent.com>
Subject Re: Using HDHT
Date Thu, 29 Oct 2015 15:09:51 GMT
Hi Gyula,

It is a key-value store, embedded into the Apache Apex operator model. You
can read more about it here:

https://www.datatorrent.com/data-store-for-scalable-stream-processing/

The underlying abstraction are block indexed files with ordered keys in
HDFS.

Please note that post 2.x version this operator is supported as DT RTS
extension only, but you can look at the code as of 2.x here:

https://github.com/apache/incubator-apex-malhar/tree/release-2.1/contrib/src/main/java/com/datatorrent/contrib/hdht

Thanks,
Thomas



On Thu, Oct 29, 2015 at 7:00 AM, Gyula Fóra <gyfora@apache.org> wrote:

> Hey guys,
>
> I am very interested in learning a little bit more about the HDHT
> implementation that I found here
> <
> https://github.com/apache/incubator-apex-malhar/tree/hdhtOrcExport/contrib/src/main/java/com/datatorrent/contrib/hdht
> >
> .
>
> Could you please point me to some examples using this feature? I am
> actually interested in using this logic outside of the streaming context,
> just to store multiple elements per key for up to billions of keys on HDFS.
>
> Is there some more documentation on this somewhere?
>
> Cheers,
> Gyula
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message