directmemory-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Roman Levenstein <romix...@gmail.com>
Subject Re: New backend storage system
Date Fri, 05 Apr 2013 05:35:05 GMT
On Thu, Apr 4, 2013 at 11:52 PM, Raffaele P. Guidi <
raffaele.p.guidi@gmail.com> wrote:

> Hi Chris, Sorry I didn't answer before, but I'm seriously lacking spare
> time lately - too much daylight work and personal issues. Your idea is
> intriguing, but honestly I didn't have a single moment to look into it.
> Also, I have to say that I didn't receive too much feedback after I
> released the unsafe based backend, so I guess we are also lacking overall
> attention on alternative storage systems - or, maybe, on the whole project.
>

Just an idea:
As one possible idea to attract attention to DM (or some related projects),
one could try to implement a few Big Data use-cases using the DM. For
example a backend for a Hadoop, Cassandra storage or something similar, so
that operations can be done in memory. This could provide a significant
speedup for many workloads.

The whole Big Data area (especially Hadoop based systems) seems to be very
hot this days with a lot of attention and press/news coverage. In-memory
computations and storage are also getting hotter. Think about Hana from SAP
and ExoLogic from Oracle. Also GridGain recently announced their Hadoop
backend, which is a drop in replacement and so on. But most of those
solutions are pretty expensive. So, if DM & Co  could show that they can
achieve almost the same for free or much cheaper it could be a very
convincing argument for many people.

Regards,
  -Roman

It's probably largely my fault, as I didn't invest too much attention
> myself. In any case I believe a simpler approach could quite improve on
> performance and maintanability. I'd say go on experimenting, and see what
> comes up :)
>
> Ciao,
>     R
> Il giorno 04/apr/2013 21:56, "Christoph Engelbert" <noctarius@apache.org>
> ha scritto:
>
> > Is the DM development fallen asleep? Would be bad because it has the
> > potential to be a good competitor to BigMemory or ElasticMemory.
> >
> > Chris
> >
> >
> > Am 02.04.2013 21:30, schrieb Christoph Engelbert:
> > > Hey guys,
> > >
> > > some time ago I started a new small pooled (or unpooled),
> > > partitioned storage implementation for using with ByteBuffer
> > > (Direct, Heap) and Unsafe. It has different selection algorithms for
> > > free partitions / slices (a partition buffer is sliced into smaller
> > > parts). Currently there is a simple RoundRobin selector, one with
> > > ThreadLocal allocation (very similar to the TLAB in the JVM) and one
> > > which uses the id of the currently thread executing cpu core
> > > (ProcessorLocal) which uses OS api (available on Windows / Linux).
> > >
> > > It features a rich SPI to plug in your own selector / partition /
> > > slice implementations so that many parts are easily extendable.
> > >
> > > Maybe we could use some ideas or the storage engine as the backend
> > > engine in DirectMemory.
> > > But as always I'm happy about any comments or suggestions on the
> > > implementation.
> > >
> > > At the moment a lot of documentation / Javadoc is missing but maybe
> > > someone will have a look into it.
> > >
> > > https://github.com/noctarius/direct-ring-cache
> > >
> > > Chris / Noc
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message