ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pavel Kovalenko <jokse...@gmail.com>
Subject Ignite as distributed file storage
Date Sat, 30 Jun 2018 14:24:37 GMT
Igniters,

I would like to start a discussion about designing a new feature because I
think it's time to start making steps towards it.
I noticed, that some of our users have tried to store large homogenous
entries (> 1, 10, 100 Mb/Gb/Tb) to our caches, but without big success.

IGFS project has the possibility to do it, but as for me it has one big
disadvantage - it's in-memory only, so users have a strict size limit of
their data and have data loss problem.

Our durable memory has a possibility to persist a data that doesn't fit to
RAM to disk, but page structure of it is not supposed to store large pieces
of data.

There are a lot of projects of distributed file systems like HDFS,
GlusterFS, etc. But all of them concentrate to implement high-grade file
protocol, rather than user-friendly API which leads to high entry threshold
to start implementing something over it.
We shouldn't go in this way. Our main goal should be providing to user easy
and fast way to use file storage and processing here and now.

If take HDFS as closest possible by functionality project, we have one big
advantage against it. We can use our caches as files metadata storage and
have the infinite possibility to scale it, while HDFS is bounded by
Namenode capacity and has big problems with keeping a large number of files
in the system.

We achieved very good experience with persistence when we developed our
durable memory, and we can couple together it and experience with services,
binary protocol, I/O and start to design a new IEP.

Use cases and features of the project:
1) Storing XML, JSON, BLOB, CLOB, images, videos, text, etc without
overhead and data loss possibility.
2) Easy, pluggable, fast and distributed file processing, transformation
and analysis. (E.g. ImageMagick processor for images transformation,
LuceneIndex for texts, whatever, it's bounded only by your imagination).
3) Scalability out of the box.
4) User-friendly API and minimal steps to start using this storage in
production.

I repeated again, this project is not supposed to be a high-grade
distributed file system with full file protocol support.
This project should primarily focus on target users, which would like to
use it without complex preparation.

As for example, a user can deploy Ignite with such storage and web-server
with REST API as Ignite service and get scalable, performant image server
out of the box which can be accessed using any programming language.

As a far target goal, we should focus on storing and processing a very
large amount of the data like movies, streaming, which is the big trend
today.

I would like to say special thanks to our community members Alexey Stelmak
and Dmitriy Govorukhin which significantly helped me to put together all
pieces of that puzzle.

So, I want to hear your opinions about this proposal.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message