ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From vijayendra bhati <veejayend...@yahoo.com>
Subject Re: Ignite Data Grid with write through mode to HDFS with layer of IGFS
Date Thu, 14 Apr 2016 12:17:07 GMT
Thanks Vladimir !!
The drawback with using HDFS as persistent store behind Ignite cache is how we will take care
of appending single key value pair to HDFS file.Ideally we should use some NoSQL store or
RDBMS as persistent back up behind Ignite cache and then run some scheduled batch to transfer
the data to HDFS as it happens in normal Lambda Architecture.
Now question comes why we want to use Ignite Cache ? Answer is it gives SQL interface that
means we can query on any attribute on the fly.Other we could have used any other NoSQL. But
NoSQL data model is entirely based upon query pattern so to bring the flexibility at the time
of query we think Ignite cache would be better.
For our use case we want to put the latest 2 week data in Ignite cache to meet the latency
requirements and then for any back date get the data from backend persistent storage, for
which we are thinking about HDFS.Thats why we were thinking if we can make Ignite cache write
through cache with HDFS as backed up persistent storage it would serve the purpose.
Please let me know whats your view on this. 
Many thanks,Vij 

    On Wednesday, April 13, 2016 8:58 PM, Vladimir Ozerov <vozerov@gridgain.com> wrote:
 

 Vij,
No, it doesn't. IGFS serves the very different purpose - it is Hadoop-compatible file system.
It means that, for example, you can load data to IGFS and then query it using Hive. But native
Ignite SQL is not applicable here.
Vladimir.
On Wed, Apr 13, 2016 at 3:55 PM, vijayendra bhati <veejayendraa@yahoo.com> wrote:

Thanks Vladimir,
I have not gone through complete documentation but if you could let me know does IGFS provide
SQL support like Ignite cache does ?
Regards,Vij 

    On Wednesday, April 13, 2016 5:54 PM, Vladimir Ozerov <vozerov@gridgain.com> wrote:
 

 Hi Vijayendra,
IGFS is designed to be a distributed file system which could cache data from Hadoop file systems.
It cannot be used as cache store by design.Ignite doesn't have store implementation for HDFS,
so you should implement your own if needed. Particularly, you should implement org.apache.ignite.cache.store.CacheStore
interface.
Vladimir.
On Wed, Apr 13, 2016 at 2:38 PM, vijayendra bhati <veejayendraa@yahoo.com> wrote:

Hi,
Can some body please provide me any pointers regarding how I can use Ignite Data Grid/ In
Memory caching with write through/write behind mode and writing to HDFS ?
I know Ignite provides IGFS but its different from what I am looking for.
The other way could be I can use IGFS as my In Memory store but is it the right approach ?
Regards,Vijayendra Bhati



   



  
Mime
View raw message