manifoldcf-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Karl Wright (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CONNECTORS-286) Get ManifoldCF to run on top of a key/value store like Voldemort, for potential massive scalability improvements and speed gains
Date Sun, 14 Aug 2016 11:33:20 GMT

    [ https://issues.apache.org/jira/browse/CONNECTORS-286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15420305#comment-15420305
] 

Karl Wright commented on CONNECTORS-286:
----------------------------------------

Hi [~kamaci],

Experiments were done to figure out if ManifoldCF's queue could be implemented reasonably
on top of a key-value store.  The results were not hopeful; even with totally local (in-memory)
key/value implementation, the performance profile did not come close to what we get with a
SQL database.


> Get ManifoldCF to run on top of a key/value store like Voldemort, for potential massive
scalability improvements and speed gains
> --------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: CONNECTORS-286
>                 URL: https://issues.apache.org/jira/browse/CONNECTORS-286
>             Project: ManifoldCF
>          Issue Type: New Feature
>          Components: Framework core
>            Reporter: Karl Wright
>            Assignee: Karl Wright
>             Fix For: ManifoldCF next
>
>
> ManifoldCF's reliance on a relational database limits its throughput and scalability.
 I am now convinced it is possible to build all the structures we need within a distributed
key-value store like Voldemort, which has the nice side effect of permitting massive scaling.
 I envision there will be several layers to this project, some of which may have broader utility
in the open-source community at large:
> (1) An atomic serialization layer, which adds serialization capabilities to an non-transactional
substrate;
> (2) A transaction layer, which uses atomic serialization to build a notion of light transactions;
> (3) A table and index layer, which defines SQL-like concepts of tables and btree indexes
on top of the transaction layer, via a Java API;
> (4) A generic "database abstraction" layer, which is capable of representing both standard
SQL databases as well as this NoSQL variant, so that ManifoldCF can support both models.
> This is obviously a major development task, and as such is not envisioned to be completed
by the next standard release.  Work will indeed need to be done in a branch.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message