incubator-cvs mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Apache Wiki <wikidi...@apache.org>
Subject [Incubator Wiki] Update of "OmidProposal" by daijy
Date Thu, 17 Mar 2016 18:19:38 GMT
Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Incubator Wiki" for change notification.

The "OmidProposal" page has been changed by daijy:
https://wiki.apache.org/incubator/OmidProposal?action=diff&rev1=1&rev2=2

   1. [[https://www.youtube.com/watch?v=Rhdmo9pVGgU&index=68&list=PLSAiKuajRe2luyqLU464Nxz4aQe7EPBus|Omid
presentation at Hadoop Summit 2013]]
  
  = Background =
- An Omid prototype was first released as an open-source project back in 2011. Inspired by
Google Percolator [1], it offered a lock-free approach to transactions in NoSQL datastores
(See [2]). However, during these years, the design of Omid has evolved significantly. Whilst
the current open-sourced version maintains many aspects of the original implementation, it
is the result of a major redesign of the first prototype released in 2011. 
+ An Omid prototype was first released as an open-source project back in 2011. Inspired by
Google Percolator [1], it offered a lock-free approach to transactions in NoSQL datastores
(See [2]). However, during these years, the design of Omid has evolved significantly. Whilst
the current open-sourced version maintains many aspects of the original implementation, it
is the result of a major redesign of the first prototype released in 2011.
  
  Omid has now a more decentralized design that does not sacrifice the consistency and performance
of the original version. The current design also enables Omid to scale to thousands of clients
executing transactions concurrently on application data stored in HBase. Internally, Omid
still utilizes a lock-free approach to support multiple concurrent clients. Its design also
relies on a centralized conflict detection component, the TSO, which now resolves in an efficient
manner writeset collisions among concurrent transactions without having to piggyback commit
information to the clients. Another important benefit of Omid is that it doesn't require any
modification of the underlying key-value datastore, HBase in this case. Moreover, the recently
added high availability algorithm allows to eliminate the single point of failure represented
by the TSO in those system deployments requiring a higher degree of dependability. Last but
not least, the provided user API is very simple, mimicking transaction managers in the relational
world: begin, commit, rollback.
  
@@ -31, +31 @@

  = Rationale =
  Programming with ACID (Atomicity, Consistency, Isolation, Durability) transactions is very
popular and it is featured in relational databases. However, in the Big Data ecosystem, applications
typically use NoSQL datastores, which do not provide ACID transactions. Such NoSQL datastores
used to give up transactional support for greater agility and scalability. However, while
early NoSQL data store implementations did not include transaction support, the need for transactions
soon emerged in Big Data applications when accessing shared data; for  example, transactions
are very important  for modern, scalable systems that process content incrementally.
  
- NoSQL datastores -including HBase- don’t provide transactional frameworks to coordinate
the access to the underlying data for preserving consistency. By using Omid, Big Data applications
that need to bundle multiple read and write operations on HBase into logically indivisible
units of work can execute transactions with ACID properties, just as they would use transactions
in the relational database world. Omid extends the HBase key-value access APl with transaction
semantics. It can be exercised either directly, or via higher level data management API’s.
For example, Apache Phoenix (SQL-on-top-of-HBase) might use Omid as its transaction management
component. 
+ NoSQL datastores -including HBase- don’t provide transactional frameworks to coordinate
the access to the underlying data for preserving consistency. By using Omid, Big Data applications
that need to bundle multiple read and write operations on HBase into logically indivisible
units of work can execute transactions with ACID properties, just as they would use transactions
in the relational database world. Omid extends the HBase key-value access APl with transaction
semantics. It can be exercised either directly, or via higher level data management API’s.
For example, Apache Phoenix (SQL-on-top-of-HBase) might use Omid as its transaction management
component.
  
  The following features make Omid an attractive choice for system designers and other projects
in the Apache community:
  
@@ -39, +39 @@

  
   * Performance and Scalability. Omid  provides a highly scalable, lock-free implementation
of SI. To the best of our knowledge, it is also one of the few open source NoSQL transactional
platforms that can execute more than 100K transactions per second [1]. A new prototype still
in development can go even further, up to ~380K TPS.
  
-  * Reliability.  Omid has a high-availability (HA) mode, in which the core service performing
writeset conflict resolution operates as primary-backup process pair with automatic failover.
The HA support has zero overhead on the mainstream operation.  
+  * Reliability.  Omid has a high-availability (HA) mode, in which the core service performing
writeset conflict resolution operates as primary-backup process pair with automatic failover.
The HA support has zero overhead on the mainstream operation.
  
   * Adaptability. Omid current version provides transactions on data stored in Apache HBase.
However, Omid’s components are generic enough to be adapted to any other key-value NoSQL
datasource that supports MVCC.
  
-  * Development. Omid provides a very simple interface that mimics standard HBase APIs, making
it developer friendly. Only minimal extensions to the standard interfaces have been introduced
to enable transactions.  
+  * Development. Omid provides a very simple interface that mimics standard HBase APIs, making
it developer friendly. Only minimal extensions to the standard interfaces have been introduced
to enable transactions.
  
   * Simplicity. Omid leverages the HBase infrastructure for managing its own metadata. It
entails no additional services apart from those provided and used by HBase.
  
-  * Track Record. As we have mentioned, Omid is already in use by very-large-scale production
systems at Yahoo. Also, Hortonworks is integrating Omid in a metastore implementation for
Hive based on HBase.
+  * Track Record. As we have mentioned, Omid is already in use by very-large-scale production
systems at Yahoo. Also, Hortonworks is integrating Omid in a metastore implementation for
Hive based on HBase
  
  
-  1. See also [[https://github.com/vcnc/haeinsa/wiki/Performance|Haeinsa]]
+ [1] See also [[https://github.com/vcnc/haeinsa/wiki/Performance|Haeinsa]]
  
  = Current Status =
  Current Omid implementation is available in both, Yahoo’s internal Github repository for
internal use at Yahoo as well as in [[https://github.com/yahoo/omid.git|Yahoo’s Github public
repository]]. Both repositories are managed by Omid’s current developers at Yahoo.
  
- 
  As it is mentioned above, Yahoo is currently using Omid for providing transactions in Sieve,
a web-scale content management platform that powers Yahoo’s next-generation search and personalization
products.
  
  == Meritocracy ==
- The first version of Omid was originally created in 2011 by Maysam Yabandeh, Daniel Gomez-Ferro,
Ivan B. Kelly, Benjamin Reed and Flavio Junqueira at the R&D Scalable Computing Group
of Yahoo Labs in Spain. 
+ The first version of Omid was originally created in 2011 by Maysam Yabandeh, Daniel Gomez-Ferro,
Ivan B. Kelly, Benjamin Reed and Flavio Junqueira at the R&D Scalable Computing Group
of Yahoo Labs in Spain.
  
  During the years after its inception, Omid has matured to operate at Web scale and has been
used internally by strategic projects at Yahoo such as Sieve. The current base of committers
belong to the Yahoo team that took over the initial Omid prototype and rewrote it to meet
the high availability and scalability requirements of the Sieve project. This base of committers
has recently incorporated Hortonworks members that helped in the Omid adaptation to HBase
1.x versions.
  
@@ -79, +78 @@

  
  The current set of developers include experienced committers from Apache HBase, Hive and
Hadoop projects that have been working with us in the current codebase found in Github.
  
- 
  Finally, some of the core developers are currently NOT affiliated with the ASF and would
require new ICLAs to be filed.
  
  = Alignment =
  Omid enhances with transactions the already successful Apache HBase datastore project. We
have collaborated with other developers inside and outside Yahoo which are involved in the
Apache HBase community, so we have had reliable feedback from them.
- 
  
  Although Omid brings value into HBase, the design of the current version provides a general
transaction scheme that can potentially be adapted to other MVCC key-value datastores such
as Apache Cassandra.
  
@@ -101, +98 @@

  Omid has been supported by Yahoo since its inception in 2011. All current committers are
employed at Yahoo or Hortonworks.
  
  == Reliance on Salaried Developers ==
- All the current developers are paid by their employers to contribute to this project. Yahoo
developers will also continuing maintaining the internal Omid repository at their company.
+ All the current developers are paid by their employers to contribute to this project. Yahoo
developers will also continuing maintaining the internal Omid repository at their company.
Of course, other developers are welcomed to contribute to this project after it is open sourced
in Apache.
- Of course, other developers are welcomed to contribute to this project after it is open
sourced in Apache.
  
  == Relationships with Other Apache Product ==
  Current Omid incarnation serves transactional contexts to applications storing their data
in HBase. However Omid design potentially allows to be adapted to serve transactions on top
of other MVCC-based key-value datastores in Apache community such as Cassandra.
@@ -120, +116 @@

  Current documentation about the project is available in the wiki of Omid’s [[https://github.com/yahoo/omid/wiki|Github
repository]]. It will be moved under https://omid.incubator.apache.org/docs if the project
is accepted as an Apache Incubator.
  
  = Initial Source =
- Initial source code is currently hosted in Github for general viewing and contribution:
+ Initial source code is currently hosted in Github for general viewing and contribution:
https://github.com/yahoo/omid.git
- https://github.com/yahoo/omid.git
  
  Omid source code is written in Java code (99%) mixed with some shell script (1%) in order
to configure and trigger the execution of main components.
  
@@ -131, +126 @@

  The current Omid License for the code published in Github is Apache 2.0. If Omid fulfills
and passes the conditions for being an Incubator project in the ASF, the source code will
be transitioned via the Software Grant Agreement onto the ASF infrastructure and in turn made
available under the Apache License, version 2.0.
  
  = External Dependencies =
- 
  The required external dependencies that are not Apache projects are all Apache licenses
or other compatible Licenses:
  
   * Maven & Maven plugins (http://maven.apache.org/) [Apache 2.0]
   * JDK7 or OpenJDK 7 (http://java.com/) [Oracle or Openjdk JDK License]
   * Google Guava v11.0.2 (https://github.com/google/guava) [Apache 2.0]
   * Google Guice v3.0 (https://github.com/google/guice/wiki) [Apache 2.0]
-  * Testng v6.8.8  (http://testng.org) [Apache 2.0] 
+  * Testng v6.8.8  (http://testng.org) [Apache 2.0]
   * SLF4J (http://www.slf4j.org/) v1.7.7 [MIT License]
   * Netty (http://netty.io) v3.2.6.Final [Apache 2.0]
   * Google Protocol Buffers v2.5.0 (https://developers.google.com/protocol-buffers/) [BSD
License]
@@ -198, +192 @@

   * Maysam Yabandeh, Dropbox Inc.
  
  = Sponsors =
- 
- 
  == Champion ==
- Daniel Dai, Hortonworks (daijy<AT>hortonworks<DOT>com)
+  * Daniel Dai, Hortonworks (daijy<AT>hortonworks<DOT>com)
  
  == Nominated Mentors ==
   * Alan Gates, Hortonworks, (gates<AT>hortonworks<DOT>com)

---------------------------------------------------------------------
To unsubscribe, e-mail: cvs-unsubscribe@incubator.apache.org
For additional commands, e-mail: cvs-help@incubator.apache.org


Mime
View raw message