accumulo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From dlmarion <>
Subject RE: [Accumulo Contrib Proposal] Graphulo: Server-side Matrix Math library
Date Fri, 28 Aug 2015 12:08:26 GMT

  I am a little confused about whether you want to place this in the contrib area or whether
you want to create a sub-project as both are mentioned in your proposal. Also, if you intend
for this to be a sub-project, have you looked at the incubator process? From what I understand
given that this is a code contribution,it will have to go through that process.

-------- Original message --------
From: Dylan Hutchison <> 
Date: 08/28/2015  2:43 AM  (GMT-05:00) 
To: Accumulo Dev List <> 
Cc: Accumulo User List <> 
Subject: [Accumulo Contrib Proposal] Graphulo: Server-side Matrix Math library 

Dear Accumulo community,
I humbly ask your consideration of Graphulo as a new contrib project to Accumulo.  Let's
use this thread to discuss what Graphulo is, how it fits into the Accumulo community, where
we can take it together as a new community, and how you can use it right now.  Please see
the README at Graphulo's Github, and for a more in-depth look see the docs/ folder or the
Graphulo is a Java library for the Apache Accumulo database delivering server-side sparse
matrix math primitives that enable higher-level graph algorithms and analytics.
Pitch: Organizations use Accumulo for high performance indexed and distributed data storage. 
What do they do after their data is stored?  Many use cases perform analytics and algorithms
on data in Accumulo, which aside from simple iterators uses, require scanning data out from
Accumulo to a computation engine, only to write computation results back to Accumulo.  Graphulo
enables a class of algorithms to run inside the Accumulo server like a stored procedure, especially
(but not restricted to) those written in the language of graphs and linear algebra.  Take
breadth first search as a simple use case and PageRank as one more complex.  As a stretch
goal, imagine analysts and mathematicians executing PageRank and other high level algorithms
on top of the Graphulo library on top of Accumulo at high performance.
I have developed Graphulo at the MIT Lincoln Laboratory with support from the NSF since last
March.  I owe thanks to Jeremy Kepner, Vijay Gadepally, and Adam Fuchs for high level comments
during design and performance testing phases.  I proposed a now-obsolete design document
last Spring to the Accumulo community too which received good feedback.
The time is ripe for Graphulo to graduate my personal development into larger participation. 
Beyond myself and beyond the Lincoln Laboratory, Graphulo is for the Accumulo community. 
Users need a place where they can interact, developers need a place where they can look, comment,
and debate designs and diffs, and both users and developers need a place where they can interact
and see Graphulo alongside its Accumulo base.  
The following outlines a few reasons why I see contrib to Accumulo as Graphulo's proper place:Establishing
Graphulo as an Apache (sub-)project is a first step toward building a community.  The spirit
of Apache--its mailing list discussions, low barrier to interactions between users and developers
new and old, open meritocracy and more--is a surefire way to bring Graphulo to the people
it will help and the people who want to help it in turn.

Committing to core Accumulo doesn't seem appropriate for all of Graphulo, because Graphulo
uses Accumulo in a specific way (server-side computation) in support of algorithms and applications. 
Parts of Graphulo that are useful for all Accumulo users (not just matrix math for algorithms)
could be transferred from Graphulo to Accumulo, such as ApplyIterator or SmallLargeRowFilter
or DynamicIterator.  

Leaving Graphulo as an external project leaves Graphulo too decoupled from Accumulo.  Graphulo
has potential to drive features in core Accumulo such as ACCUMULO-3978, ACCUMULO-3710, and ACCUMULO-3751. 
By making Graphulo a contrib sub-project, the two can maintain a tight relationship while
still maintaining independent versions.  
Historically, contrib projects have gone into Accumulo contrib and become stale.  I assure
you I do not intend Graphulo this fate.  The Lincoln Laboratory has interests in Graphulo,
and I will continue developing Graphulo at the very least to help transition Graphulo to greater
community involvement.  However, since I will start a PhD program at UW next month, I cannot
make Graphulo a full time job as I have in recent history.  I do have ideas for using Graphulo
as part of my PhD database research.  
Thus, in the case of large community support, I can transition to a support role while others
in the community step up.  If smaller community support, I can continue working on Graphulo
as before at my own pace and perhaps more publicly.  There are only a few steps left before
Graphulo could be considered "finished software": Developing a new interface to Graphulo's
core functions using immutable argument objects, which simplifies developer APIs, increases
generalizability, and facilitates features like asynchronous and parallel operations.  It
would be good if other developers weigh their opinions on designs as we propose them, since
this decides how Graphulo users interact with Graphulo.

Instrumenting Graphulo for monitoring, profiling and benchmarking.  I have a blueprint on
how to use HTrace to make these tasks as easy as browsing a web page.  Needs careful thought
and discussion before implementing, since this instrumentation will go everywhere.  It would
be nice if Graphulo and Accumulo mirror instrumentation strategies, so it would be good to
have that discussion in the same venue.

Rigorous scale testing.  Good instrumentation is key.  With successful scale testing, we
paint a clear picture for which operations Graphulo excels to potential adopters, ultimately
plotting where Graphulo stands in the world of big data software.

Explicitly supporting the GraphBLAS spec, once it is agreed upon.  Graphulo was designed
from the ground up with the GraphBLAS in mind, so this should be an easy task.  Aligning
with this upcoming industry standard bodes well for ease of developing Graphulo algorithms.Developing
more algorithms and applications will follow too, and I imagine this as an excellent place
where newcomer users can get involved.  
Some other places Graphulo needs work worth mentioning are creating a proper release framework
(the release here could use improvement, starting with signed artifacts) and reviewing the
way Graphulo runs tests (currently centered around a critical file called
which is great for one developer, whereas a config file probably works better).  Both of
these are places more experienced developers could help.  I should also mention that Graphulo
has groundwork in place for operations between Accumulo instances, but I doubt many users
would need that level of control.
Regarding IP, I'm happy to donate my commits to the ASF, which covers 99% of the Graphulo
code base.  I'm sure other issues will arise and we can sort them out.  Sean Busbey, perhaps
I could ask your assistance as someone more knowledgeable in this area.  Regarding dependencies, effectively
every direct dependency is org.apache, so nothing to worry about here.
I acknowledge that I will lose dictatorial power and gain some bureaucratic / discussion overhead
by moving from sole developer to an Apache model.  The benefits of a community are well worth
If we as a community decide that contrib is the right place for Graphulo, then there are lots
of logistical questions to decide like where the code will live, where JIRA will live, what
mailing lists to use, what URL to give Graphulo within, etc.  We can tackle these
at our leisure.  Let's discuss Graphulo and Accumulo here first.
Warmly, Dylan Hutchison
View raw message