accumulo-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From mvangeert...@comcast.net
Subject Re: Camel Accumulo
Date Wed, 26 Oct 2011 13:39:23 GMT


Eric, 



I'm confused about the need for a username/password. Does Accumulo validate against that,
or does it accept a set of priveledges for a user?  Also, in your discussion of zookeeper,
what does the "zookeepers" attribute represent? 



For my first stab at this, I'm using the ReadandWrite example to show me how to connect to
Accumulo and perform reads/writes .  Is there a better one that I should be using?  This
example class seems pretty heavy-weight... 



For the table name, I'd like to keep the camel component as generic as possible. So, to allow
for this, I'm going to require folks to pass in the table name  on their accumulate url.
Is this ok? 



Full Ack on the "accumulate" versus "accum" uri.  The only reason I attempted to shorten
it is that looks like the trend for other camel-components.  That said, I can live with a
longer name if it makes it easier to use.  Along these lines, I've thought that having a
different uri for reads, writes, and mutates would be an easy way to handle interactions through
the camel component. To help with this, I'm thinking that he following uri's would be use
ful (developed in this order) : 

accumulate-read 

accumulate-write 

accumulate-mutate 

accumulate-batch-read 

accumulate-batch-write 



How does that sound? 



Also, how do you feel about changing the build so that it creates bundles instead of .jar
files?  All a bundle is, is a .jar file with a MANIFEST.MF file that has some extra attributes. 
The maven-bundle-plugin can take care of that pretty easily, but it will change the packaging
type from jar to bundle.  This would really only impact folks who are looking for a packaging
type in thier .pom file dependencies, which is not common. 



Mike Van 



----- Original Message -----


From: "Eric Newton" <eric.newton@gmail.com> 
To: accumulo-dev@incubator.apache.org 
Sent: Wednesday, October 26, 2011 9:06:19 AM 
Subject: Re: Camel Accumulo 

Hi Mike, 

Scan auths are needed for reading, not for writing. 

Looking at constructors and factory methods looks like we'll need: 

ZooKeeperInstance: 
    zookeepers 
    instance name 

getConnector(): 
    username 
    password 

getMultiTableBatchWriter(): 
     memory 
     latency 
     write threads 

This would allow you to stream in tuples of (table, row, cf, cq, visibility, 
timestamp).  If you already have a connector for HBase, we would probably 
want something compatible with that, and that is probably table oriented 
(just guessing, being unfamiliar with Camel).  So, perhaps if you specify a 
table you would accept tuples of (row, cf, cq, visibility, timestamp).  You 
could make visibility and timestamp optional, too. 

A minimal uri would look something like this?  You could provide reasonable 
defaults for the other arguments. 

   accumulo://instance/write?zookeeper=zoohost&userName=root&password=secret 

Unfortunately, "accumulo" doesn't have a nice short abbreviation.  I would 
lean towards using the whole word, but that's just a personal preference. 

-Eric 

On Wed, Oct 26, 2011 at 12:17 AM, <mvangeertruy@comcast.net> wrote: 

> 
> 
> Hi there. 
> 
> 
> 
> I recently read about your project and like the direction it is taking. 
> Currently, I am a committer to another incubator project, Kalumet, and have 
> been a long-time contributor to the Karaf and Camel projects. After talking 
> with a few of the Accumulo project members, it looks like the most immediate 
> hurdle is growing the user community.  I believe I can help with that. 
> 
> 
> 
> Making associations between incubator projects and top-level-projects has 
> been a proven mechanism to pique developer interest and garner more 
> dedicated contributors and commiters.  Because of the wide integration of 
> Camel and NOSQL databases, creating a Camel component for interaction with 
> Accumulo seems like a no-brainer. 
> 
> 
> 
> In order to help grow the Accumulo community, tonight I began writing a 
> camel-accumulo component.  This will allow Camel users to route files to 
> Accumulo in the same manner as they currently route files to HDFS . 
> 
> 
> 
> 
> 
> For some background, Camel is the open-source implementation of Enterprise 
> Integration Patterns. Most modern ESB's use OSGI and Camel to perform 
> routing and orchestration of data through endpoints. Camel has been written 
> to allow various technologies to create Camel Components that folks can use 
> when they define the route that a given file or data will be processed 
> through. In this model, users would define a "route" in Camel that contains 
> various Accumulo endpoints for reading, writing, or mutating data persisted 
> through Accumulo. To make this work, I need to define a URI that folks will 
> use.  Would you folks be able to help me define the URI and URL parameters? 
> 
> 
> 
> Right now, I'm using the URI "accum".  For the first iteration of this 
> component, I'm thinking it would be simplest to create an endpoint folks 
> could write a single re cord to. Then, follow it up with a scan, and mutate 
>  components. Once these are done, I'd like to do the batch-versions of these 
> operations. 
> 
> 
> 
> In Camel, an endpoint usually looks similar to a web-service endpoint: 
> URI://location/service ?[arg1=value1][&arg<x>=value<x>] 
> 
> 
> 
> With this in mind, I'm thinking the following would be the minimally 
> acceptable Camel-Accumulo endpoint for simplistic write  operations: 
> 
> accum://location /write ?zookeeperName=value&\ 
> 
> tableName=value& \ 
> 
> userName=value&\ 
> 
> password=value&\ 
> 
> userPrivs=value&\ 
> 
> scanAuths=value&\ 
> 
> debug=value 
> 
> 
> 
> Does this contain a c omplete listing of the properties? A m I missing 
> anything, did I put something in that's not needed, or are there other 
> options a user should be able to pass in? Also, is the URI of "accum" ok for 
> this camel component? 
> 
> 
> 
> Because Camel is written to play nicely inside of OSGI (like Karaf/Felix), 
> the .jar files this camel component relies on should be bundle-ized. This 
> shouldn't be too hard to do, and as a Karaf contributor, I've done this with 
> hundreds of third party .jar files. Basically, we would replace the 
> maven-jar-plugin with a light implementation of the maven-bundle-plugin 
> along with some fairly generic attributes.  If you folks would like, I can 
> do this for you on a seperate branch so that you can test it. 
> 
> 
> 
> Mike Van 
> 
> Committer - ASF Kalumet 
> 
> Contributor - ASF Karaf, Camel 

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message