accumulo-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From mvangeert...@comcast.net
Subject Re: Camel Accumulo
Date Wed, 26 Oct 2011 15:08:10 GMT


Jesse, 



The only difference between a bundle and a .jar is the inclusion of additional fields in the
MANIFEST.MF file. I think there's a way to allow the maven-bundle-plugin to create this file,
and then to include it into a .jar file created by the maven-jar-plugin. Basically, after
the code is compiled, the maven-bundle-plugin goes through all of the compiled .class files,
and creates a set of Import-Package statements. Then, you can either have the bundle plugin
export everything automatically, or specificy the packages you want to export. 



If you'd like, tonight I can implement this in the parent pom.xml file and submit a patch
for your review. 



Also, did anyone take a look at the Jira ticket I created last night referencing exec plugin
attempting to find /bin/bash?  I'd really like to be able to compile accumulo and get it
working in a stand-alone mode with a single hadoop and zookeeper instance to allow me to test
the camel-accumulo component once it is complete. 



Lastly, do you folks envision creating a "sandbox" area in your svn repository?  This would
allow me to commit the camel component along with examples and tests to allow you folks to
review it. 



----- Original Message -----


From: "Jesse Yates" <jesse.k.yates@gmail.com> 
To: accumulo-dev@incubator.apache.org 
Sent: Wednesday, October 26, 2011 10:37:16 AM 
Subject: Re: Camel Accumulo 

Comments inline. 
------------------- 
Jesse Yates 
240-888-2200 
@jesse_yates 

On Wed, Oct 26, 2011 at 6:39 AM, <mvangeertruy@comcast.net> wrote: 

> 
> 
> Eric, 
> 
> 
> 
> I'm confused about the need for a username/password. Does Accumulo validate 
> against that, or does it accept a set of priveledges for a user? 


It checks that user and password against an internal db of users, giving the 
user ability to read from a subset of the permissions on the system 
('Authorizations' in Accumulo parlance). 


> Also, in your discussion of zookeeper, what does the "zookeepers" attribute 
> represent? 
> 

The address of the zookeeper servers keeping track of the accumulo instance. 
This is the 'root' truth of where things are for clients. 


> 
> 
> 
> For my first stab at this, I'm using the ReadandWrite example to show me 
> how to connect to Accumulo and perform reads/writes .  Is there a better one 
> that I should be using?  This example class seems pretty heavy-weight... 
> 
> 
It might be a little bit heavy, but I would rather have a complete example, 
than a bunch of scattered sub-pieces. I think once you have the connection 
setup, the actual read/write shouldn't be too bad. 


> 
> 
> For the table name, I'd like to keep the camel component as generic as 
> possible. So, to allow for this, I'm going to require folks to pass in the 
> table name  on their accumulate url. Is this ok? 
> 
> 
> 
> Full Ack on the "accumulate" versus "accum" uri.  The only reason I 
> attempted to shorten it is that looks like the trend for other 
> camel-components.  That said, I can live with a longer name if it makes it 
> easier to use.  Along these lines, I've thought that having a different uri 
> for reads, writes, and mutates would be an easy way to handle interactions 
> through the camel component. To help with this, I'm thinking that he 
> following uri's would be use ful (developed in this order) : 
> 
> accumulate-read 
> 
> accumulate-write 
> 
> accumulate-mutate 
> 
> accumulate-batch-read 
> 
> accumulate-batch-write 
> 
> 
> 
> How does that sound? 
> 

shouldn't this be 'accumulo' or is this for implying accumulation of 
information via Camel? 


> 
> 
> 
> Also, how do you feel about changing the build so that it creates bundles 
> instead of .jar files?  All a bundle is, is a .jar file with a MANIFEST.MF 
> file that has some extra attributes.  The maven-bundle-plugin can take care 
> of that pretty easily, but it will change the packaging type from jar to 
> bundle.  This would really only impact folks who are looking for a packaging 
> type in thier .pom file dependencies, which is not common. 
> 
> 
I would argue vehemently against changing the core build for an external 
process (no offense mike). I think it would be okay if we have a profile in 
the build that builds a bundle, but since that is a pretty uncommon distro, 
people will easily get confused, leading to lower adoption, lots of 
questions on user@, etc. Yes, its a low likelyhood, but I would rather make 
it is as easy as possible to 'do the right thing'. 

Camel (and other systems that need a bundle) are a special case, so already 
people are going to be be a little more advanced and expect slightly 
'special' versions. 



> 
> 
> Mike Van 
> 
> 
> 
> ----- Original Message ----- 
> 
> 
> From: "Eric Newton" <eric.newton@gmail.com> 
> To: accumulo-dev@incubator.apache.org 
> Sent: Wednesday, October 26, 2011 9:06:19 AM 
> Subject: Re: Camel Accumulo 
> 
> Hi Mike, 
> 
> Scan auths are needed for reading, not for writing. 
> 
> Looking at constructors and factory methods looks like we'll need: 
> 
> ZooKeeperInstance: 
>     zookeepers 
>     instance name 
> 
> getConnector(): 
>     username 
>     password 
> 
> getMultiTableBatchWriter(): 
>      memory 
>      latency 
>      write threads 
> 
> This would allow you to stream in tuples of (table, row, cf, cq, 
> visibility, 
> timestamp).  If you already have a connector for HBase, we would probably 
> want something compatible with that, and that is probably table oriented 
> (just guessing, being unfamiliar with Camel).  So, perhaps if you specify a 
> table you would accept tuples of (row, cf, cq, visibility, timestamp).  You 
> could make visibility and timestamp optional, too. 
> 
> A minimal uri would look something like this?  You could provide reasonable 
> defaults for the other arguments. 
> 
> 
> accumulo://instance/write?zookeeper=zoohost&userName=root&password=secret 
> 
> Unfortunately, "accumulo" doesn't have a nice short abbreviation.  I would 
> lean towards using the whole word, but that's just a personal preference. 
> 
> -Eric 
> 
> On Wed, Oct 26, 2011 at 12:17 AM, <mvangeertruy@comcast.net> wrote: 
> 
> > 
> > 
> > Hi there. 
> > 
> > 
> > 
> > I recently read about your project and like the direction it is taking. 
> > Currently, I am a committer to another incubator project, Kalumet, and 
> have 
> > been a long-time contributor to the Karaf and Camel projects. After 
> talking 
> > with a few of the Accumulo project members, it looks like the most 
> immediate 
> > hurdle is growing the user community.  I believe I can help with that. 
> > 
> > 
> > 
> > Making associations between incubator projects and top-level-projects has 
> > been a proven mechanism to pique developer interest and garner more 
> > dedicated contributors and commiters.  Because of the wide integration of 
> > Camel and NOSQL databases, creating a Camel component for interaction 
> with 
> > Accumulo seems like a no-brainer. 
> > 
> > 
> > 
> > In order to help grow the Accumulo community, tonight I began writing a 
> > camel-accumulo component.  This will allow Camel users to route files to 
> > Accumulo in the same manner as they currently route files to HDFS . 
> > 
> > 
> > 
> > 
> > 
> > For some background, Camel is the open-source implementation of 
> Enterprise 
> > Integration Patterns. Most modern ESB's use OSGI and Camel to perform 
> > routing and orchestration of data through endpoints. Camel has been 
> written 
> > to allow various technologies to create Camel Components that folks can 
> use 
> > when they define the route that a given file or data will be processed 
> > through. In this model, users would define a "route" in Camel that 
> contains 
> > various Accumulo endpoints for reading, writing, or mutating data 
> persisted 
> > through Accumulo. To make this work, I need to define a URI that folks 
> will 
> > use.  Would you folks be able to help me define the URI and URL 
> parameters? 
> > 
> > 
> > 
> > Right now, I'm using the URI "accum".  For the first iteration of this 
> > component, I'm thinking it would be simplest to create an endpoint folks 
> > could write a single re cord to. Then, follow it up with a scan, and 
> mutate 
> >  components. Once these are done, I'd like to do the batch-versions of 
> these 
> > operations. 
> > 
> > 
> > 
> > In Camel, an endpoint usually looks similar to a web-service endpoint: 
> > URI://location/service ?[arg1=value1][&arg<x>=value<x>] 
> > 
> > 
> > 
> > With this in mind, I'm thinking the following would be the minimally 
> > acceptable Camel-Accumulo endpoint for simplistic write  operations: 
> > 
> > accum://location /write ?zookeeperName=value&\ 
> > 
> > tableName=value& \ 
> > 
> > userName=value&\ 
> > 
> > password=value&\ 
> > 
> > userPrivs=value&\ 
> > 
> > scanAuths=value&\ 
> > 
> > debug=value 
> > 
> > 
> > 
> > Does this contain a c omplete listing of the properties? A m I missing 
> > anything, did I put something in that's not needed, or are there other 
> > options a user should be able to pass in? Also, is the URI of "accum" ok 
> for 
> > this camel component? 
> > 
> > 
> > 
> > Because Camel is written to play nicely inside of OSGI (like 
> Karaf/Felix), 
> > the .jar files this camel component relies on should be bundle-ized. This 
> > shouldn't be too hard to do, and as a Karaf contributor, I've done this 
> with 
> > hundreds of third party .jar files. Basically, we would replace the 
> > maven-jar-plugin with a light implementation of the maven-bundle-plugin 
> > along with some fairly generic attributes.  If you folks would like, I 
> can 
> > do this for you on a seperate branch so that you can test it. 
> > 
> > 
> > 
> > Mike Van 
> > 
> > Committer - ASF Kalumet 
> > 
> > Contributor - ASF Karaf, Camel 
> 

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message