accumulo-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jesse Yates <jesse.k.ya...@gmail.com>
Subject Re: Camel Accumulo
Date Wed, 26 Oct 2011 14:37:16 GMT
Comments inline.
-------------------
Jesse Yates
240-888-2200
@jesse_yates

On Wed, Oct 26, 2011 at 6:39 AM, <mvangeertruy@comcast.net> wrote:

>
>
> Eric,
>
>
>
> I'm confused about the need for a username/password. Does Accumulo validate
> against that, or does it accept a set of priveledges for a user?


It checks that user and password against an internal db of users, giving the
user ability to read from a subset of the permissions on the system
('Authorizations' in Accumulo parlance).


> Also, in your discussion of zookeeper, what does the "zookeepers" attribute
> represent?
>

The address of the zookeeper servers keeping track of the accumulo instance.
This is the 'root' truth of where things are for clients.


>
>
>
> For my first stab at this, I'm using the ReadandWrite example to show me
> how to connect to Accumulo and perform reads/writes .  Is there a better one
> that I should be using?  This example class seems pretty heavy-weight...
>
>
It might be a little bit heavy, but I would rather have a complete example,
than a bunch of scattered sub-pieces. I think once you have the connection
setup, the actual read/write shouldn't be too bad.


>
>
> For the table name, I'd like to keep the camel component as generic as
> possible. So, to allow for this, I'm going to require folks to pass in the
> table name  on their accumulate url. Is this ok?
>
>
>
> Full Ack on the "accumulate" versus "accum" uri.  The only reason I
> attempted to shorten it is that looks like the trend for other
> camel-components.  That said, I can live with a longer name if it makes it
> easier to use.  Along these lines, I've thought that having a different uri
> for reads, writes, and mutates would be an easy way to handle interactions
> through the camel component. To help with this, I'm thinking that he
> following uri's would be use ful (developed in this order) :
>
> accumulate-read
>
> accumulate-write
>
> accumulate-mutate
>
> accumulate-batch-read
>
> accumulate-batch-write
>
>
>
> How does that sound?
>

shouldn't this be 'accumulo' or is this for implying accumulation of
information via Camel?


>
>
>
> Also, how do you feel about changing the build so that it creates bundles
> instead of .jar files?  All a bundle is, is a .jar file with a MANIFEST.MF
> file that has some extra attributes.  The maven-bundle-plugin can take care
> of that pretty easily, but it will change the packaging type from jar to
> bundle.  This would really only impact folks who are looking for a packaging
> type in thier .pom file dependencies, which is not common.
>
>
I would argue vehemently against changing the core build for an external
process (no offense mike). I think it would be okay if we have a profile in
the build that builds a bundle, but since that is a pretty uncommon distro,
people will easily get confused, leading to lower adoption, lots of
questions on user@, etc. Yes, its a low likelyhood, but I would rather make
it is as easy as possible to 'do the right thing'.

Camel (and other systems that need a bundle) are a special case, so already
people are going to be be a little more advanced and expect slightly
'special' versions.



>
>
> Mike Van
>
>
>
> ----- Original Message -----
>
>
> From: "Eric Newton" <eric.newton@gmail.com>
> To: accumulo-dev@incubator.apache.org
> Sent: Wednesday, October 26, 2011 9:06:19 AM
> Subject: Re: Camel Accumulo
>
> Hi Mike,
>
> Scan auths are needed for reading, not for writing.
>
> Looking at constructors and factory methods looks like we'll need:
>
> ZooKeeperInstance:
>     zookeepers
>     instance name
>
> getConnector():
>     username
>     password
>
> getMultiTableBatchWriter():
>      memory
>      latency
>      write threads
>
> This would allow you to stream in tuples of (table, row, cf, cq,
> visibility,
> timestamp).  If you already have a connector for HBase, we would probably
> want something compatible with that, and that is probably table oriented
> (just guessing, being unfamiliar with Camel).  So, perhaps if you specify a
> table you would accept tuples of (row, cf, cq, visibility, timestamp).  You
> could make visibility and timestamp optional, too.
>
> A minimal uri would look something like this?  You could provide reasonable
> defaults for the other arguments.
>
>
> accumulo://instance/write?zookeeper=zoohost&userName=root&password=secret
>
> Unfortunately, "accumulo" doesn't have a nice short abbreviation.  I would
> lean towards using the whole word, but that's just a personal preference.
>
> -Eric
>
> On Wed, Oct 26, 2011 at 12:17 AM, <mvangeertruy@comcast.net> wrote:
>
> >
> >
> > Hi there.
> >
> >
> >
> > I recently read about your project and like the direction it is taking.
> > Currently, I am a committer to another incubator project, Kalumet, and
> have
> > been a long-time contributor to the Karaf and Camel projects. After
> talking
> > with a few of the Accumulo project members, it looks like the most
> immediate
> > hurdle is growing the user community.  I believe I can help with that.
> >
> >
> >
> > Making associations between incubator projects and top-level-projects has
> > been a proven mechanism to pique developer interest and garner more
> > dedicated contributors and commiters.  Because of the wide integration of
> > Camel and NOSQL databases, creating a Camel component for interaction
> with
> > Accumulo seems like a no-brainer.
> >
> >
> >
> > In order to help grow the Accumulo community, tonight I began writing a
> > camel-accumulo component.  This will allow Camel users to route files to
> > Accumulo in the same manner as they currently route files to HDFS .
> >
> >
> >
> >
> >
> > For some background, Camel is the open-source implementation of
> Enterprise
> > Integration Patterns. Most modern ESB's use OSGI and Camel to perform
> > routing and orchestration of data through endpoints. Camel has been
> written
> > to allow various technologies to create Camel Components that folks can
> use
> > when they define the route that a given file or data will be processed
> > through. In this model, users would define a "route" in Camel that
> contains
> > various Accumulo endpoints for reading, writing, or mutating data
> persisted
> > through Accumulo. To make this work, I need to define a URI that folks
> will
> > use.  Would you folks be able to help me define the URI and URL
> parameters?
> >
> >
> >
> > Right now, I'm using the URI "accum".  For the first iteration of this
> > component, I'm thinking it would be simplest to create an endpoint folks
> > could write a single re cord to. Then, follow it up with a scan, and
> mutate
> >  components. Once these are done, I'd like to do the batch-versions of
> these
> > operations.
> >
> >
> >
> > In Camel, an endpoint usually looks similar to a web-service endpoint:
> > URI://location/service ?[arg1=value1][&arg<x>=value<x>]
> >
> >
> >
> > With this in mind, I'm thinking the following would be the minimally
> > acceptable Camel-Accumulo endpoint for simplistic write  operations:
> >
> > accum://location /write ?zookeeperName=value&\
> >
> > tableName=value& \
> >
> > userName=value&\
> >
> > password=value&\
> >
> > userPrivs=value&\
> >
> > scanAuths=value&\
> >
> > debug=value
> >
> >
> >
> > Does this contain a c omplete listing of the properties? A m I missing
> > anything, did I put something in that's not needed, or are there other
> > options a user should be able to pass in? Also, is the URI of "accum" ok
> for
> > this camel component?
> >
> >
> >
> > Because Camel is written to play nicely inside of OSGI (like
> Karaf/Felix),
> > the .jar files this camel component relies on should be bundle-ized. This
> > shouldn't be too hard to do, and as a Karaf contributor, I've done this
> with
> > hundreds of third party .jar files. Basically, we would replace the
> > maven-jar-plugin with a light implementation of the maven-bundle-plugin
> > along with some fairly generic attributes.  If you folks would like, I
> can
> > do this for you on a seperate branch so that you can test it.
> >
> >
> >
> > Mike Van
> >
> > Committer - ASF Kalumet
> >
> > Contributor - ASF Karaf, Camel
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message