accumulo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Geoffry Roberts <threadedb...@gmail.com>
Subject Re: Accumulo and OSGi
Date Thu, 10 Apr 2014 14:18:28 GMT
I might be interested in taking on the work.  Can you help me find your
patch?

I have a code base already in OSGi that works with MongoDB.  I need to
point it at Accumulo instead, and if I can do so without porting out of
OSGi that would be nice.

You say the community would be well-accepting of bundling up the Accumulo
client.  If that's the case, I'd like to hear from them.


On Wed, Apr 9, 2014 at 2:09 PM, Corey Nolet <cjnolet@gmail.com> wrote:

> Geoffry,
>
>
> Interesting you have Hadoop working in Karaf.  I'm using equinox.
>
> Sure, but are we talking Karaf-specific features here? You just want a
> Hadoop Client bundle that works, right? The author of the Karaf-Hadoop
> project already worked through the classpath issues so it's not a bad
> starting place. Same with the ServiceMix features files.
>
> If I understand you correctly, I only need have Text available.  I'll
> look into that. It does answer my question and maybe I can avoid the JAAS
> mishmash.
>
> Without needing a connection to the FileSystem, can the JAAS stuff be set
> as an optional import?
>
> Close on to 100% of my troubles has always been with getting 3rd party
> libraries working.  Accumulo/Hadoop is only the latest round.
>
> You are getting no arguments from me. I've been using Karaf consistently
> for a few years now and no doubt most of my time maintaining the services
> is spent making sure 3rd party things work (and continue to work after
> updates). I've come to expect it at this point.
>
>
> It's security, scaleability, relationship to Hadoop's HDFS and MR all
> conspire to make it attractive.  But creating an uberbundle?
>
> Sure, packaging all the Accumulo classes into a single bundle got me up
> and running. I still took time wrapping all of its transitive dependencies
> in their own bundles and wiring up the imports/exports, but it works.
>
> As mentioned in previous posts, I think the community would be
> well-excepting of a first-class Accumulo bundle. If you are interested in
> taking on the work, the ticket I posted previously is open and ready for a
> patch.
>
> Both of these were advised by one of the bndtools gurus--neither worked.
>  When I did the Import-Package other things broke.
>
> I had to import several nested packages to get mine to work properly.
> Which other things broke when you did the Import Package of the JAAS
> packaes?
>
>
>
> On Wed, Apr 9, 2014 at 11:49 AM, Geoffry Roberts <threadedblue@gmail.com>wrote:
>
>> Corey,
>>
>>
>> Interesting you have Hadoop working in Karaf.  I'm using equinox.  It
>> also sounds as if I don't need to access HFDS in order to get Accumulo to
>> work in OSGi. If I understand you correctly, I only need have Text
>> available.  I'll look into that. It does answer my question and maybe I
>> can avoid the JAAS mishmash.
>>
>>
>> I have been using OSGi off and on for a few years now. Close on to 100%
>> of my troubles has always been with getting 3rd party libraries working.
>>  Accumulo/Hadoop is only the latest round.
>>
>>
>> I'm gearing to do some major work based on Accumulo.  It's security,
>> scaleability, relationship to Hadoop's HDFS and MR all conspire to make it
>> attractive.  But creating an uberbundle? I'm sure I could get it working as
>> a proof of concept, but will it play in prime-time?
>>
>>
>> How hard would it be to make a proper Accumulo bundle?  Would the
>> community accept it?
>>
>>
>> Thanks
>>
>>
>> So far as my travails with JAAS is concerned, I did this in my bndtools
>> *.bndrun file:
>>
>>
>> -runproperties:
>> org.osgi.framework.system.packages.extra=com.sun.security.auth.module
>>
>> I also tried:
>> Import-Package: com.sun.security.auth.module
>> in my bundle that calls Hadoop.
>>
>> Both of these were advised by one of the bndtools gurus--neither worked.
>>  When I did the Import-Package other things broke.
>>
>> On Wed, Apr 9, 2014 at 9:46 AM, Corey Nolet <cjnolet@gmail.com> wrote:
>>
>>> Geoffry,
>>>
>>> As Josh pointed out, you should only need the Hadoop libraries on the
>>> client side to use the Text object. This means you won't have to go through
>>> the pain of placing the xml files in your root bundles.
>>>
>>> Did you try the JAAS export from the packages in your container? Did
>>> that help?
>>>
>>> I agree with your comment that we *should* be able to look at these
>>> bundles as black boxes but we're not dealing with true bundles, yet. Once I
>>> got my Hadoop client bundle ready to go, I haven't had to touch it (and the
>>> Hadoop Karaf project solved most of the import/export guessing work for me
>>> already).  For Accumulo, on the other hand, it's up to you if you want to
>>> create an uber-bundle with the necessary Accumulo packages or just wrap
>>> each one and provide the necessary imports/exports across them. For my
>>> needs, I just created one uber-bundle for Accumulo packages and imported
>>> the necessary non-Accumulo packages (many of those I had to wrap as well).
>>> I didn't find that part too painful on the Accumulo side. While not being
>>> the ideal situation from the OSGI side, with lack of an existing bundle
>>> artifact, it did solve my problem and I'm actively using both Accumulo and
>>> Hadoop in OSGi. My recommendation would be that until we get a proper
>>> bundle, that solution would certainly work in the short-term.
>>>
>>> I believe Josh posted this already but check out [1]. A ready-to-go OSGi
>>> bundle for Accumulo would be useful but the Hadoop client dependency would
>>> need to be wrapped (or exposed as its own bundle). IMO, with proper
>>> documentation this shouldn't be too painful for users. Thoughts?
>>>
>>>
>>> [1] https://issues.apache.org/jira/browse/ACCUMULO-2518
>>>
>>>
>>> On Mon, Apr 7, 2014 at 11:27 AM, Geoffry Roberts <threadedblue@gmail.com
>>> > wrote:
>>>
>>>> Ahh,  let me try and address where I might have gone off the linguistic
>>>> reservation.
>>>>
>>>> bndtools -- is an eclipse plugin that is very helpful when developing
>>>> OSGi bundles.  It does a lot of grimy, boilerplate things for you.
>>>>
>>>> inlining -- is where one places dependent *.jar files inside the OSGi
>>>> bundle and therefore on said bundle's class path.  It tends to promote
>>>> bloated bundles--not in the spirit of OSGi--but sometimes necessary.
>>>>
>>>> componentizing -- is the business of converting a class into a
>>>> component.  In the bndtools way of doing things, this can be a easy as
>>>> annotating a class with @Component.
>>>>
>>>> bundle -- You probably know what this is already, but I'll include it
>>>> for good measure.  A bundle is a body of code that is on the same class
>>>> path, and often acts as a service to there bundles.
>>>>
>>>> I don't know what could be done upstream other that making Accumulo's
>>>> client OAGI ready.  Would we like to do that?
>>>>
>>>>
>>>> On Mon, Apr 7, 2014 at 11:02 AM, Josh Elser <josh.elser@gmail.com>wrote:
>>>>
>>>>> You just used a lot of words that don't mean anything to me :)
>>>>>
>>>>> Hopefully you don't have to do much on your own. If there are things
>>>>> we can change upstream to make this process easier, please feel free
to let
>>>>> us know.
>>>>>
>>>>>
>>>>> On 4/7/14, 10:55 AM, Geoffry Roberts wrote:
>>>>>
>>>>>> Thanks Josh,
>>>>>>
>>>>>> My container for the moment is equinox, but all should work in Felix
>>>>>> as
>>>>>> well.  I've been using bndtools for my other OSGi work so I'm faced
>>>>>> with
>>>>>> either annotating the Accumulo Code or wrapping it somehow.  What
do
>>>>>> you
>>>>>> want to bet I wind up inlining it?  Still, the annotated (read
>>>>>> componentized) approach would be less kloogy.  I hesitate because
I'd
>>>>>> wind up maintaining my own code line.
>>>>>>
>>>>>>
>>>>>> On Mon, Apr 7, 2014 at 10:28 AM, Josh Elser <josh.elser@gmail.com
>>>>>> <mailto:josh.elser@gmail.com>> wrote:
>>>>>>
>>>>>>     On 4/7/14, 10:07 AM, Geoffry Roberts wrote:
>>>>>>
>>>>>>         My original question remains: Is the Accumulo Client dependent
>>>>>>         on the
>>>>>>         Hadoop Client fully?  This determination can be made through
>>>>>>         trial and
>>>>>>         error.  But I'm looking to leverage OPE (other people's
>>>>>>         experience) if
>>>>>>         it exists.
>>>>>>
>>>>>>
>>>>>>     I thought someone had already said this (but I may be confusing
>>>>>>     threads): the Accumulo API uses Text throughout. Hadoop is a
>>>>>>     required dependency.
>>>>>>
>>>>>>
>>>>>>         In the same spirit, does anyone know if all the following
are
>>>>>>         required
>>>>>>         to run an Accumulo Client? core, fate, start, trace?  If
I
>>>>>>         attempt to
>>>>>>         OSGify, I'm trying to figure how much trouble am I getting
>>>>>> into.
>>>>>>
>>>>>>
>>>>>>     Yes, that should be about it from within Accumulo. You might
need
>>>>>>     some other foss dependencies also available, but I'm not aware
on
>>>>>>     what your "container" (or w/e the proper terminology would be)
>>>>>> provides.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> There are ways and there are ways,
>>>>>>
>>>>>> Geoffry Roberts
>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> There are ways and there are ways,
>>>>
>>>> Geoffry Roberts
>>>>
>>>
>>>
>>
>>
>> --
>> There are ways and there are ways,
>>
>> Geoffry Roberts
>>
>
>


-- 
There are ways and there are ways,

Geoffry Roberts

Mime
View raw message