accumulo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Josh Elser <josh.el...@gmail.com>
Subject Re: Unable to get Mini to use native maps - 1.6.2
Date Tue, 23 Feb 2016 19:18:02 GMT
It would be really helpful if you could write up a minimal Groovy 
project that we can run that is doing exactly what you're trying.

I'm personally not sure what else to tell you: there are specific 
methods exposed which set up the native maps for MAC and these are 
exercised by our internal tests. If there's something we can pull 
down/run, it'll be much easier for us to provide more help/recommendations.

Dan Blum wrote:
> I understand, but I should be seeing the same configuration as the MAC
> process - or if not, why would it be different?
>
> The MAC logs have mostly been what I have been looking at. As noted, they
> have logging from NativeMap at all, which means it isn't even trying to load
> the library. Granted I have to make sure it can find the library, but that
> doesn't help if it never looks for it at all.
>
> -----Original Message-----
> From: Josh Elser [mailto:josh.elser@gmail.com]
> Sent: Tuesday, February 23, 2016 2:03 PM
> To: user@accumulo.apache.org
> Subject: Re: Unable to get Mini to use native maps - 1.6.2
>
> MiniAccumuloCluster spawns its own processes, though. Calling
> NativeMap.isLoaded() in your test JVM isn't proving anything.
>
> That's why you need to call these methods on MAC, you would need to
> check the TabletServer*.log file(s), and make sure that its
> configuration is set up properly to find the .so.
>
> Does that make sense? Did I misinterpret you?
>
> Dan Blum wrote:
>> I'll see what I can do, but there's no simple way to pull out something
>> small we can share (and it would have to be a gradle project).
>>
>> I confirmed that the path is not the immediate issue by adding an explicit
>> call to NativeMap.isLoaded() at the start of my test - that produces
> logging
>> from NativeMap saying it can't find the library, which is what I expect.
>> Without this call NativeMap still logs nothing so the setting that should
>> cause it to be referenced is getting overridden somewhere. Calling
>> InstanceOperations.getSiteConfiguration and getSystemConfiguration shows
>> that the native maps are enabled, however.
>>
>> -----Original Message-----
>> From: Josh Elser [mailto:josh.elser@gmail.com]
>> Sent: Tuesday, February 23, 2016 12:56 PM
>> To: user@accumulo.apache.org
>> Subject: Re: Unable to get Mini to use native maps - 1.6.2
>>
>> Well, I'm near positive that 1.6.2 had native maps working, so there
>> must be something unexpected happening :). MAC should be very close to
>> what a real standalone instance is doing -- if you have the ability to
>> share some end-to-end project with where you are seeing this, that'd be
>> extremely helpful (e.g. a Maven project that we can just run would be
>> superb).
>>
>> Dan Blum wrote:
>>> I'll take a look but I don't think the path is the problem - NativeMap
>>> should try to load the library regardless of whether this path is set and
>>> will log if it can't find it. This isn't happening.
>>>
>>> -----Original Message-----
>>> From: Josh Elser [mailto:josh.elser@gmail.com]
>>> Sent: Tuesday, February 23, 2016 12:27 PM
>>> To: user@accumulo.apache.org
>>> Subject: Re: Unable to get Mini to use native maps - 1.6.2
>>>
>>> Hi Dan,
>>>
>>> I'm seeing in our internal integration tests that we have some
>>> configuration happening which (at least, intends to) configure the
>>> native maps for the minicluster.
>>>
>>> If you're not familiar, the MiniAccumuloConfig and MiniAccumuloCluster
>>> classes are thin wrappers around MiniAccumuloConfigImpl and
>>> MiniAccumuloClusterImpl. There is a setNativeLibPaths method on
>>> MiniAccumuloConfigImpl which you can use to provide the path to the
>>> native library shared object (.so). You will probably have to switch
>>> from MiniAccumuloConfig/MiniAccumuloCluster to
>>> MiniAccumuloConfigImpl/MiniAccumuloClusterImpl to use the "hidden"
>> methods.
>>> You could also look at MiniClusterHarness.java in>=1.7 if you want a
>>> concrete example of how we initialize things for our tests.
>>>
>>> - Josh
>>>
>>> Dan Blum wrote:
>>>> In order to test to make sure we don't have more code that needs a
>>>> workaround for https://issues.apache.org/jira/browse/ACCUMULO-4148 I am
>>>> trying again to enable the native maps for Mini, which we use for
>> testing.
>>>> I set tserver.memory.maps.native.enabled to true in the site XML, and
>> this
>>>> is getting picked up since I see this in the Mini logs:
>>>>
>>>> [server.Accumulo] INFO : tserver.memory.maps.native.enabled = true
>>>>
>>>> However, NativeMap should log something when it tries to load the
>> library,
>>>> whether it succeeds or fails, but it logs nothing. The obvious
> conclusion
>>> is
>>>> that something about how MiniAccumuloCluster starts means that this
>>> setting
>>>> is ignored or overridden, but I am not finding it. (I see the mergeProp
>>> call
>>>> in MiniAccumuloConfigImpl.initialize which will set
>>> TSERV_NATIVEMAP_ENABLED
>>>> to false, but that should only set it if it's not already in the
>>> properties,
>>>> which it should be, and as far as I can tell the log message above is
>>> issued
>>>> after this.)
>>>>
>

Mime
View raw message