accumulo-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hayden Marchant <HAY...@il.ibm.com>
Subject Re: Running Accumulo on the IBM JVM
Date Mon, 23 Jun 2014 10:00:42 GMT
Josh (and all who commented),

Thanks for the comments. I'll take them into account, and will create the 
JIRAs. 

I was not intending on removing the CMS options, but rather only including 
them in the JVM in which they are relevant, and including the equivalent 
in different JVMs (i.e. IBM ) - all through the bootstrap_config.sh.

Here's my newbie question: Should I be making this patch based on 1.6.1, 
or should I always be working against the 'master' branch, and then 
backport the fix(es) to any desired older version?

Regards,



Hayden



From:   Josh Elser <josh.elser@gmail.com>
To:     dev@accumulo.apache.org, 
Date:   19/06/2014 06:43 PM
Subject:        Re: Running Accumulo on the IBM JVM



<snip/>

>>>          b.
>>>
>>>
>> 
org.apache.accumulo.core.security.crypto.BlockedIOStreamTest.testGiantWrite.
>>>          This test assumes a max heap of about 1GB. This fails on IBM 
JRE,
>>> since the default max heap is not specified, and on IBM JRE this 
depends
>>> on the OS (see
>>>
>>>
>> 
http://www-01.ibm.com/support/knowledgecenter/SSYKE2_6.0.0/com.ibm.java.doc.diagnostics.60/diag/appendixes/defaults.html?lang=en

>>> ).
>>>          Proposal: add -Xmx1g to the surefire maven plugin reference 
in
>>> parent maven pom.
>>>
>>
> This might be https://issues.apache.org/jira/browse/ACCUMULO-2774

Yup! I actually bumped this up to 1G already after I started seeing 
failures (again) from the ACCUMULO-2774 patch which set a 768M heap. 
Pull the upstream changes and feel free to submit something to address 
any problem you still have.

>
>
>>   >         c. Both org.apache.accumulo.core.security.crypto.CrypoTest 
&
>>> org.apache.accumulo.core.file.rfile.RFileTest have lots of failures 
due
>> to
>>> calls to SEcureRandom with Random Number Generator Provider hard-coded 
as
>>> Sun. The IBM JRE has it's own built in RNG Provider called IBMJCE. 2
>>> issues - hard-coded calls to SecureRandom.getInstance(<algo>,"SUN") 
and
>>> also default value in Property class is "SUN".
>>>          Proposal: Add mechanism to override default Property through
>>> System property through new annotator in Property class. Only usage 
will
>>> be by Property.CRYPTO_SECURE_RNG_PROVIDER
>>
>>
>>
> I'm not sure about adding new annotators to Property. However, the
> CryptoTest should be getting the value from the conf instead of 
hard-coding
> it. Then you can specify the correct value in accumulo-site.xml
>
> I think another part of the issue is in
> CryptoModuleFactory::fillParamsObjectFromStringMap because it looks like
> that ignores the default setting.
>
>>   >
>>> 2. Environment/Configuration
>>>          a. The generated configuration files contain references to GC
>>> params that are specific to Sun JVM. In accumulo-env.sh, the
>>> ACCUMULO_TSERVER_OPTS contains -XX:NewSize and -XX:MaxNewSize , and 
also
>>> in ACCUMULO_GENERAL_OPTS,
>>> -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 are 
used.
>>>          b. in bin/accumulo, get ClassNotFoundException due to
>>> specification of JAXP Doc Builder:
>>>
>>>
>> 
-Djavax.xml.parsers.DocumentBuilderFactory=com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderFactoryImpl
>>> .
>>>          The Sun implementation of Document Builder Factory does not
>> exists
>>> in IBM JDK, so a ClassNotFoundException is thrown on running accumulo
>>> script
>>>
>>>          c. MiniAccumuloCluster - in the MiniAccumuloClusterImpl,
>>> Sun-speciifc GC params are passed as params to the java process 
(similar
>>> to section a. )
>>>
>>>          Single proposal for solving all three above issues:
>>>          Enhance bootstrap_config.sh with request to select Java 
vendor.
>>> Selecting this will set correct values for GC params (they differ 
between
>>> IBM and Sun), inclusion/ommision of JAXP setting. The
>>> MiniAccumuloClusterImpl can read the same env variable that was set in
>>> code for the GC Params, and use in the exec command.
>>>
>>
> I don't know enough about the IBM JDK to comment on this part
> intelligently. Go ahead and generate a patch, and we can use that as a
> starting point for discussion.

I'm a little hesitant to remove the CMS configuration (as it really 
helps). My first thought about how to address this is you can submit 
some example Accumulo configurations that work with IBM JDK or you can 
add something to the configuration template/script (conf/examples and 
conf/templates with bin/bootstrap_config.sh, respectively). I think 
you're on the right path.

>
>>   >
>>>   So far, my work has been focused on getting unit tests working for 
all
>>> Java vendors in a clean manner. I have not yet run intensive testing 
of
>>> real clusters following these changes, and would be happy to get 
pointers
>>> to what else might need treatment.
>>
>>
>>
> Unit tests is a good first pass. Integration tests (mvn verify) is 
probably
> the minimum that you want on your continuous integration once you have
> things set up.
>
> Accumulo also comes with a set of longer running, cluster based tests,
> since we know that there are some pieces too complex for unit tests to
> catch. have a look in the test module for the Continuous Ingest test. 
Once
> you get to that point, we can help you set it up if the README is 
unclear.
>
>>   I would also like to hear if these changes make sense, and if so, 
should
>>> I go ahead and create some JIRAs, and attach my patches for commit
>>> approval?
>>>
>>
> Filing JIRAs is going to be the most straightforward path, yes.
>
>   >  Looking forward to hearing feedback!

Likewise. Looking forward to applying some patches!



Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message