accumulo-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Josh Elser <josh.el...@gmail.com>
Subject Re: Running Accumulo on the IBM JVM
Date Mon, 23 Jun 2014 13:10:03 GMT
One correction: "... the oldest *active* branch." This would be
1.5.2-SNAPSHOT

I think some things, like the configuration generator, were only introduced
in 1.6.0. Ultimately, you can choose what you'd like to target as long as
the changes aren't invasive/breaking for the bugfix releases (1.5.2 and
1.6.1).
On Jun 23, 2014 8:46 AM, "William Slacum" <wilhelm.von.cloud@accumulo.net>
wrote:

> Work on the oldest branch possible and merge forward, please.
>
>
> On Mon, Jun 23, 2014 at 6:00 AM, Hayden Marchant <HAYDEN@il.ibm.com>
> wrote:
>
> > Josh (and all who commented),
> >
> > Thanks for the comments. I'll take them into account, and will create the
> > JIRAs.
> >
> > I was not intending on removing the CMS options, but rather only
> including
> > them in the JVM in which they are relevant, and including the equivalent
> > in different JVMs (i.e. IBM ) - all through the bootstrap_config.sh.
> >
> > Here's my newbie question: Should I be making this patch based on 1.6.1,
> > or should I always be working against the 'master' branch, and then
> > backport the fix(es) to any desired older version?
> >
> > Regards,
> >
> >
> >
> > Hayden
> >
> >
> >
> > From:   Josh Elser <josh.elser@gmail.com>
> > To:     dev@accumulo.apache.org,
> > Date:   19/06/2014 06:43 PM
> > Subject:        Re: Running Accumulo on the IBM JVM
> >
> >
> >
> > <snip/>
> >
> > >>>          b.
> > >>>
> > >>>
> > >>
> >
> >
> org.apache.accumulo.core.security.crypto.BlockedIOStreamTest.testGiantWrite.
> > >>>          This test assumes a max heap of about 1GB. This fails on IBM
> > JRE,
> > >>> since the default max heap is not specified, and on IBM JRE this
> > depends
> > >>> on the OS (see
> > >>>
> > >>>
> > >>
> >
> >
> http://www-01.ibm.com/support/knowledgecenter/SSYKE2_6.0.0/com.ibm.java.doc.diagnostics.60/diag/appendixes/defaults.html?lang=en
> >
> > >>> ).
> > >>>          Proposal: add -Xmx1g to the surefire maven plugin reference
> > in
> > >>> parent maven pom.
> > >>>
> > >>
> > > This might be https://issues.apache.org/jira/browse/ACCUMULO-2774
> >
> > Yup! I actually bumped this up to 1G already after I started seeing
> > failures (again) from the ACCUMULO-2774 patch which set a 768M heap.
> > Pull the upstream changes and feel free to submit something to address
> > any problem you still have.
> >
> > >
> > >
> > >>   >         c. Both org.apache.accumulo.core.security.crypto.CrypoTest
> > &
> > >>> org.apache.accumulo.core.file.rfile.RFileTest have lots of failures
> > due
> > >> to
> > >>> calls to SEcureRandom with Random Number Generator Provider
> hard-coded
> > as
> > >>> Sun. The IBM JRE has it's own built in RNG Provider called IBMJCE.
2
> > >>> issues - hard-coded calls to SecureRandom.getInstance(<algo>,"SUN")
> > and
> > >>> also default value in Property class is "SUN".
> > >>>          Proposal: Add mechanism to override default Property through
> > >>> System property through new annotator in Property class. Only usage
> > will
> > >>> be by Property.CRYPTO_SECURE_RNG_PROVIDER
> > >>
> > >>
> > >>
> > > I'm not sure about adding new annotators to Property. However, the
> > > CryptoTest should be getting the value from the conf instead of
> > hard-coding
> > > it. Then you can specify the correct value in accumulo-site.xml
> > >
> > > I think another part of the issue is in
> > > CryptoModuleFactory::fillParamsObjectFromStringMap because it looks
> like
> > > that ignores the default setting.
> > >
> > >>   >
> > >>> 2. Environment/Configuration
> > >>>          a. The generated configuration files contain references to
> GC
> > >>> params that are specific to Sun JVM. In accumulo-env.sh, the
> > >>> ACCUMULO_TSERVER_OPTS contains -XX:NewSize and -XX:MaxNewSize , and
> > also
> > >>> in ACCUMULO_GENERAL_OPTS,
> > >>> -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 are
> > used.
> > >>>          b. in bin/accumulo, get ClassNotFoundException due to
> > >>> specification of JAXP Doc Builder:
> > >>>
> > >>>
> > >>
> >
> >
> -Djavax.xml.parsers.DocumentBuilderFactory=com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderFactoryImpl
> > >>> .
> > >>>          The Sun implementation of Document Builder Factory does not
> > >> exists
> > >>> in IBM JDK, so a ClassNotFoundException is thrown on running accumulo
> > >>> script
> > >>>
> > >>>          c. MiniAccumuloCluster - in the MiniAccumuloClusterImpl,
> > >>> Sun-speciifc GC params are passed as params to the java process
> > (similar
> > >>> to section a. )
> > >>>
> > >>>          Single proposal for solving all three above issues:
> > >>>          Enhance bootstrap_config.sh with request to select Java
> > vendor.
> > >>> Selecting this will set correct values for GC params (they differ
> > between
> > >>> IBM and Sun), inclusion/ommision of JAXP setting. The
> > >>> MiniAccumuloClusterImpl can read the same env variable that was set
> in
> > >>> code for the GC Params, and use in the exec command.
> > >>>
> > >>
> > > I don't know enough about the IBM JDK to comment on this part
> > > intelligently. Go ahead and generate a patch, and we can use that as a
> > > starting point for discussion.
> >
> > I'm a little hesitant to remove the CMS configuration (as it really
> > helps). My first thought about how to address this is you can submit
> > some example Accumulo configurations that work with IBM JDK or you can
> > add something to the configuration template/script (conf/examples and
> > conf/templates with bin/bootstrap_config.sh, respectively). I think
> > you're on the right path.
> >
> > >
> > >>   >
> > >>>   So far, my work has been focused on getting unit tests working for
> > all
> > >>> Java vendors in a clean manner. I have not yet run intensive testing
> > of
> > >>> real clusters following these changes, and would be happy to get
> > pointers
> > >>> to what else might need treatment.
> > >>
> > >>
> > >>
> > > Unit tests is a good first pass. Integration tests (mvn verify) is
> > probably
> > > the minimum that you want on your continuous integration once you have
> > > things set up.
> > >
> > > Accumulo also comes with a set of longer running, cluster based tests,
> > > since we know that there are some pieces too complex for unit tests to
> > > catch. have a look in the test module for the Continuous Ingest test.
> > Once
> > > you get to that point, we can help you set it up if the README is
> > unclear.
> > >
> > >>   I would also like to hear if these changes make sense, and if so,
> > should
> > >>> I go ahead and create some JIRAs, and attach my patches for commit
> > >>> approval?
> > >>>
> > >>
> > > Filing JIRAs is going to be the most straightforward path, yes.
> > >
> > >   >  Looking forward to hearing feedback!
> >
> > Likewise. Looking forward to applying some patches!
> >
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message