accumulo-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sean Busbey <bus...@cloudera.com>
Subject Re: Running Accumulo on the IBM JVM
Date Mon, 23 Jun 2014 14:14:58 GMT
Hi Hayden!

It might also help to take a look at our workflow guide for contributors:

http://accumulo.apache.org/git.html#contributors

One thing to note from the guide: we generally only expect the contributors
to make the initial patch. If the committer who handles pushing things runs
into trouble merging forward, they may ask for an additional patch as
assistance.

As Josh mentioned, the configuration builder was introduced in 1.6.0. If
you'd like changes in the 1.5.x branch as well, you'll need to add a
configuration example.



On Mon, Jun 23, 2014 at 5:00 AM, Hayden Marchant <HAYDEN@il.ibm.com> wrote:

> Josh (and all who commented),
>
> Thanks for the comments. I'll take them into account, and will create the
> JIRAs.
>
> I was not intending on removing the CMS options, but rather only including
> them in the JVM in which they are relevant, and including the equivalent
> in different JVMs (i.e. IBM ) - all through the bootstrap_config.sh.
>
> Here's my newbie question: Should I be making this patch based on 1.6.1,
> or should I always be working against the 'master' branch, and then
> backport the fix(es) to any desired older version?
>
> Regards,
>
>
>
> Hayden
>
>
>
> From:   Josh Elser <josh.elser@gmail.com>
> To:     dev@accumulo.apache.org,
> Date:   19/06/2014 06:43 PM
> Subject:        Re: Running Accumulo on the IBM JVM
>
>
>
> <snip/>
>
> >>>          b.
> >>>
> >>>
> >>
>
> org.apache.accumulo.core.security.crypto.BlockedIOStreamTest.testGiantWrite.
> >>>          This test assumes a max heap of about 1GB. This fails on IBM
> JRE,
> >>> since the default max heap is not specified, and on IBM JRE this
> depends
> >>> on the OS (see
> >>>
> >>>
> >>
>
> http://www-01.ibm.com/support/knowledgecenter/SSYKE2_6.0.0/com.ibm.java.doc.diagnostics.60/diag/appendixes/defaults.html?lang=en
>
> >>> ).
> >>>          Proposal: add -Xmx1g to the surefire maven plugin reference
> in
> >>> parent maven pom.
> >>>
> >>
> > This might be https://issues.apache.org/jira/browse/ACCUMULO-2774
>
> Yup! I actually bumped this up to 1G already after I started seeing
> failures (again) from the ACCUMULO-2774 patch which set a 768M heap.
> Pull the upstream changes and feel free to submit something to address
> any problem you still have.
>
> >
> >
> >>   >         c. Both org.apache.accumulo.core.security.crypto.CrypoTest
> &
> >>> org.apache.accumulo.core.file.rfile.RFileTest have lots of failures
> due
> >> to
> >>> calls to SEcureRandom with Random Number Generator Provider hard-coded
> as
> >>> Sun. The IBM JRE has it's own built in RNG Provider called IBMJCE. 2
> >>> issues - hard-coded calls to SecureRandom.getInstance(<algo>,"SUN")
> and
> >>> also default value in Property class is "SUN".
> >>>          Proposal: Add mechanism to override default Property through
> >>> System property through new annotator in Property class. Only usage
> will
> >>> be by Property.CRYPTO_SECURE_RNG_PROVIDER
> >>
> >>
> >>
> > I'm not sure about adding new annotators to Property. However, the
> > CryptoTest should be getting the value from the conf instead of
> hard-coding
> > it. Then you can specify the correct value in accumulo-site.xml
> >
> > I think another part of the issue is in
> > CryptoModuleFactory::fillParamsObjectFromStringMap because it looks like
> > that ignores the default setting.
> >
> >>   >
> >>> 2. Environment/Configuration
> >>>          a. The generated configuration files contain references to GC
> >>> params that are specific to Sun JVM. In accumulo-env.sh, the
> >>> ACCUMULO_TSERVER_OPTS contains -XX:NewSize and -XX:MaxNewSize , and
> also
> >>> in ACCUMULO_GENERAL_OPTS,
> >>> -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 are
> used.
> >>>          b. in bin/accumulo, get ClassNotFoundException due to
> >>> specification of JAXP Doc Builder:
> >>>
> >>>
> >>
>
> -Djavax.xml.parsers.DocumentBuilderFactory=com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderFactoryImpl
> >>> .
> >>>          The Sun implementation of Document Builder Factory does not
> >> exists
> >>> in IBM JDK, so a ClassNotFoundException is thrown on running accumulo
> >>> script
> >>>
> >>>          c. MiniAccumuloCluster - in the MiniAccumuloClusterImpl,
> >>> Sun-speciifc GC params are passed as params to the java process
> (similar
> >>> to section a. )
> >>>
> >>>          Single proposal for solving all three above issues:
> >>>          Enhance bootstrap_config.sh with request to select Java
> vendor.
> >>> Selecting this will set correct values for GC params (they differ
> between
> >>> IBM and Sun), inclusion/ommision of JAXP setting. The
> >>> MiniAccumuloClusterImpl can read the same env variable that was set in
> >>> code for the GC Params, and use in the exec command.
> >>>
> >>
> > I don't know enough about the IBM JDK to comment on this part
> > intelligently. Go ahead and generate a patch, and we can use that as a
> > starting point for discussion.
>
> I'm a little hesitant to remove the CMS configuration (as it really
> helps). My first thought about how to address this is you can submit
> some example Accumulo configurations that work with IBM JDK or you can
> add something to the configuration template/script (conf/examples and
> conf/templates with bin/bootstrap_config.sh, respectively). I think
> you're on the right path.
>
> >
> >>   >
> >>>   So far, my work has been focused on getting unit tests working for
> all
> >>> Java vendors in a clean manner. I have not yet run intensive testing
> of
> >>> real clusters following these changes, and would be happy to get
> pointers
> >>> to what else might need treatment.
> >>
> >>
> >>
> > Unit tests is a good first pass. Integration tests (mvn verify) is
> probably
> > the minimum that you want on your continuous integration once you have
> > things set up.
> >
> > Accumulo also comes with a set of longer running, cluster based tests,
> > since we know that there are some pieces too complex for unit tests to
> > catch. have a look in the test module for the Continuous Ingest test.
> Once
> > you get to that point, we can help you set it up if the README is
> unclear.
> >
> >>   I would also like to hear if these changes make sense, and if so,
> should
> >>> I go ahead and create some JIRAs, and attach my patches for commit
> >>> approval?
> >>>
> >>
> > Filing JIRAs is going to be the most straightforward path, yes.
> >
> >   >  Looking forward to hearing feedback!
>
> Likewise. Looking forward to applying some patches!
>
>
>


-- 
Sean

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message