hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daniel Vimont <dan...@commonvox.org>
Subject Re: [DISCUSS] new build infrastructure for creation/maintenance of hbase maven archetypes (HBASE-14877)
Date Thu, 28 Jan 2016 18:23:46 GMT
Jon,

Thanks for offering to take a look!

Your question prompts me to come up with a bit more explicit rubric for
assessment!

We want to be sure that the proposed hbase-archetypes infrastructure
"works" on at least three levels:

   1. it has to be workably consistent with the philosophy and structures
   of the overall hbase project;
   2. it has to be workable for HBase contributors and committers for the
   full SDLC of current and future archetypes developed using this
   infrastructure; and
   3. the projects that the archetypes ultimately generate for end-users
   need to be as close to 100% rock-solid, robust, and straightforward as
   possible. (Using these archetypes could become many people's first hands-on
   experience with HBase, and it had better result in pleasure and wonder, not
   disappointment and confusion.)

To assess items (1) and (2) above seems to require (a) reading the
hbase-archetype README text (which is now rendered here in markdown format
<https://github.com/dvimont/test_hbasearchetypes_readme> -- a little easier
on the eyes), and (b) potentially inspecting the components of the
hbase-archetypes infrastructure described in the README.

Specific questions for items (1) and (2):

   - With regard to the new infrastructure described in the README, does
   the infrastructure seem robust and workable enough to commit to for present
   and future development/maintenance of archetypes?
   - Is the README text readily comprehensible?
   - Is the README text comprehensive enough in its instructions to
   contributors, or is something missing that needs to be there?

To assess item (3) above requires installing the patch (to a clone of the
master branch) in a sandbox, running a "clean install", and following the
instructions given in the README under the heading "How to do additional
testing/inspection of an archetype in this collection". (Those unfamiliar
with Maven archetypes can follow footnotes 1 and 4 in the README for basic
information.) Following the instructions there should result in creation of
a new Maven project, preconfigured with dependency on hbase-client and
test-scope dependency on hbase-testing-util.

Specific questions for item (3):

   - Were any problems encountered in the project generation process?
   - Does the generated project provide all basic components needed in a
   simple hbase-client dependent project?
   - Is the sample code that is provided (in both the main and test
   subdirectories) clear in its structures and comments, so as to give a good
   starting foundation to a programmer unfamiliar with the HBase client API?

That is all that comes to mind at the moment!

Thanks,

Dan


On Thu, Jan 28, 2016 at 9:53 PM, Jonathan Hsieh <jon@cloudera.com> wrote:

> I think this is a great dev and new user friendly idea.  I saw some jiras
> relate to this being created.
>
> Aside from folks trying it and reviewing, is there anything else you need?
> My time is limited but will try to give it a spin in the next week.
>
> Jon
>
> On Wednesday, January 27, 2016, Daniel Vimont <daniel@commonvox.org>
> wrote:
>
> > One point to add to this discussion: the intention is to backport this to
> > make it available in an upcoming 1.x minor release, so that new
> archetypes
> > are publicly available via the Maven central repository as soon as
> > possible.
> >
> > As Sean brings up over in the HBASE-14877 JIRA entry
> > <https://issues.apache.org/jira/browse/HBASE-14877>, we need to
> determine
> > "what level of 'done' we're looking for before we do that".
> >
> > While I imagine that this will not be anywhere near as involved as the
> > backporting of the new hbase-spark stuff to branch-1 (see HBASE-14160
> > <https://issues.apache.org/jira/browse/HBASE-14160>), this patch does
> > introduce *new infrastructure* for developing and maintaining new
> artifacts
> > (in this case, Maven archetypes), so it certainly does merit some review
> > before we open up a new JIRA and start adding this functionality to
> > branch-1.
> >
> > On Wed, Jan 27, 2016 at 4:07 PM, Daniel Vimont <daniel@commonvox.org
> > <javascript:;>> wrote:
> >
> > > Work has been underway for the last month on task HBASE-14877
> > > <https://issues.apache.org/jira/browse/HBASE-14877>, the first subtask
> > of
> > > HBASE-14876 (in Nick's original wording from that JIRA entry: "To help
> > > onboard new users, we should *provide maven archetypes for hbase client
> > > applications*").
> > >
> > > The three originally proposed *Maven archetypes* in HBASE-14876 were
> > > split into three subtasks, for:
> > >
> > >    1. an hbase-client dependent application,
> > >    2. an hbase-shaded-client dependent application, and
> > >    3. a map-reduce job example with hbase-server dependency.
> > >
> > > *(Note that a fourth subtask has since been added -- for an hbase-spark
> > > oriented archetype.)*
> > >
> > > *The first subtask (HBASE-14877) provides:*
> > >
> > > *[a] a new infrastructure* for creation and maintenance of
> HBase-related
> > > Maven archetypes, and
> > > [*b] the first archetype* from the list above (to be used by end-users
> to
> > > autogenerate a fully-configured Maven project with hbase-client
> > dependency
> > > and fully-functional sample Java code).
> > >
> > >
> > > *It is item [a] above -- a new infrastructure for creation/maintenance
> of
> > > archetypes -- that we would particularly like for
> contributors/committers
> > > to take a look at, and to offer up comments, questions, suggestions,
> > etc.*
> > > The README text in the new hbase-archetypes subdirectory is intended to
> > be
> > > a brief yet comprehensive introduction to how this new infrastructure
> > > works, and how a contributor would add new Maven archetypes in the
> > future. *(It
> > > is intended that it be very simple to contribute new archetypes,
> without
> > > the need for any special expertise with regard to Maven archetypes in
> > > general.)*
> > >
> > > For ease of access, I'll just paste in the README text below. *(Note
> that
> > > this text will soon be reformatted into markdown format, but this is
> the
> > > intended content.)*
> > >
> > > Besides reviewing this text, you can earn extra gratitude-points if
> you:
> > >
> > >    -
> > > *set up a sandbox environment, *
> > >    -
> > > *install the latest patch (HBASE-14877-v4.patch
> > >    <
> >
> https://issues.apache.org/jira/secure/attachment/12784323/HBASE-14877-v4.patch
> > >)
> > >    against the HBase master branch, *
> > >    -
> > > *follow the instructions at the end of the README (below) to generate a
> > >    new project from the archetype (emulating the experience of an
> > end-user),
> > >    and *
> > >    - *review the results of that process, as well!*
> > >
> > > THANKS VERY MUCH,
> > > Dan Vimont
> > >
> > >
> > >
> > >
> >
> -------------------------------------------------------------------------------
> > > *README.txt*
> > >
> > >
> >
> -------------------------------------------------------------------------------
> > > *The hbase-archetypes subproject of hbase provides an infrastructure
> for*
> > > *creation and maintenance of Maven archetypes[1] pertinent to HBase.
> > Upon*
> > > *deployment to the archetype catalog[2] of the central Maven
> > > repository[3],*
> > > *these archetypes may be used by end-user developers to autogenerate
> > > completely*
> > > *configured Maven projects (including fully-functioning sample code)
> > > through*
> > > *invocation of the archetype:generate goal of the
> > > maven-archetype-plugin[4].*
> > >
> > > *Notes for contributors and committers to the HBase project*
> > > *==========================================================*
> > > *The hbase-archetypes project contains a separate subproject for each
> > > archetype.*
> > > *The top level components of such a subproject comprise a complete,
> > > standalone*
> > > *exemplar Maven project containing (a) a src directory with sample,
> > fully-*
> > > *functioning code in the ./main and ./test subdirectories, (b) a
> pom.xml
> > > file*
> > > *defining all required dependencies, and (c) any additional resources
> > > required*
> > > *by the exemplar project. (For example, the components of the
> > > hbase-client-*
> > > *project consist of (a) sample code ./src/main/.../HelloHBase.java and*
> > > *./src/test/.../TestHelloHBase.java, (b) a pom.xml file establishing
> > > dependency*
> > > *upon hbase-client and test-scope dependency upon hbase-testing-util,
> and
> > > (c) a*
> > > *log4j.properties resource file.)*
> > >
> > > *** Overview of the creation of archetypes during the hbase install
> > > process: ***
> > > *During the mvn install process, all standalone exemplar projects in
> the*
> > > *hbase-archetypes subdirectory are first packaged/tested/installed, and
> > > then the*
> > > *following steps are executed in
> hbase-archetypes/hbase-archetype-builder
> > > (via*
> > > *the pom.xml, bash scripts, and xsl templates in that subdirectory):*
> > > *  (1) For each exemplar project, resources are copied (via
> > > maven-resources-*
> > > *plugin) and transformed (via xml-maven-plugin xslt functionality) to
> > the*
> > > *exemplar project's ./target/build-archetype subdirectory[5].*
> > > *  (2) The script createArchetypes.sh is executed to invoke the
> > > maven-archetype-*
> > > *plugin's create-from-project goal within each exemplar project's*
> > > *./target/build-archetype subdirectory. For each exemplar project, this
> > > creates*
> > > *a corresponding Maven archetype in the ./target/build-*
> > > *archetype/target/generate-sources/archetype subdirectory. (Note that
> > this
> > > step*
> > > *always issues two platform-encoding warnings per archetype, due to
> > > hard-wired*
> > > *behavior of the maven-archetype-plugin[6].)*
> > > *  (3) The pom.xml file of each newly-created archetype is copied (via
> > > maven-*
> > > *resources-plugin) and transformed (via xml-maven-plugin xslt
> > > functionality)[7].*
> > > *  (4) The script installArchetypes.sh is executed to install each
> > > archetype*
> > > *into the local Maven repository, ready for deployment to the central
> > > Maven*
> > > *repository. (Note that installation of an archetype automatically
> > > includes*
> > > *invocation of integration-testing prior to install, which performs a
> > test*
> > > *generation of a project from the archetype.)*
> > >
> > > *** To add a new archetype to the hbase-archetypes collection: ***
> > > *  (1) Create a new subdirectory in hbase-archetypes, populated with a*
> > > *completely configured Maven project, which will serve as the exemplar
> > > project*
> > > *of the new archetype. (It may be most straightforward to simply copy
> the
> > > src*
> > > *and pom.xml components from one of the existing exemplar projects,
> > > replace the*
> > > *src/main and src/test code, and modify the pom.xml file's
> > <dependencies>,*
> > > *<artifactId>, <name>, and <description> elements.)*
> > > *  (2) Modify the hbase-archetype-builder/pom.xml file: (a) add the new
> > > exemplar*
> > > *project to the <modules> element, and (b) add appropriate <execution>
> > > elements*
> > > *and <transformationSet> elements within the <plugin> elements
(using
> > the*
> > > *existing entries from already-existing exemplar projects as a guide).*
> > > *  (3) Add appropriate entries for the new exemplar project to the*
> > > *createArchetypes.sh and installArchetypes.sh scripts in the
> > > hbase-archetype-*
> > > *builder subdirectory (using the existing entries as a guide).*
> > >
> > > *** For additional testing/inspection of an archetype in this
> collection:
> > > ***
> > > *Although integration-testing (which is automatically performed for
> each*
> > > *archetype during the install process) already performs test generation
> > of
> > > a*
> > > *project from an archetype, it may often be advisable to do further
> > manual*
> > > *testing of a newly built and installed archetype, particularly to
> > examine
> > > and*
> > > *test a project generated from the archetype (mimicking the end-user
> > > experience*
> > > *of utilizing the archetype). Upon completion of the install process
> > > outlined*
> > > *above, all archetypes will have been installed in the local Maven
> > > repository*
> > > *and can be tested locally by executing the following:*
> > > *    mvn archetype:generate -DarchetypeCatalog=local*
> > > *This displays a numbered list of all locally-installed archetypes for
> > the
> > > user*
> > > *to choose from for generation of a new Maven project.*
> > > *=================================================================*
> > > *END of notes for contributors and committers to the HBase project*
> > >
> > > *Footnotes:*
> > > *===========================================*
> > > *[1] http://maven.apache.org/archetype/index.html
> > > <http://maven.apache.org/archetype/index.html>*
> > > *[2] http://repo1.maven.org/maven2/archetype-catalog.xml
> > > <http://repo1.maven.org/maven2/archetype-catalog.xml>*
> > > *[3] http://search.maven.org/ <http://search.maven.org/>*
> > > *[4]
> > >
> >
> http://maven.apache.org/archetype/maven-archetype-plugin/generate-mojo.html
> > > <
> >
> http://maven.apache.org/archetype/maven-archetype-plugin/generate-mojo.html
> > >*
> > > *[5] Prior to archetype creation, each exemplar project's pom.xml is
> > > transformed*
> > > *    as follows to make it into a standalone project: resource
> filtering*
> > > *    replaces ${project.version} with the literal value of the current*
> > > *    project.version; and xslt tranformation (a) copies <groupId> and
> > > <version>*
> > > *    subelements of <parent> to make them child elements of the root
> > > element,*
> > > *    and (b) removes the <parent> and <description> elements.*
> > > *[6] For an explanation of the platform-encoding warning issued during*
> > > *    maven-archetype-plugin processing, see the first answer to this*
> > > *    stackoverflow posting:
> http://stackoverflow.com/a/24161287/4112172
> > > <http://stackoverflow.com/a/24161287/4112172>*
> > > *[7] Prior to archetype installation, each archetype's pom.xml is
> > > transformed*
> > > *    as follows: a <project.build.sourceEncoding> subelement with
> value*
> > > *    'UTF-8' is added to the <properties> element. This prevents*
> > > *    platform-encoding warnings from being issued when an end-user
> > > generates*
> > > *    a project from the archetype.*
> > >
> > >
> >
> -------------------------------------------------------------------------------
> > >
> >
>
>
> --
> // Jonathan Hsieh (shay)
> // HBase Tech Lead, Software Engineer, Cloudera
> // jon@cloudera.com // @jmhsieh
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message