brooklyn-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From henev...@apache.org
Subject [1/2] brooklyn-docs git commit: docs for using submodules
Date Mon, 01 Feb 2016 21:32:34 GMT
Repository: brooklyn-docs
Updated Branches:
  refs/heads/master 6e86cb020 -> f1da1755a


docs for using submodules


Project: http://git-wip-us.apache.org/repos/asf/brooklyn-docs/repo
Commit: http://git-wip-us.apache.org/repos/asf/brooklyn-docs/commit/6a8aef76
Tree: http://git-wip-us.apache.org/repos/asf/brooklyn-docs/tree/6a8aef76
Diff: http://git-wip-us.apache.org/repos/asf/brooklyn-docs/diff/6a8aef76

Branch: refs/heads/master
Commit: 6a8aef762895be69141fbaef2866b8553cd13cf2
Parents: 6e86cb0
Author: Alex Heneveld <alex.heneveld@cloudsoftcorp.com>
Authored: Mon Feb 1 21:08:43 2016 +0000
Committer: Alex Heneveld <alex.heneveld@cloudsoftcorp.com>
Committed: Mon Feb 1 21:08:43 2016 +0000

----------------------------------------------------------------------
 guide/dev/code/index.md         | 132 ++++++++++++++++----------------
 guide/dev/code/no-submodules.md |  20 +++++
 guide/dev/code/structure.md     |  59 ++++++++++++++
 guide/dev/code/submodules.md    | 144 +++++++++++++++++++++++++++++++++++
 guide/dev/index.md              |   5 +-
 5 files changed, 290 insertions(+), 70 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/brooklyn-docs/blob/6a8aef76/guide/dev/code/index.md
----------------------------------------------------------------------
diff --git a/guide/dev/code/index.md b/guide/dev/code/index.md
index bf2e22a..a589edf 100644
--- a/guide/dev/code/index.md
+++ b/guide/dev/code/index.md
@@ -1,91 +1,87 @@
 ---
-title: About the Code
+title: Get the Code
 layout: website-normal
+children:
+- submodules.md
+- no-submodules.md
 ---
 
 ## The Basics
 
-Brooklyn is available at [GitHub apache/incubator-brooklyn](http://github.com/apache/incubator-brooklyn).
 Check it out using:
+The Apache Brooklyn source code is available at [GitHub apache/brooklyn](http://github.com/apache/brooklyn),
+together with many [`brooklyn-*` sub-module projects](https://github.com/apache?query=brooklyn).
+Checkout and build all the submodules with:
 
 {% highlight bash %}
-git clone git@github.com:apache/incubator-brooklyn.git
+git clone http://github.com/apache/brooklyn/
 cd brooklyn
-{% endhighlight %}
-
-Build it with:
+git submodule init
+git submodule update --remote --merge --recursive
 
-{% highlight bash %}
 mvn clean install
 {% endhighlight %}
 
-And launch it with:
+This will produce an artifact in `brooklyn-dist/dist/brooklyn-dist-0.9.0-SNAPSHOT-dist.tar.gz`
<!-- BROOKLYN_VERSION -->
+which you can use [in the usual way](../../start/running.html).
+Some options which may be useful:
 
-{% highlight bash %}
-cd usage/dist/target/brooklyn-dist/
-bin/brooklyn launch
-{% endhighlight %}
+* Use `--depth 1` with `git clone` to skip the history (much faster but your `git log` will
be incomplete)
+* Use `-DskipTests` with `mvn` to skip tests (again much faster but it won't catch failures)
+* Drop `--recursive` on the `git clone` and set up some submodules manually 
+  (e.g. with `git submodule init && git submodule update --remote [module]`) 
+  or avoid submodules, following the directions below
+
+Thereafter to update the code in submodules, we strongly recommend doing this:
+
+    git pull && git submodule update --remote --merge --recursive
+
+This merges the latest upstream changes into the current branch of each sub-module on your
local machine,
+giving nice errors on conflicts.
+It's fine also to do branching and pulling in each submodule,
+but running `update` without these parameters can cause chaos!
+This chaos -- and a `git sup` alias for this command -- are described in the [submodules](submodules.html)
page.
+
+
+### If You Can't Stand Submodules
+
+[These instructions](no-submodules.html) can help setting up a local environment
+which does not rely on submodules.
+
+
+### Contributing a Small Change
+
+If you're making a small change in one project, consider just using that project.
+Whether you use this uber-project or not, to [contribute]({{ site.path.website }}/developers/how-to-contribute.html)

+you'll need to follow the usual fork->work->push->pull-request process.
+
+To understand where you might want to make your change,
+look at the [code structure](structure.html).
+
+
+### Bigger and Regular Changes
+
+Regular contributors will typically have their own fork for each of the submodule projects,
+and will probably want some other settings and tips [as described here](submodules.html).
+
+
+## History, Tags, and Workflow
+
+There are branches for each released version and tags for various other milestones.
+
+As described in more detail [here](submodules.html), we primarily use submodule remote branch
tracking
+rather than submodule SHA1 ID's.
 
-{% comment %}
-TODO examples
-Plenty of examples are in the **examples** sub-dir,
-described [here]({{site.path.guide}}/use/examples).
-{% endcomment %}
-
-Information on using Brooklyn -- configuring locations (in `brooklyn.properties`)
-and adding new projects to a catalog -- can be found in the [User's Guide]({{site.path.guide}}).
-This document is intended to help people become familiar with the codebase.
-
-## Project Structure
-
-Brooklyn is split into the following projects and sub-projects:
-
-* **``camp``**: the components for a server which speaks with the CAMP REST API and understands
the CAMP YAML plan language
-* **``api``**: the pure-Java interfaces for interacting with the system
-* **``core``**: the base class implementations for entities and applications, entity traits,
locations, policies, sensor and effector support, tasks, and more
-* **``locations``**: specific location integrations
-    * **``jclouds``**: integration with many cloud APIs and providers via Apache jclouds
-* **``policies``**: collection of useful policies for automating entity activity  
-* **``software``**: entities which are mainly launched by launched software processes on
machines, and collections thereof
-    * **``base``**: software process lifecycle abstract classes and drivers (e.g. SSH) 
-    * **``webapp``**: web servers (JBoss, Tomcat), load-balancers (Nginx), and DNS (Geoscaling)

-    * **``database``**: relational databases (SQL) 
-    * **``nosql``**: datastores other than RDBMS/SQL (often better in distributed environments)

-    * **``messaging``**: messaging systems, including Qpid, Apache MQ, RabbitMQ 
-    * **``monitoring``**: monitoring tools, including Monit
-    * **``osgi``**: OSGi servers 
-    * **...**
-* **``utils``**: projects with lower level utilities
-    * **common**: Utility classes and methods developed for Brooklyn but not dependent on
Brooklyn
-    * **groovy**: Groovy extensions and utility classes and methods developed for Brooklyn
but not dependent on Brooklyn
-    * **jmx/jmxmp-ssl-agent**: An agent implementation that can be attached to a Java process,
to give expose secure JMXMP
-    * **jmx/jmxrmi-agent**: An agent implementation that can be attached to a Java process,
to give expose JMX-RMI without requiring all high-number ports to be open
-    * **rest-swagger**: Swagger REST API utility classes and methods developed for Brooklyn
but not dependent on Brooklyn
-    * **test-support**: Test utility classes and methods developed for Brooklyn but not dependent
on Brooklyn
-* **``usage``**: projects which make Brooklyn easier to use, either for end-users or Brooklyn
developers
-    * **all**: maven project to supply a shaded JAR (containing all dependencies) for convenience
-    * **archetypes**: A maven archetype for easily generating the structure of new downstream
projects
-    * **camp**: Brooklyn bindings for the CAMP REST API
-    * **cli**: backing implementation for Brooklyn's command line interface
-    * **dist**: builds brooklyn as a downloadable .zip and .tar.gz
-    * **jsgui**: Javascript web-app for the brooklyn management web console (builds a WAR)
-    * **launcher**: for launching brooklyn, either using a main method or invoked from the
cli project
-    * **logback-includes**: Various helpful logback XML files that can be included; does
not contain logback.xml 
-    * **logback-xml**: Contains a logback.xml that references the include files in brooklyn-logback-includes
-    * **rest-api**: The API classes for the Brooklyn REST api
-    * **rest-client**: A client Java implementation for using the Brooklyn REST API 
-    * **rest-server**: The server-side implementation of the Brooklyn REST API
-    * **scripts**: various scripts useful for building, updating, etc. (see comments in the
scripts)
-    * **qa**: longevity and stress tests
-    * **test-support**: provides Brooklyn-specific support for tests, used by nearly all
projects in scope ``test``
-* **``docs``**: the markdown source code for this documentation
-* **``examples``**: some canonical examples
-* **``sandbox``**: various projects, entities and policies which the Brooklyn Project is
incubating
+The history prior to `0.9.0` is imported from the legacy `incubator-brooklyn` repo for reference
and history only.
+Visit that repo to build those versions; they are not intended to build here.
+(Although this works:
+`mkdir merged ; for x in brooklyn-* ; do pushd $x ; git checkout 0.8.0-incubating ; cp -r
* ../merged ; popd ; cd merged ; mvn clean install`.)
 
  
 ## Next Steps
 
-If you're interested in building and editing the code, check out:
+If you're interested in building and editing the code, you probably want to become familiar
with these:
 
+* [Product structure](structure.html)
 * [Maven setup](../env/maven-build.html)
 * [IDE setup](../env/ide/)
 * [Tests](tests.html)

http://git-wip-us.apache.org/repos/asf/brooklyn-docs/blob/6a8aef76/guide/dev/code/no-submodules.md
----------------------------------------------------------------------
diff --git a/guide/dev/code/no-submodules.md b/guide/dev/code/no-submodules.md
new file mode 100644
index 0000000..4516653
--- /dev/null
+++ b/guide/dev/code/no-submodules.md
@@ -0,0 +1,20 @@
+---
+title: Not Using Submodules
+layout: website-normal
+---
+
+If you don't want to use submodules, this will achieve the same result:
+
+{% highlight bash %}
+mkdir apache-brooklyn
+cd apache-brooklyn
+git clone http://github.com/apache/brooklyn/
+git clone http://github.com/apache/brooklyn-ui/
+git clone http://github.com/apache/brooklyn-server/
+git clone http://github.com/apache/brooklyn-client/
+git clone http://github.com/apache/brooklyn-docs/
+git clone http://github.com/apache/brooklyn-library/
+git clone http://github.com/apache/brooklyn-dist/
+ln -s brooklyn/pom.xml .
+mvn clean install
+{% endhighlight %}

http://git-wip-us.apache.org/repos/asf/brooklyn-docs/blob/6a8aef76/guide/dev/code/structure.md
----------------------------------------------------------------------
diff --git a/guide/dev/code/structure.md b/guide/dev/code/structure.md
new file mode 100644
index 0000000..bf1139b
--- /dev/null
+++ b/guide/dev/code/structure.md
@@ -0,0 +1,59 @@
+---
+title: Code Structure
+layout: website-normal
+---
+
+Brooklyn is split into the following subprojects:
+
+* **brooklyn-server**:
+    * **api**: the pure-Java interfaces for interacting with the system
+    * **camp**: the components for a server which speaks with the CAMP REST API and understands
the CAMP YAML plan language
+    * **core**: the base class implementations for entities and applications, entity traits,
locations, policies, sensor and effector support, tasks, and more
+    * **karaf**: OSGi support
+    * **launcher**: for launching brooklyn, either using a main method or invoked from the
cli project
+    * **locations**: specific location integrations
+        * **jclouds**: integration with many cloud APIs and providers via Apache jclouds
+    * **logging**: how we enable configurable logging
+        * **logback-includes**: Various helpful logback XML files that can be included; does
not contain logback.xml 
+        * **logback-xml**: Contains a logback.xml that references the include files in brooklyn-logback-includes
+    * **parent**: a meta-project parent to collect dependencies and other maven configuration
for re-use  
+    * **policy**: collection of useful policies for automating entity activity
+    * **rest**: supporting the REST API
+        * **rest-api**: The API classes for the Brooklyn REST api
+        * **rest-client**: A client Java implementation for using the Brooklyn REST API 
+        * **rest-server**: The server-side implementation of the Brooklyn REST API
+    * **server-cli**: implementation of the Brooklyn *server* command line interface; not
to be confused with the client CLI
+    * **software**: support frameworks for creating entities which mainly launch software
processes on machines
+        * **base**: software process lifecycle abstract classes and drivers (e.g. SSH) 
+        * **winrm**: support for connecting to Windows machines
+    * **test-framework**: provides Brooklyn entities for building YAML tests for other entities
+    * **test-support**: provides Brooklyn-specific support for Java TestNG tests, used by
nearly all projects in scope ``test``, building on `utils/test-support`
+    * **utils**: projects with lower level utilities
+        * **common**: Utility classes and methods developed for Brooklyn but not dependent
on Brooklyn
+        * **groovy**: Groovy extensions and utility classes and methods developed for Brooklyn
but not dependent on Brooklyn
+        * **jmx/jmxmp-ssl-agent**: An agent implementation that can be attached to a Java
process, to give expose secure JMXMP
+        * **jmx/jmxrmi-agent**: An agent implementation that can be attached to a Java process,
to give expose JMX-RMI without requiring all high-number ports to be open
+        * **rest-swagger**: Swagger REST API utility classes and methods developed for Brooklyn
but not dependent on Brooklyn
+        * **test-support**: Test utility classes and methods developed for Brooklyn but not
dependent on Brooklyn
+
+* **brooklyn-ui**: Javascript web-app for the brooklyn management web console (builds a WAR)
+
+* **brooklyn-library**: a library of useful blueprints
+    * **examples**: some canonical examples
+    * **qa**: longevity and stress tests
+    * **sandbox**: experimental items
+    * **software**: blueprints for software processes
+        * **webapp**: web servers (JBoss, Tomcat), load-balancers (Nginx), and DNS (Geoscaling)

+        * **database**: relational databases (SQL) 
+        * **nosql**: datastores other than RDBMS/SQL (often better in distributed environments)

+        * **messaging**: messaging systems, including Qpid, Apache MQ, RabbitMQ 
+        * **monitoring**: monitoring tools, including Monit
+        * **osgi**: OSGi servers 
+        
+* **brooklyn-docs**: the markdown source code for this documentation
+
+* **brooklyn-dist**: projects for packaging Brooklyn and making it easier to consume
+        * **all**: maven project to supply a shaded JAR (containing all dependencies) for
convenience
+        * **archetypes**: A maven archetype for easily generating the structure of new downstream
projects
+        * **dist**: builds brooklyn as a downloadable .zip and .tar.gz
+        * **scripts**: various scripts useful for building, updating, etc. (see comments
in the scripts)

http://git-wip-us.apache.org/repos/asf/brooklyn-docs/blob/6a8aef76/guide/dev/code/submodules.md
----------------------------------------------------------------------
diff --git a/guide/dev/code/submodules.md b/guide/dev/code/submodules.md
new file mode 100644
index 0000000..06380c2
--- /dev/null
+++ b/guide/dev/code/submodules.md
@@ -0,0 +1,144 @@
+---
+title: Git Submodules
+layout: website-normal
+---
+
+## Setting Up Forks
+
+If you're contributing regularly, you'll want your submodules to pull from the `apache` repo
+and push to your fork.
+You'll need to create a fork and the set up your local submodules should then have:
+
+* your fork as the `origin` remote repo
+* the Apache github repo as the `upstream` remote repo
+* optionally the repo at the Apache git server as the `apache` remote repo  
+
+This can be done automatically with the following commands:
+
+{% highlight bash %}
+    hub fork
+    git remote add apache https://git-wip-us.apache.org/repos/asf/$x
+    for x in brooklyn-* ; do
+      pushd $x
+      hub fork
+      git remote add apache https://git-wip-us.apache.org/repos/asf/$x
+      popd
+    done
+{% endhighlight %}
+
+This requires the command-line tool `hub` [from github](https://github.com/github/hub) (or
`sudo npm install -g hub`), 
+run in the directory of the uber-project checked out earlier.
+
+You can then pull and update from upstream, push to origin, and create pull requests in each
sub-project.
+Cross-project changes will require multiple PRs: 
+try to minimise these, especially where one depends on another,
+and especially especially where two depend on each other -- that is normally a sign of broken
backwards compatibility!
+Open the PRs in dependency order and assist reviewers by including the URLs of any upstream
dependency PRs 
+in the dependent PR to help reviewers 
+(dependency PRs will then include a "mention" comment of the dependent PR).
+
+
+## Things You Should Know
+
+Our submodules track **branches**, rather than specific commits,
+although due to the way `git` works there are still references to specific commits.
+
+We track `master` for the master branch and the version branch for other official branches,

+starting with `0.9.0`.
+We update the uber-project recorded SHA reference to subprojects on releases but not regularly
-- 
+that just creates noise and is unnecessary with the `--remote` option on `submodule update`.
+In fact, `git submodule update --remote --merge` pretty much works well;
+the `git sup` alias (below) makes it even easier.
+
+On the other hand, `git status` is not very nice in the uber-project:
+it will show a "new commits" message for submodules, 
+unless you're exactly at the uber-project's recorded reference.
+Ignore these.
+It will tell you if you have uncommitted changes, 
+but it's not very useful for telling whether you're up to date or if you have newer changes
committed 
+in the subproject or in your push branch.
+If you go in to each sub-project, `git status` works better, but it can be confusing
+to track which branch each subproject is one.
+A `summary` script is provided below which solves these issues,
+showing useful status across all subprojects.
+
+
+### Pitfalls of Submodules
+
+Submodules can be confusing; if you get stuck the references at the bottom may be useful.
+You can also work [without submodules](no-submodules.html).
+
+Some of the things to be careful of are:
+
+* **Don't copy submodule directories.** This doesn't act as you'd expect;
+  its `.git` record simply points at the parent project's `.git` folder,
+  which in turn points back at it.  So if you copy it and make changes in the copy,
+  it's rather surprising where those changes actually get made.
+  Worse, `git` doesn't report errors; you'll only notice it when you see files change bizarrely.
+  
+* **Be careful committing in the uber-project.**
+  You can update commit IDs, but if these accidentally point to an ID that isn't committed,

+  everyone else sees errors.
+  It's useful to do this on release (and update the target branch then also)
+  and maybe occasionally at other milestones but so much at other times as these ID's 
+  very quickly become stale on `master`.
+
+
+### Useful Aliases
+
+{% highlight bash %}
+git config --global alias.sup 'submodule update --remote --merge --recursive'
+git config --global alias.sdiff '!git diff && git submodule foreach "git diff"'
+{% endhighlight %}
+
+
+### Getting a Summary of Submodules
+
+In addition, the `git-summary` script [here]() makes working with submodules much more enjoyable,
+simply install and use `git ss` in the uber-project to see the status of each submodule:
+
+{% highlight bash %}
+curl https://gist.githubusercontent.com/ahgittin/6399a29df1229a37b092/raw/05f99aa95a5e8eb541bb79c6707324e26fc0f579/git-summary.sh
\
+  | sudo tee /usr/local/bin/git-summary
+sudo chmod 755 /usr/local/bin/git-summary  
+git config --global alias.ss '!git-summary ; echo ; git submodule foreach --quiet "git summary"'
+{% endhighlight %}
+
+
+### Other Handy Commands
+
+{% highlight bash %}
+# run a git command (eg pull) in each submodule
+git submodule foreach 'git pull'
+
+# iterate across submodules in bash, e.g. doing git status
+for x in brooklyn-* ; do pushd $x ; git status ; popd ; done
+{% endhighlight %}
+
+
+### Legacy Incubator Pull Requests
+
+If you need to apply code changes made pre-graduation, against the incubator repository,
+splitting it up into submodules, it's fairly straightforward:
+
+1. In the incubator codebase, start at its final state, in `master`.
+2. `git checkout -b making-a-diff`
+3. Merge or rebase the required commits. 
+   Ensure conflicts are resolved, but don't worry about commit messages.
+4. `git diff > /tmp/diff`
+5. Go to the new `brooklyn` uber-project directory.
+   Ensure you are at master and all subprojects updated (`git sup`).
+6. `patch -p1 < /tmp/diff` 
+7. Run `git ss` to inspect the changes.
+8. Test it, commit each changed project on a branch and create pull requests.
+   Where applicable, record the original author(s) and message(s) in the commit.
+
+
+
+## Git Submodule References
+
+* [1] [Git SCM Book](https://git-scm.com/book/en/v2/Git-Tools-Submodules)
+* [2] [Medium blog: Mastering Git Submodules](https://medium.com/@porteneuve/mastering-git-submodules-34c65e940407#.r7677prhv)
+* [3] `git submodule --help` section on `update`
+* [4] [StackOverflow: Git Submodules Branch Tag](http://stackoverflow.com/questions/1777854/git-submodules-specify-a-branch-tag/18797720#18797720)
+

http://git-wip-us.apache.org/repos/asf/brooklyn-docs/blob/6a8aef76/guide/dev/index.md
----------------------------------------------------------------------
diff --git a/guide/dev/index.md b/guide/dev/index.md
index 2c294b6..3fb28dd 100644
--- a/guide/dev/index.md
+++ b/guide/dev/index.md
@@ -6,14 +6,15 @@ breadcrumbs:
 - /website/documentation/index.md
 - /guide/dev/index.md
 children:
-- env/
 - code/
-- { link: "http://github.com/apache/incubator-brooklyn", title: "GitHub" }
+- env/
+- code/structure.md
 - code/tests.md
 - code/licensing.md
 - tips/
 - tips/logging.md
 - tips/debugging-remote-brooklyn.md
+- { link: "http://github.com/apache/incubator-brooklyn", title: "GitHub" }
 - { link: "https://brooklyn.apache.org/v/latest/misc/javadoc", title: "Javadoc" }
 ---
 


Mime
View raw message