Return-Path: X-Original-To: apmail-allura-commits-archive@www.apache.org Delivered-To: apmail-allura-commits-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id D233D10FD1 for ; Fri, 6 Mar 2015 12:35:09 +0000 (UTC) Received: (qmail 49197 invoked by uid 500); 6 Mar 2015 12:35:09 -0000 Delivered-To: apmail-allura-commits-archive@allura.apache.org Received: (qmail 49150 invoked by uid 500); 6 Mar 2015 12:35:09 -0000 Mailing-List: contact commits-help@allura.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@allura.apache.org Delivered-To: mailing list commits@allura.apache.org Received: (qmail 48540 invoked by uid 99); 6 Mar 2015 12:35:09 -0000 Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Mar 2015 12:35:09 +0000 Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33) id DD272E10A2; Fri, 6 Mar 2015 12:35:08 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: jetmind@apache.org To: commits@allura.apache.org Date: Fri, 06 Mar 2015 12:35:24 -0000 Message-Id: <117da3b7991943159bc8009127f7584c@git.apache.org> In-Reply-To: <04c082cf8f574e1988f12ffc9f0b7abc@git.apache.org> References: <04c082cf8f574e1988f12ffc9f0b7abc@git.apache.org> X-Mailer: ASF-Git Admin Mailer Subject: [17/26] allura git commit: [#7835] Restructured the docs and updated the theme. http://git-wip-us.apache.org/repos/asf/allura/blob/ae5d2746/Allura/docs/getting_started/using.rst ---------------------------------------------------------------------- diff --git a/Allura/docs/getting_started/using.rst b/Allura/docs/getting_started/using.rst new file mode 100644 index 0000000..84d9f93 --- /dev/null +++ b/Allura/docs/getting_started/using.rst @@ -0,0 +1,64 @@ +.. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +************ +Using Allura +************ + + +We don't have much end-user help for Allura yet. SourceForge projects use Allura, +though, so their support documentation may be useful to anyone using Allura: + + +Configuring your project +------------------------ + +See SourceForge help page: https://sourceforge.net/p/forge/documentation/Create%20a%20New%20Project/ + +Note there are some SourceForge-specific references that don't apply to other Allura instances. + + +Using tickets +------------- + +See SourceForge help page: https://sourceforge.net/p/forge/documentation/Tickets/ + + +Using the wiki +-------------- + +See SourceForge help page: https://sourceforge.net/p/forge/documentation/Wiki/ + + +Using a discussion forum +------------------------ + +See SourceForge help page: https://sourceforge.net/p/forge/documentation/Discussion/ + + +Adding an external link +----------------------- + +See SourceForge help page: https://sourceforge.net/p/forge/documentation/External%20Link/ + + +Using markdown syntax +--------------------- + +Everything in Allura uses Markdown formatting, with several customizations and macros +specifically for Allura. There are "Formatting Help" buttons throughout Allura for +easy reference to the Markdown syntax. One such page is https://forge-allura.apache.org/p/allura/wiki/markdown_syntax/ http://git-wip-us.apache.org/repos/asf/allura/blob/ae5d2746/Allura/docs/guides/email.rst ---------------------------------------------------------------------- diff --git a/Allura/docs/guides/email.rst b/Allura/docs/guides/email.rst deleted file mode 100644 index d88105e..0000000 --- a/Allura/docs/guides/email.rst +++ /dev/null @@ -1,69 +0,0 @@ -.. Licensed to the Apache Software Foundation (ASF) under one - or more contributor license agreements. See the NOTICE file - distributed with this work for additional information - regarding copyright ownership. The ASF licenses this file - to you under the Apache License, Version 2.0 (the - "License"); you may not use this file except in compliance - with the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, - software distributed under the License is distributed on an - "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY - KIND, either express or implied. See the License for the - specific language governing permissions and limitations - under the License. - -Guide to email integration in the Allura -======================================== - -Email routing -------------- - -routing mechanism will be a dotted path from the project to -the application/tool to the specific artifact within that app that is -used like this:: - - subproject.app.artifact@project.example.com - -Which would translate into the devtools project which is a subproject of -turbogears, and it's bug tracker and ticket 142 in that tracker:: - - devtools.bug.142@turbogears.sf.net - - -And it in turn would be published to the message bus, which will assure -that all tools that are registered to be notified for that e-mail -addresses are called like that. - -If your app has more than one artifact type, you could nest them inside -`project.app.something.id.*` - -If you were working with the bug tracker directly on the TurboGears project:: - - bug.142@turbogears.sf.net - -The Allura platform allows you to setup other message types, such as commit -messages, to go into amqp with the same routing information, and turn into -"messages" just like e-mail. - -Email Content Handling ----------------------- - -On Allura message bodies should be composed as markdown. -Multi-part mime encoded messages should be sent include plain text -(the markdown) and html (rendered from the markdown). - -Users are allowed to register that they want plain text only. - -We will also include some text in the footer of the e-mail message with a -link to the message online. We can use this link to guess where in the -thread the message belongs in the case of a messed up e-mail client that -does not set the headers for the reply properly. - -The nice thing about this is that it's pretty much already implemented -for us via the meta tool. - -This metadata syntax will let you set fields on tickets and otherwise -interact with the system via e-mail, assuming you have such permissions. http://git-wip-us.apache.org/repos/asf/allura/blob/ae5d2746/Allura/docs/guides/message_bus.rst ---------------------------------------------------------------------- diff --git a/Allura/docs/guides/message_bus.rst b/Allura/docs/guides/message_bus.rst deleted file mode 100644 index cef23a0..0000000 --- a/Allura/docs/guides/message_bus.rst +++ /dev/null @@ -1,97 +0,0 @@ -.. Licensed to the Apache Software Foundation (ASF) under one - or more contributor license agreements. See the NOTICE file - distributed with this work for additional information - regarding copyright ownership. The ASF licenses this file - to you under the Apache License, Version 2.0 (the - "License"); you may not use this file except in compliance - with the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, - software distributed under the License is distributed on an - "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY - KIND, either express or implied. See the License for the - specific language governing permissions and limitations - under the License. - -Guide to the Allura task and event system -========================================= - -Our event system is driven by a MongoDB-based queuing system, most of which you -can ignore, because we've simplified it down to two ideas: *tasks* and *event handlers*. - -Glossary --------- - -Before we get into the details perhaps a few definitions are in order: - -* **app** -- tool for allura such as the tracker, scm, or wiki apps -* **task** -- callable defined in a app that gets invoked by the `taskd` daemon -* **event handler** -- callable defined in an app that gets called on message - events. Event handlers are identified by a string name, so a single event can - fan out to multiple callables. - -Tasks ------ - -The `MonQTask` class is central to the Allura asynchronous processing system. -Simply put, a `MonQTask` is a document in MongoDB that contains a context -(project/app/user), a function pointer (specified as a dotted string), and -arguments to that function. Tasks are scheduled by creating `MonQTask` -documents, and the `taskd` daemon executes them as though they were happening in -the web context. - -To simplify the use of tasks, Allura provides a decorator `@task` that marks a -function as 'taskable.' This decorator adds a `.post` method to the function -object that allows the function to be scheduled as a MonQTask. For instance, the -`commit` task (flushing Solr caches) is defined in as the following:: - - @task - def commit(): - g.solr.commit() - -In order to schedule this task for execution by taskd, simply use the `.post` -method:: - - commit.post() - -If we wanted to call `commit` directly (e.g. in a test), we can still do that as -well:: - - commit() - -.. _events: - -Events ------- - -Events provide fanout capability for messages, letting several functions get -called in response to the same 'event.' To note a function as an event handler, -you use the `@event_handler` decorator. For instance, there is an event handler -on all project updates to subscribe the project's admins to project changes:: - - @event_handler('project_updated') - def subscribe_admins(topic): - c.app.subscribe_admins() - -In order to invoke all the event handlers for a particular topic, we use the -`g.post_event` helper:: - - g.post_event('project_updated') - -Under the covers, this is scheduling an `event` task that calls all the handlers -for a particular named event. Note that you can pass arguments (\*args, and -\*\*kwargs) to event handlers just like you do to tasks, with the exception that -the topic name (above, this would be 'project_updated') is always the first -parameter passed to the event handler. - -Running the Task Daemon ------------------------ - -In order to actually run the asynchronous tasks, we have written a paster command -`taskd`. This creates a configurable number of worker processes that watch for -changes to the `MonQTask` collection and execute requested tasks. `taskd` can be -run on any server, but should have similar access to the MongoDB databases and -configuration files used to run the web app server, as it tries to replicate the -request context as closely as possible when running tasks. http://git-wip-us.apache.org/repos/asf/allura/blob/ae5d2746/Allura/docs/guides/permissions.rst ---------------------------------------------------------------------- diff --git a/Allura/docs/guides/permissions.rst b/Allura/docs/guides/permissions.rst deleted file mode 100644 index 713b08c..0000000 --- a/Allura/docs/guides/permissions.rst +++ /dev/null @@ -1,60 +0,0 @@ -.. Licensed to the Apache Software Foundation (ASF) under one - or more contributor license agreements. See the NOTICE file - distributed with this work for additional information - regarding copyright ownership. The ASF licenses this file - to you under the Apache License, Version 2.0 (the - "License"); you may not use this file except in compliance - with the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, - software distributed under the License is distributed on an - "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY - KIND, either express or implied. See the License for the - specific language governing permissions and limitations - under the License. - -Guide to Users, Groups and Permissions in Allura -================================================ - -User/Group model ----------------- - -In the allura system `users` can be assigned to various `groups` or -roles on a per-project basis. - -Users can be members of many groups, and `groups` can -be assigned a list of `permissions` like `edit`, -`mdoderate` or `read`. Tools can define their own -set of permissions, for their artifacts. - -Individual artifacts and ACL's ------------------------------- - -You may want to assign a permission -to particular people or roles for a specific `Artifact` such as -one bug in the ticket tracker. The Allura platform supports this via -an additive ACL field on every `Artifact` instance. It is not exposed -via the UI currently. - -Permission hierarchy --------------------- - -Projects and subprojects can define user groups, but for any particular -subproject the set of groups the user belongs to is additive. This follows -the basic principle that sub-project permissions and artifact permissions -can *allow* additional access, but can't *restrict* it beyond -what permissions are allowed by a higher level project. - -Permission predicates ---------------------- - -Predicates are simple functions, several of which are defined in Allura -itself, and which can be added by any tool, which return true if -permission is granted, and false if it is not. - -An example predicate function `has_project_access` takes two params, an object -and a `permission` string. It then checks to see if the current user -(picked up from the environment) has permission to perform that action on -that object, following the rules above. http://git-wip-us.apache.org/repos/asf/allura/blob/ae5d2746/Allura/docs/guides/scm.rst ---------------------------------------------------------------------- diff --git a/Allura/docs/guides/scm.rst b/Allura/docs/guides/scm.rst deleted file mode 100644 index 9d8e987..0000000 --- a/Allura/docs/guides/scm.rst +++ /dev/null @@ -1,151 +0,0 @@ -.. Licensed to the Apache Software Foundation (ASF) under one - or more contributor license agreements. See the NOTICE file - distributed with this work for additional information - regarding copyright ownership. The ASF licenses this file - to you under the Apache License, Version 2.0 (the - "License"); you may not use this file except in compliance - with the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, - software distributed under the License is distributed on an - "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY - KIND, either express or implied. See the License for the - specific language governing permissions and limitations - under the License. - -Guide to SCM tools in Allura -============================ - -Overview --------- - -The interface API and most of the controller and view structure of -code repository type apps is defined in the base classes in the -Allura package, which consists of the classes in the following -packages: - -* `allura.lib.repository` for the base application and implementation classes -* `allura.controllers.repository` for the base controllers -* `allura.model.repository` for the repo metadata models -* `allura.model.repo_refresh` for the repo metadata refresh logic - - -Application and Implementation ------------------------------- - -The `Application` structure for SCM apps follows the normal pattern for -Allura applications, though they should inherit from -`allura.lib.repository.RepositoryApp` instead of `allura.app.Application`. -The apps are then responsible for implementing subclasses of -`allura.lib.repository.Repository` and `allura.lib.repository.RepositoryImplementation`. - -The `Repository` subclass is responsible for implementing tool-specific -logic using the metadata models and proxying the rest of its logic to the -`RepositoryImplementation` subclass, which talks directly to the underlying -SCM tool, such as `GitPython`, `Mercurial`, or `pysvn`. - -Historically, more was done in the `Repository` subclass using the metadata -models, but we are trying to move away from those toward making the SCM apps -thin wrappers around the underlying SCM tool (see `Indexless`_, below). - - -Controller / View Dispatch --------------------------- - -All of the SCM apps use the base controllers in `allura.controllers.repository` -with only minimal customization through subclassing (primarily to -override the template for displaying instructions for setting up a newly -created repository), so the dispatch for all SCM apps follows the same -pattern. - -The root controller for SCM apps is `allura.controllers.repository.RepoRootController`. -This controller has views for repo-level actions, such as forking and merging, -and the SCM app attaches a refs and a commits (ci) controller to dispatch symbolic -references and explicit commit IDs, respectively. (This should be refactored to be -done in the `RepoRootController` so that the dispatch can be followed more easily.) -(Also, `ForgeSVN` actually eschews this class and uses `BranchBrowser` directly as -its root, in order to tweak the URL slightly, but it monkeypatches the relevant -views over, so the dispatch ends up working more or less the same.) - -The refs controller, `allura.controllers.repository.RefsController`, handles -symbolic references, and in particular handles custom escape handling to detect -what is part of the ref name vs part of the remainder of the URL to dispatch. -This is then handed off to the `BranchBrowserClass` which is a pointer to -the implementation within the specific SCM app which handles the empty -repo instructions or hands it back to the generic commits controller. - -The commits controller, `allura.controllers.repository.CommitsController`, -originally only handled explicit commit IDs, but was modified to allow for -persistent symbolic refs in the URL ("/p/allura/git/ci/master/", e.g.), -it was changed to have the same escape parsing logic as the refs controller. -Regardless, it just parses out the reference / ID from the URL and hands -off to the commit browser. - -The commit browser, `allura.controllers.repository.CommitBrowser`, holds -the views related to a specific commit, such as viewing the commit details -(message and changes), a log of the commit history starting with the commit, -or creating a snapshot of the code as of the commit. It also has a "tree" -endpoint for browsing the file system tree as of the commit. - -The tree browser, `allura.controllers.repository.TreeBrowser`, holds the -view for viewing the file system contents at a specific path for a given -commit. The only complication here is that, instead of parsing out the -entire tree path at once, it recursively dispatches to itself to build -up the path a piece at a time. Tree browsing also depends on the -`Last Commit Logic`_ to create the data needed to display the last -commit that touched each file or directory within the given directory. - - -Last Commit Logic ------------------ - -Determining which commit was the last to touch a given set of files or -directories can be complicated, depending on the specific SCM tool. -Git and Mercurial require manually walking up the commit history to -discover this information, while SVN can return it all from a singlle -`info2` command (though the SVN call will be signficantly slower than -any individual call to Git or Mercurial). Because this can sometimes -be costly to generate, it is cached via the `allura.model.repository.LastCommit` -model. This will generate the data on demand by calling the underlying -SCM tool, if necessary, but the data is currently pre-generated during -the post-push refresh logic for Git and Mercurial. - -The overall logic for generating this data for Git and Mercurial is as follows: - -1. All items modified in the current commit get their info from the - current commit - -2. The info for the remaining items is found: - - * If there is a `LastCommit` record for the parent directory, all of - the remaining items get their info from the previous `LastCommit` - - * Otherwise, the list of remaining items is sent to the SCM implementation, - which repeatedly asks the SCM for the last commit to touch any of the - items for which we are missing information, removing items from the list - as commits are reported as having modified them - -3. Once all of the items have information, or if a processing timeout is reached, - the gathered information is saved in the `LastCommit` model and returned - - - -Indexless ---------- - -Currently, there are model classes which encapsulate SCM metadata -(such as commits, file system structure, etc) in a generic (agnostic to -the underlying tool implementation) way. However, this means that we're -duplicating in mongo a lot of data that is already tracked by the -underlying SCM tool, and this data must also be indexed for new repos -and after every subsequent push before the commits or files are browsable -via the web interface. - -To minimize this duplication of data and reduce or eliminate the delay -between commits being pushed and them being visible, we are trying to -move toward a lightweight API layer that requests the data from the -underlying SCM tool directly, with intelligent caching at the points -and in the format that makes the most sense to make rendering the SCM -pages as fast as possible. http://git-wip-us.apache.org/repos/asf/allura/blob/ae5d2746/Allura/docs/index.rst ---------------------------------------------------------------------- diff --git a/Allura/docs/index.rst b/Allura/docs/index.rst index 7cb4686..fe3a65b 100644 --- a/Allura/docs/index.rst +++ b/Allura/docs/index.rst @@ -20,79 +20,18 @@ You can adapt this file completely to your liking, but it should at least contain the root `toctree` directive. -Introduction -============ -.. toctree:: - :maxdepth: 2 - - intro - -Running Allura -============== - -.. toctree:: - :maxdepth: 2 - - installation - administration - scm_host - -Using Allura -============ - -.. toctree:: - :maxdepth: 2 - - using - -Extending Allura -================ - -.. toctree:: - :maxdepth: 3 - - extending - -* Writing an Allura-based app - * `Getting Started `_ - * `Forms, Artifacts, and Testing `_ - * `Adding a Custom Icon `_ - * `Adding a Sidebar Menu `_ - * `Adding Custom CSS `_ - -Developing Allura -================= - -.. toctree:: - :maxdepth: 3 - - contributing - platform - platform_tour - guides/message_bus - guides/email - guides/permissions - guides/scm - tutorials/testing - - -API Documentation -================= - -.. toctree:: - :maxdepth: 2 - :glob: - - api/* +***************** +Table of contents +***************** -Background Info -=============== .. toctree:: - :maxdepth: 1 + :maxdepth: 3 - faq - online + getting_started/index + platform/index + development/index + api/index Indices and tables ================== http://git-wip-us.apache.org/repos/asf/allura/blob/ae5d2746/Allura/docs/installation.rst ---------------------------------------------------------------------- diff --git a/Allura/docs/installation.rst b/Allura/docs/installation.rst deleted file mode 100644 index ce26d06..0000000 --- a/Allura/docs/installation.rst +++ /dev/null @@ -1,108 +0,0 @@ -.. Licensed to the Apache Software Foundation (ASF) under one - or more contributor license agreements. See the NOTICE file - distributed with this work for additional information - regarding copyright ownership. The ASF licenses this file - to you under the Apache License, Version 2.0 (the - "License"); you may not use this file except in compliance - with the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, - software distributed under the License is distributed on an - "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY - KIND, either express or implied. See the License for the - specific language governing permissions and limitations - under the License. - -Installation -============ - -Install -------- - -Our step-by-step setup instructions are in our INSTALL.markdown file. You can read it online at https://forge-allura.apache.org/p/allura/git/ci/master/tree/INSTALL.markdown You should be able to get Allura up and running in well under an hour by following those instructions. - -For a faster and easier setup, see our `Vagrant/VirtualBox installation guide `_ - -Configuring Optional Features ------------------------------ - -The `development.ini` file has many options you can explore and configure. It is geared towards development, so you will want to review -carefully and make changes for production use. - -To run SVN and Git services, see the :doc:`scm_host` page. - -Some features may be added as separate `Allura extensions `_ - -Enabling inbound email -^^^^^^^^^^^^^^^^^^^^^^ - -Allura can listen for email messages and update tools and artifacts. For example, every ticket has an email address, and -emails sent to that address will be added as comments on the ticket. To set up the SMTP listener, run: - -.. code-block:: shell-session - - (env-allura)~/src/forge/Allura$ nohup paster smtp_server development.ini > /var/log/allura/smtp.log & - -By default this uses port 8825. Depending on your mail routing, you may need to change that port number. -And if the port is in use, this command will fail. You can check the log file for any errors. -To change the port number, edit `development.ini` and change `forgemail.port` to the appropriate port number for your environment. - -SMTP in development -^^^^^^^^^^^^^^^^^^^ - -The following command can be used for quick and easy monitoring of smtp during development. -Just be sure the port matches the `smtp_port` from your `development.ini` (8826 by default). - -.. code-block:: shell-session - - (env-allura)~/src/forge/Allura$ python -m smtpd -n -c DebuggingServer localhost:8826 - -This will create a new debugging server that discards messages and prints them to stdout. - - -Using LDAP -^^^^^^^^^^ - -Allura has a pluggable authentication system, and can use an existing LDAP system. In your config -file (e.g. :file:`development.ini`), there are several "ldap" settings to set: - -* Change auth.method to: :samp:`auth.method = ldap` -* Set all the :samp:`auth.ldap.{*}` settings to match your LDAP server configuration. (:samp:`auth.ldap.schroot_name` won't be - used, don't worry about it.) -* Keep :samp:`auth.ldap.autoregister = true` This means Allura will use existing users from your LDAP - server. -* Set :samp:`auth.allow_user_registration = false` since your users already are present in LDAP. -* Change user_prefs_storage.method to :samp:`user_prefs_storage.method = ldap` -* Change :samp:`user_prefs_storage.ldap.fields.display_name` if needed (e.g. if display names are stored - in a different LDAP attribute). - -Restart Allura and you should be all set. Now users can log in with their LDAP credentials and their -Allura records will be automatically created the first time they log in. - -Note: if you want users to register new accounts into your LDAP system via Allura, you should turn -off :samp:`autoregister` and turn on :samp:`allow_user_registration` - -Enabling RabbitMQ -^^^^^^^^^^^^^^^^^ - -For faster notification of background jobs, you can use RabbitMQ. Assuming a base setup from the INSTALL, run these commands -to install rabbitmq and set it up: - -.. code-block:: shell-session - - (env-allura)~$ sudo aptitude install rabbitmq-server - (env-allura)~$ sudo rabbitmqctl add_user testuser testpw - (env-allura)~$ sudo rabbitmqctl add_vhost testvhost - (env-allura)~$ sudo rabbitmqctl set_permissions -p testvhost testuser "" ".*" ".*" - (env-allura)~$ pip install amqplib==0.6.1 kombu==1.0.4 - -Then edit Allura/development.ini and change `amqp.enabled = false` to `amqp.enabled = true` and uncomment the other `amqp` settings. - -If your `paster taskd` process is still running, restart it: - -.. code-block:: shell-session - - (env-allura)~/src/forge/Allura$ pkill -f taskd - (env-allura)~/src/forge/Allura$ nohup paster taskd development.ini > /var/log/allura/taskd.log & \ No newline at end of file http://git-wip-us.apache.org/repos/asf/allura/blob/ae5d2746/Allura/docs/intro.rst ---------------------------------------------------------------------- diff --git a/Allura/docs/intro.rst b/Allura/docs/intro.rst deleted file mode 100644 index 341ec25..0000000 --- a/Allura/docs/intro.rst +++ /dev/null @@ -1,41 +0,0 @@ -.. Licensed to the Apache Software Foundation (ASF) under one - or more contributor license agreements. See the NOTICE file - distributed with this work for additional information - regarding copyright ownership. The ASF licenses this file - to you under the Apache License, Version 2.0 (the - "License"); you may not use this file except in compliance - with the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, - software distributed under the License is distributed on an - "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY - KIND, either express or implied. See the License for the - specific language governing permissions and limitations - under the License. - -Rather than build yet another forge, we decided to do something new. We wanted to build a new kind of extensible forge, and a new set of highly integrated forge tools. - -Allura is an **open** platform for **open** processes ------------------------------------------------------ - -It's easy to get frustrated with existing development tools. Too often they are overbearing, complex, and make assumptions that get in your way. And even if they are open source, it's often difficult to get them to work the way you need them too. - -Which is why we created Allura. It's designed to be truly **open**, in many different senses of the word. - -It's open in bunch of ways: - -* It's a combination of tools available under *Free* or *Open Source* licenses. -* It's designed around a plugin architecture, and anybody willing to contribute a tool can play. -* It's being hosted publicly, and is built through contributions of individuals and companies who want to promote Open Source development. -* It's designed to provide a structure around which welcoming (open) communities can grow. -* Its core tools are designed around inclusive development processes. - -We looked at existing forges, but to achieve all those goals, we decided we needed to build something new. - -Allura is designed to support an **ecosystem** ----------------------------------------------- - -Allura is at once a **set of tools** to help people collaboratively develop software, and an **open platform** on which innovative new tools be built. - http://git-wip-us.apache.org/repos/asf/allura/blob/ae5d2746/Allura/docs/online.rst ---------------------------------------------------------------------- diff --git a/Allura/docs/online.rst b/Allura/docs/online.rst deleted file mode 100644 index e68dd44..0000000 --- a/Allura/docs/online.rst +++ /dev/null @@ -1,54 +0,0 @@ -.. Licensed to the Apache Software Foundation (ASF) under one - or more contributor license agreements. See the NOTICE file - distributed with this work for additional information - regarding copyright ownership. The ASF licenses this file - to you under the Apache License, Version 2.0 (the - "License"); you may not use this file except in compliance - with the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, - software distributed under the License is distributed on an - "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY - KIND, either express or implied. See the License for the - specific language governing permissions and limitations - under the License. - -Online References -================= - -Generated API docs, useful for browsing through the code, viewing inheritance, etc: - -* http://allura.sourceforge.net/epydoc/ - -Our project page, including tickets, discussion forums, etc.: - -* https://allura.apache.org/ - - -Much of the current forge was inspired by Roundup -------------------------------------------------- - -http://roundup.sourceforge.net/index.html - -After we told ESR about roundup, he posted some interesting ideas about how to "federate" roundup entities "namespaces" that are ultimately just url-prefixes per project: - -http://esr.ibiblio.org/?p=1359 - -Message based system needed ---------------------------- - -http://ejohn.org/blog/google-groups-is-dead/ - - -Potential pitfalls for groupware development --------------------------------------------- - -http://research.microsoft.com/en-us/um/people/jgrudin/past/papers/cacm94/cacm94.html - -AMQP Online References ----------------------- - -http://blogs.digitar.com/jjww/2009/01/rabbits-and-warrens/ -http://www.igvita.com/2009/10/08/advanced-messaging-routing-with-amqp/ http://git-wip-us.apache.org/repos/asf/allura/blob/ae5d2746/Allura/docs/platform.rst ---------------------------------------------------------------------- diff --git a/Allura/docs/platform.rst b/Allura/docs/platform.rst deleted file mode 100644 index acf0124..0000000 --- a/Allura/docs/platform.rst +++ /dev/null @@ -1,118 +0,0 @@ -.. Licensed to the Apache Software Foundation (ASF) under one - or more contributor license agreements. See the NOTICE file - distributed with this work for additional information - regarding copyright ownership. The ASF licenses this file - to you under the Apache License, Version 2.0 (the - "License"); you may not use this file except in compliance - with the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, - software distributed under the License is distributed on an - "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY - KIND, either express or implied. See the License for the - specific language governing permissions and limitations - under the License. - -Platform Architecture overview -============================== - -I'm told that the reason you build a platform is to "reduce the marginal cost -of developing applications." Sounds good. Well, actually it sounds a bit -dry. But it's about right, we want to make creating new online development -tools faster, easier, and more fun, which I guess is the "reduce the marginal -cost" thing. - -Platform building blocks ------------------------- - -Before we get into the details of how to extend the Allura platform, perhaps -it would be smart to explain some of the big pieces and why they are there. - -We wanted Allura tools to be fast, we needed them to scale, and we had some -complex requirements for data storage and extensibility. So, we needed a -**fast,** flexible, and easy to use data persistence system. - -We were very impressed by the general message architecture of Roundup, but we -wanted to extend it from just email messages to include scm commits, and we -added a message bus (RabbitMQ which we'll talk about in a second), to make -it fast. - -.. image:: _static/images/messages.png - :alt: Message Architecture - -We were also impressed by the flexibility of Roundup's Hypertable system in -allowing for ad-hock ticket schema additions. - -It definitely seemed like something we wanted in a next generation forge, -because we wanted app tools to be able to: - -* create and version their own document types, -* extend existing document structures, -* and to mange document revisions, access control lists, and other - platform level data. - -In spite of the power and flexibility of the Roundup HyperTable -implementation, we had some concerns about performance and scalability. - -Fortunately several of the Allura authors used MongoDB -in rewriting the download flow of SourceForge.net, and knew that it could -handle huge loads (we saturated a 2gb network connection on the server -with 6% cpu utilization). - -We also knew that MongoDB's flexible replication system would allow us -to build the forge in such a way that we could easily provide a -package of all project data to developers concerned about lock-in. - -Not only that but Rick Copeland had built a couple of custom Object -*Non*-Relational Mappers (ONRMs?) before, including one for MongoDB, -and he whipped up Ming, which backed on MongoDB and gave us exactly -what we needed. - -As I mentioned before we also needed a fast, flexible message bus and queuing -system. RabbitMQ was (lightning) fast, (shockingly) flexible, but not super -easy to use. Fortunately we didn't have to roll our own wrapper here, as -the Python community already whipped up Carrot, and Celery, which made -working with the RabbitMQ based AMQP bus a LOT easer. - - -Application Tools ------------------ - -Writing a tool for Allura is as simple as defining a few controllers -to handle particular URL's, templates to render pages, and defining the schemas -of any Allura document types that your tool requires. - -.. image:: _static/images/tools.png - :alt: App Tools - :align: right - -When you write Allura tools, you'll get lots of stuff for free: - -* Search-ability of your Artifacts -* Artifact versioning for accountability and transparency -* Ability to extend existing Artifacts -* Reuse central User/group/permission management -* A robust and flexible permissions system -* Access to a real-time event publishing system - -What's in a tool? -~~~~~~~~~~~~~~~~~ - -The most basic app tool consists of a few things: - -* A controller object (instantiated per request) -* Template files (optional) -* UI Widgets (optional) -* Extensions to existing Artifacts (optional) -* New Artifact types (optional) -* Event listener tools (optional) -* Event publisher (optional) - -Users/groups and Permissions -~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -In order to facilitate more open processes, where more users can contribute --- while still protecting data -- documents can easily be "versioned", and -the platform provides tools to manage versioned documents for you. http://git-wip-us.apache.org/repos/asf/allura/blob/ae5d2746/Allura/docs/platform/email.rst ---------------------------------------------------------------------- diff --git a/Allura/docs/platform/email.rst b/Allura/docs/platform/email.rst new file mode 100644 index 0000000..754a278 --- /dev/null +++ b/Allura/docs/platform/email.rst @@ -0,0 +1,71 @@ +.. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +***** +Email +***** + + +Email routing +------------- + +routing mechanism will be a dotted path from the project to +the application/tool to the specific artifact within that app that is +used like this:: + + subproject.app.artifact@project.example.com + +Which would translate into the devtools project which is a subproject of +turbogears, and it's bug tracker and ticket 142 in that tracker:: + + devtools.bug.142@turbogears.sf.net + + +And it in turn would be published to the message bus, which will assure +that all tools that are registered to be notified for that e-mail +addresses are called like that. + +If your app has more than one artifact type, you could nest them inside +`project.app.something.id.*` + +If you were working with the bug tracker directly on the TurboGears project:: + + bug.142@turbogears.sf.net + +The Allura platform allows you to setup other message types, such as commit +messages, to go into amqp with the same routing information, and turn into +"messages" just like e-mail. + +Email Content Handling +---------------------- + +On Allura message bodies should be composed as markdown. +Multi-part mime encoded messages should be sent include plain text +(the markdown) and html (rendered from the markdown). + +Users are allowed to register that they want plain text only. + +We will also include some text in the footer of the e-mail message with a +link to the message online. We can use this link to guess where in the +thread the message belongs in the case of a messed up e-mail client that +does not set the headers for the reply properly. + +The nice thing about this is that it's pretty much already implemented +for us via the meta tool. + +This metadata syntax will let you set fields on tickets and otherwise +interact with the system via e-mail, assuming you have such permissions. http://git-wip-us.apache.org/repos/asf/allura/blob/ae5d2746/Allura/docs/platform/index.rst ---------------------------------------------------------------------- diff --git a/Allura/docs/platform/index.rst b/Allura/docs/platform/index.rst new file mode 100644 index 0000000..36fd94e --- /dev/null +++ b/Allura/docs/platform/index.rst @@ -0,0 +1,30 @@ +.. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +******** +Platform +******** + +.. toctree:: + :maxdepth: 2 + + platform + platform_tour + email + message_bus + permissions + scm http://git-wip-us.apache.org/repos/asf/allura/blob/ae5d2746/Allura/docs/platform/message_bus.rst ---------------------------------------------------------------------- diff --git a/Allura/docs/platform/message_bus.rst b/Allura/docs/platform/message_bus.rst new file mode 100644 index 0000000..8211936 --- /dev/null +++ b/Allura/docs/platform/message_bus.rst @@ -0,0 +1,101 @@ +.. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +*********** +Message Bus +*********** + +Guide to the Allura task and event system +========================================= + +Our event system is driven by a MongoDB-based queuing system, most of which you +can ignore, because we've simplified it down to two ideas: *tasks* and *event handlers*. + +Glossary +-------- + +Before we get into the details perhaps a few definitions are in order: + +* **app** -- tool for allura such as the tracker, scm, or wiki apps +* **task** -- callable defined in a app that gets invoked by the `taskd` daemon +* **event handler** -- callable defined in an app that gets called on message + events. Event handlers are identified by a string name, so a single event can + fan out to multiple callables. + +Tasks +----- + +The `MonQTask` class is central to the Allura asynchronous processing system. +Simply put, a `MonQTask` is a document in MongoDB that contains a context +(project/app/user), a function pointer (specified as a dotted string), and +arguments to that function. Tasks are scheduled by creating `MonQTask` +documents, and the `taskd` daemon executes them as though they were happening in +the web context. + +To simplify the use of tasks, Allura provides a decorator `@task` that marks a +function as 'taskable.' This decorator adds a `.post` method to the function +object that allows the function to be scheduled as a MonQTask. For instance, the +`commit` task (flushing Solr caches) is defined in as the following:: + + @task + def commit(): + g.solr.commit() + +In order to schedule this task for execution by taskd, simply use the `.post` +method:: + + commit.post() + +If we wanted to call `commit` directly (e.g. in a test), we can still do that as +well:: + + commit() + +.. _events: + +Events +------ + +Events provide fanout capability for messages, letting several functions get +called in response to the same 'event.' To note a function as an event handler, +you use the `@event_handler` decorator. For instance, there is an event handler +on all project updates to subscribe the project's admins to project changes:: + + @event_handler('project_updated') + def subscribe_admins(topic): + c.app.subscribe_admins() + +In order to invoke all the event handlers for a particular topic, we use the +`g.post_event` helper:: + + g.post_event('project_updated') + +Under the covers, this is scheduling an `event` task that calls all the handlers +for a particular named event. Note that you can pass arguments (\*args, and +\*\*kwargs) to event handlers just like you do to tasks, with the exception that +the topic name (above, this would be 'project_updated') is always the first +parameter passed to the event handler. + +Running the Task Daemon +----------------------- + +In order to actually run the asynchronous tasks, we have written a paster command +`taskd`. This creates a configurable number of worker processes that watch for +changes to the `MonQTask` collection and execute requested tasks. `taskd` can be +run on any server, but should have similar access to the MongoDB databases and +configuration files used to run the web app server, as it tries to replicate the +request context as closely as possible when running tasks. http://git-wip-us.apache.org/repos/asf/allura/blob/ae5d2746/Allura/docs/platform/permissions.rst ---------------------------------------------------------------------- diff --git a/Allura/docs/platform/permissions.rst b/Allura/docs/platform/permissions.rst new file mode 100644 index 0000000..77b4a0f --- /dev/null +++ b/Allura/docs/platform/permissions.rst @@ -0,0 +1,64 @@ +.. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +*********** +Permissions +*********** + +Guide to Users, Groups and Permissions in Allura +================================================ + +User/Group model +---------------- + +In the allura system `users` can be assigned to various `groups` or +roles on a per-project basis. + +Users can be members of many groups, and `groups` can +be assigned a list of `permissions` like `edit`, +`mdoderate` or `read`. Tools can define their own +set of permissions, for their artifacts. + +Individual artifacts and ACL's +------------------------------ + +You may want to assign a permission +to particular people or roles for a specific `Artifact` such as +one bug in the ticket tracker. The Allura platform supports this via +an additive ACL field on every `Artifact` instance. It is not exposed +via the UI currently. + +Permission hierarchy +-------------------- + +Projects and subprojects can define user groups, but for any particular +subproject the set of groups the user belongs to is additive. This follows +the basic principle that sub-project permissions and artifact permissions +can *allow* additional access, but can't *restrict* it beyond +what permissions are allowed by a higher level project. + +Permission predicates +--------------------- + +Predicates are simple functions, several of which are defined in Allura +itself, and which can be added by any tool, which return true if +permission is granted, and false if it is not. + +An example predicate function `has_project_access` takes two params, an object +and a `permission` string. It then checks to see if the current user +(picked up from the environment) has permission to perform that action on +that object, following the rules above. http://git-wip-us.apache.org/repos/asf/allura/blob/ae5d2746/Allura/docs/platform/platform.rst ---------------------------------------------------------------------- diff --git a/Allura/docs/platform/platform.rst b/Allura/docs/platform/platform.rst new file mode 100644 index 0000000..7e64755 --- /dev/null +++ b/Allura/docs/platform/platform.rst @@ -0,0 +1,137 @@ +.. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +***************** +Platform Overview +***************** + + + +I'm told that the reason you build a platform is to "reduce the marginal cost +of developing applications." Sounds good. Well, actually it sounds a bit +dry. But it's about right, we want to make creating new online development +tools faster, easier, and more fun, which I guess is the "reduce the marginal +cost" thing. + + +Platform building blocks +------------------------ + +Before we get into the details of how to extend the Allura platform, perhaps +it would be smart to explain some of the big pieces and why they are there. + +We wanted Allura tools to be fast, we needed them to scale, and we had some +complex requirements for data storage and extensibility. So, we needed a +**fast,** flexible, and easy to use data persistence system. + +We were very impressed by the general message architecture of Roundup, but we +wanted to extend it from just email messages to include scm commits, and we +added a message bus (RabbitMQ which we'll talk about in a second), to make +it fast. + +.. image:: ../_static/images/messages.png + :alt: Message Architecture + +We were also impressed by the flexibility of Roundup's Hypertable system in +allowing for ad-hock ticket schema additions. + +It definitely seemed like something we wanted in a next generation forge, +because we wanted app tools to be able to: + +* create and version their own document types, +* extend existing document structures, +* and to mange document revisions, access control lists, and other + platform level data. + +In spite of the power and flexibility of the Roundup HyperTable +implementation, we had some concerns about performance and scalability. + +Fortunately several of the Allura authors used MongoDB +in rewriting the download flow of SourceForge.net, and knew that it could +handle huge loads (we saturated a 2gb network connection on the server +with 6% cpu utilization). + +We also knew that MongoDB's flexible replication system would allow us +to build the forge in such a way that we could easily provide a +package of all project data to developers concerned about lock-in. + +Not only that but Rick Copeland had built a couple of custom Object +*Non*-Relational Mappers (ONRMs?) before, including one for MongoDB, +and he whipped up Ming, which backed on MongoDB and gave us exactly +what we needed. + +As I mentioned before we also needed a fast, flexible message bus and queuing +system. RabbitMQ was (lightning) fast, (shockingly) flexible, but not super +easy to use. Fortunately we didn't have to roll our own wrapper here, as +the Python community already whipped up Carrot, and Celery, which made +working with the RabbitMQ based AMQP bus a LOT easer. + + +Application Tools +----------------- + +Writing a tool for Allura is as simple as defining a few controllers +to handle particular URL's, templates to render pages, and defining the schemas +of any Allura document types that your tool requires. + +.. image:: ../_static/images/tools.png + :alt: App Tools + :align: right + +When you write Allura tools, you'll get lots of stuff for free: + +* Search-ability of your Artifacts +* Artifact versioning for accountability and transparency +* Ability to extend existing Artifacts +* Reuse central User/group/permission management +* A robust and flexible permissions system +* Access to a real-time event publishing system + +What's in a tool? +~~~~~~~~~~~~~~~~~ + +The most basic app tool consists of a few things: + +* A controller object (instantiated per request) +* Template files (optional) +* UI Widgets (optional) +* Extensions to existing Artifacts (optional) +* New Artifact types (optional) +* Event listener tools (optional) +* Event publisher (optional) + +Users/groups and Permissions +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +In order to facilitate more open processes, where more users can contribute +-- while still protecting data -- documents can easily be "versioned", and +the platform provides tools to manage versioned documents for you. + + + +Why create all the tools as plugins? +------------------------------------ + +We know that some projects are going to want more locked down +access controls in their bug trackers, or more workflow based +processes. These things are inevitable, and we really do want +to support them, but at the same time they are going to conflict +with the way many other projects want to work. + +Building a plugin (tool in Allura terms) system, and standard +integration points makes it possible to serve everybody in one +way or another. \ No newline at end of file http://git-wip-us.apache.org/repos/asf/allura/blob/ae5d2746/Allura/docs/platform/platform_tour.rst ---------------------------------------------------------------------- diff --git a/Allura/docs/platform/platform_tour.rst b/Allura/docs/platform/platform_tour.rst new file mode 100644 index 0000000..c3ca064 --- /dev/null +++ b/Allura/docs/platform/platform_tour.rst @@ -0,0 +1,258 @@ +.. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +************* +Platform Tour +************* + +Introduction +^^^^^^^^^^^^ + +Allura is implemented as a collection of tool applications on top of a +robust and open platform. Some of the services provided by the platform include: + +- Indexing and search +- Authentication and Authorization +- Email integration (every tool application gets its own email address) +- Asynchronous processing with background tasks and events +- `Markdown `_ markup formatting +- Simple autolinking between different artifacts in the forge +- Attachment handling +- Tool administration + +Tools, on the other hand, provide the actual user interface and logic to +manipulate forge artifacts. Some of the tools currently implemented include: + +admin + This tool is installed in all projects, and allows the administration of the + project's tools, authentication, and authorization +Git, Hg, SVN + These tools allow you to host a version control system in Allura. + They also provides the ability to "fork" Git and Hg repos in order to + provide your own extensions. +Wiki + This tool provides a basic wiki with support for comments, attachments, and + notifications. +Tracker + This tool provides an extensible ticketing system for tracking feature + requests, defects, or support requests. +Discussion + This tool provides a forum interface with full email integration as well. + The forum also handles attachments to posts either via the web interface or via email. + +The Context Object +------------------ + +The Pylons "context" object `c` has several properties which are automatically +set for each request: + +project + The current project +app + The current tool application object. +user + The current user + +Allura platform provides the following functions to manage the context object, +if you need to change the context for some situation: + +.. function:: allura.lib.helpers.push_config(obj, **kw) + :noindex: + + Context manager (used with the `with` statement) used to temporarily set the + attributes of a particular object, resetting them to their previous values + at the end of the `with` block. Used most frequently with the context object + `c`:: + + c.project = some_project + with push_config(c, project=other_project): + ... + # code in this block will have c.project == other_project + # code here will have c.project == some_project + +.. function:: allura.lib.helpers.set_context(project_id, mount_point=None, app_config_id=None) + :noindex: + + Set the context object `c` according to the given `project_id` and optionally either a + `mount_point`, an `app_config_id`. `c.project` is set to the corresponding + project object. If the mount_point or app_config_id is + specified, then `c.app` will be set to the corresponding tool application + object. + + +Artifacts +--------- + +We've mentioned artifacts a couple of times now without definition. An artifact, +as used in Allura, is some object that a tool needs to store in the +forge. The platform provides facilities for controlling access to individual +artifacts if that's what the tool designer favors. For instance, the Discussion +tool allows a user to edit or delete their own posts, but not to edit or delete +others (unless the user has the 'moderate' permission on the forum itself). +Some examples of artifacts in the current tools: + +- Discussion: Forum, Thread, Post, Attachment +- Wiki: Page, Comment, Attachment +- Tracker: Ticket, Comment, Attachment +- SCM: Repository, Commit, Patch + +In order to implement your own artifact, you should override at least a few of +the methods of the `allura.model.artifact.Artifact` class:: + + from ming.orm.property import FieldProperty + from allura.model import Artifact + + class NewArtifact(Artifact): + class __mongometa__: + name='my_new_artifact' # collection where this artifact is stored + type_s = 'My Artifact' # 'type' of the artifact used in search results + + # Add your own properties here (beyond those provided by Artifact) + shortname = FieldProperty(str) + + def url(self): + 'Each artifact should have its own URL ' + return self.app.url + self.shortname + '/' + + def index(self): + 'Return the fields you want indexed on this artifact' + result = Artifact.index(self) + result.update(type_s=self.type_s, + name_s=self.shortname, + text=self.shortname) + return result + + def shorthand_id(self): + 'Used in the generation of short links like [my_artifact]' + return self.shortname + +Platform services provided for artifacts +---------------------------------------- + +Whenever you create, modify, or delete an artifact, the platform does a couple of +things for you: + +- The artifact is added to the index and will appear in searches +- A shortlink is generated for the artifact (e.g. [MyWikiPage] or [#151]). This allows you + to reference the artifact from other artifacts. Whenever the commit message + is displayed in the SCM tool, any references to `[#151]` will be + automatically linked to that Ticket's page. + +Shortlinks work only within a project hierarchy (in order to link to some other project's +page, you'll have to use the full URL). Sometimes, a shortlink may need to be +differentiated based on its location in a subproject or in one of many tools of +the same type within a project. In order to do this, shortlinks may be prefixed +by either the tool mount point or a project ID and tool mount point. + +For instance, suppose we have an ticket tracker called 'features' and one called 'bugs'. +They both have many tickets in them. To distinguish, use the tracker mount point +within the reference. For example [features:#3] or [bugs:#3] + +Asynchronous Processing +----------------------- + +Much of the actual functionality of Allura comes from code that runs +*outside* the context of a web request, in the `taskd` server (invoked by +running `paster taskd development.ini`. Asynchronous processing is performed +by two types of functions, *tasks* and *events*, differentiated as follows: + +Task + Tasks are module-level global functions. They are annotated with the `@task` + decorator and are invoked with the `.post` method. For instance, to schedule + a task `foobar` to execute in the `taskd` context, you would write:: + + @task + def foobar(a,b,c=5): ... + + foobar.post(9,1,c=15) + +Event + Events are intended for "fan-out" types of events. Events have a string + name, and are "listened" for by using the `@event_handler` decorator. The + `g.post_event()` helper is provided to run the event handlers for a + particular event in the `taskd` context. Multiple event handlers can be + registered for each event:: + + @event_handler('event_name') + def handler1(topic, *args, **kwargs): ... + + @event_handler('event_name') + def handler2(topic, *args, **kwargs): ... + + g.post_event('event_name', 1,2,3, a=5) + + +Email Integration +----------------- + +The Allura platform provides easy-to-use email integration. Forge email addresses +are of the form +@[.]*..projects.sourceforge.net. +When a message is received on such an email address, the address is parsed and +the sending user is identified (if possible). Based on the parsed address, the +pylons context attributes `c.project` and `c.app` are set, and the application is +queried to determine whether the identified user has authority to send an email +to the given app/topic combination by calling `c.app.has_access(user, topic)`. +If the user has access, the message is decomposed into its component parts (if a +multipart MIME-encoded message) and `c.app.handle_message(topic, message)` is +called for each part with the following components to the `msg` dict: + +headers + The actual headers parsed from the body of the message +message_id + The `Message-ID` header (which should be universally + unique and is + generated by the email client), used for determining which messages are replies + to which other messages +in_reply_to + The `In-Reply-To` header, used for determining which messages are replies to + which other messages +references + The `References` header, used for determining which messages refer to + which other messages +filename + Optional, if the part is an attachment with a filename, this will be populated +content_type + The MIME content_type of the message part +payload + The actual content of the message part + +The Allura platform also provides full support for *sending* email without +worrying about the specifics of SMTP or sendmail handling. In order to send an +email, simply post a task for `allura.tasks.mail_tasks.sendmail` with the +following arguments: + +fromaddr + Return address on the message (usually the topic@tool_name that generated + it) +destinations + List of email addresses and/or :class:`bson.ObjectId` s for + :class:`allura.model.auth.User` objects +text + Markdown-formatted body of the message (If the user has requested html or + combined text+html messages in their preferences, the Markdown will be so + rendered. Otherwise a plain text message will be sent.) +reply_to + Address to which replies should be sent +subject + Subject of the message +message_id + Value to put in the `Message-ID` header (the `_id` field of a + :class:`allura.model.artifact.Message` is suitable for this) +in_reply_to (optional) + Value to put in the `In-Reply-To` header (the `parent_id` field of a + :class:`allura.model.artifact.Message` is suitable for this) http://git-wip-us.apache.org/repos/asf/allura/blob/ae5d2746/Allura/docs/platform/scm.rst ---------------------------------------------------------------------- diff --git a/Allura/docs/platform/scm.rst b/Allura/docs/platform/scm.rst new file mode 100644 index 0000000..d8ab6d6 --- /dev/null +++ b/Allura/docs/platform/scm.rst @@ -0,0 +1,150 @@ +.. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +******************* +SCM tools in Allura +******************* + + +The interface API and most of the controller and view structure of +code repository type apps is defined in the base classes in the +Allura package, which consists of the classes in the following +packages: + +* `allura.lib.repository` for the base application and implementation classes +* `allura.controllers.repository` for the base controllers +* `allura.model.repository` for the repo metadata models +* `allura.model.repo_refresh` for the repo metadata refresh logic + + +Application and Implementation +------------------------------ + +The `Application` structure for SCM apps follows the normal pattern for +Allura applications, though they should inherit from +`allura.lib.repository.RepositoryApp` instead of `allura.app.Application`. +The apps are then responsible for implementing subclasses of +`allura.lib.repository.Repository` and `allura.lib.repository.RepositoryImplementation`. + +The `Repository` subclass is responsible for implementing tool-specific +logic using the metadata models and proxying the rest of its logic to the +`RepositoryImplementation` subclass, which talks directly to the underlying +SCM tool, such as `GitPython`, `Mercurial`, or `pysvn`. + +Historically, more was done in the `Repository` subclass using the metadata +models, but we are trying to move away from those toward making the SCM apps +thin wrappers around the underlying SCM tool (see `Indexless`_, below). + + +Controller / View Dispatch +-------------------------- + +All of the SCM apps use the base controllers in `allura.controllers.repository` +with only minimal customization through subclassing (primarily to +override the template for displaying instructions for setting up a newly +created repository), so the dispatch for all SCM apps follows the same +pattern. + +The root controller for SCM apps is `allura.controllers.repository.RepoRootController`. +This controller has views for repo-level actions, such as forking and merging, +and the SCM app attaches a refs and a commits (ci) controller to dispatch symbolic +references and explicit commit IDs, respectively. (This should be refactored to be +done in the `RepoRootController` so that the dispatch can be followed more easily.) +(Also, `ForgeSVN` actually eschews this class and uses `BranchBrowser` directly as +its root, in order to tweak the URL slightly, but it monkeypatches the relevant +views over, so the dispatch ends up working more or less the same.) + +The refs controller, `allura.controllers.repository.RefsController`, handles +symbolic references, and in particular handles custom escape handling to detect +what is part of the ref name vs part of the remainder of the URL to dispatch. +This is then handed off to the `BranchBrowserClass` which is a pointer to +the implementation within the specific SCM app which handles the empty +repo instructions or hands it back to the generic commits controller. + +The commits controller, `allura.controllers.repository.CommitsController`, +originally only handled explicit commit IDs, but was modified to allow for +persistent symbolic refs in the URL ("/p/allura/git/ci/master/", e.g.), +it was changed to have the same escape parsing logic as the refs controller. +Regardless, it just parses out the reference / ID from the URL and hands +off to the commit browser. + +The commit browser, `allura.controllers.repository.CommitBrowser`, holds +the views related to a specific commit, such as viewing the commit details +(message and changes), a log of the commit history starting with the commit, +or creating a snapshot of the code as of the commit. It also has a "tree" +endpoint for browsing the file system tree as of the commit. + +The tree browser, `allura.controllers.repository.TreeBrowser`, holds the +view for viewing the file system contents at a specific path for a given +commit. The only complication here is that, instead of parsing out the +entire tree path at once, it recursively dispatches to itself to build +up the path a piece at a time. Tree browsing also depends on the +`Last Commit Logic`_ to create the data needed to display the last +commit that touched each file or directory within the given directory. + + +Last Commit Logic +----------------- + +Determining which commit was the last to touch a given set of files or +directories can be complicated, depending on the specific SCM tool. +Git and Mercurial require manually walking up the commit history to +discover this information, while SVN can return it all from a singlle +`info2` command (though the SVN call will be signficantly slower than +any individual call to Git or Mercurial). Because this can sometimes +be costly to generate, it is cached via the `allura.model.repository.LastCommit` +model. This will generate the data on demand by calling the underlying +SCM tool, if necessary, but the data is currently pre-generated during +the post-push refresh logic for Git and Mercurial. + +The overall logic for generating this data for Git and Mercurial is as follows: + +1. All items modified in the current commit get their info from the + current commit + +2. The info for the remaining items is found: + + * If there is a `LastCommit` record for the parent directory, all of + the remaining items get their info from the previous `LastCommit` + + * Otherwise, the list of remaining items is sent to the SCM implementation, + which repeatedly asks the SCM for the last commit to touch any of the + items for which we are missing information, removing items from the list + as commits are reported as having modified them + +3. Once all of the items have information, or if a processing timeout is reached, + the gathered information is saved in the `LastCommit` model and returned + + + +Indexless +--------- + +Currently, there are model classes which encapsulate SCM metadata +(such as commits, file system structure, etc) in a generic (agnostic to +the underlying tool implementation) way. However, this means that we're +duplicating in mongo a lot of data that is already tracked by the +underlying SCM tool, and this data must also be indexed for new repos +and after every subsequent push before the commits or files are browsable +via the web interface. + +To minimize this duplication of data and reduce or eliminate the delay +between commits being pushed and them being visible, we are trying to +move toward a lightweight API layer that requests the data from the +underlying SCM tool directly, with intelligent caching at the points +and in the format that makes the most sense to make rendering the SCM +pages as fast as possible. http://git-wip-us.apache.org/repos/asf/allura/blob/ae5d2746/Allura/docs/platform_tour.rst ---------------------------------------------------------------------- diff --git a/Allura/docs/platform_tour.rst b/Allura/docs/platform_tour.rst deleted file mode 100644 index 06d3140..0000000 --- a/Allura/docs/platform_tour.rst +++ /dev/null @@ -1,257 +0,0 @@ -.. Licensed to the Apache Software Foundation (ASF) under one - or more contributor license agreements. See the NOTICE file - distributed with this work for additional information - regarding copyright ownership. The ASF licenses this file - to you under the Apache License, Version 2.0 (the - "License"); you may not use this file except in compliance - with the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, - software distributed under the License is distributed on an - "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY - KIND, either express or implied. See the License for the - specific language governing permissions and limitations - under the License. - -Platform Tour -============= - -Introduction ------------- - -Allura is implemented as a collection of tool applications on top of a -robust and open platform. Some of the services provided by the platform include: - -- Indexing and search -- Authentication and Authorization -- Email integration (every tool application gets its own email address) -- Asynchronous processing with background tasks and events -- `Markdown `_ markup formatting -- Simple autolinking between different artifacts in the forge -- Attachment handling -- Tool administration - -Tools, on the other hand, provide the actual user interface and logic to -manipulate forge artifacts. Some of the tools currently implemented include: - -admin - This tool is installed in all projects, and allows the administration of the - project's tools, authentication, and authorization -Git, Hg, SVN - These tools allow you to host a version control system in Allura. - They also provides the ability to "fork" Git and Hg repos in order to - provide your own extensions. -Wiki - This tool provides a basic wiki with support for comments, attachments, and - notifications. -Tracker - This tool provides an extensible ticketing system for tracking feature - requests, defects, or support requests. -Discussion - This tool provides a forum interface with full email integration as well. - The forum also handles attachments to posts either via the web interface or via email. - -The Context Object ------------------- - -The Pylons "context" object `c` has several properties which are automatically -set for each request: - -project - The current project -app - The current tool application object. -user - The current user - -Allura platform provides the following functions to manage the context object, -if you need to change the context for some situation: - -.. function:: allura.lib.helpers.push_config(obj, **kw) - :noindex: - - Context manager (used with the `with` statement) used to temporarily set the - attributes of a particular object, resetting them to their previous values - at the end of the `with` block. Used most frequently with the context object - `c`:: - - c.project = some_project - with push_config(c, project=other_project): - ... - # code in this block will have c.project == other_project - # code here will have c.project == some_project - -.. function:: allura.lib.helpers.set_context(project_id, mount_point=None, app_config_id=None) - :noindex: - - Set the context object `c` according to the given `project_id` and optionally either a - `mount_point`, an `app_config_id`. `c.project` is set to the corresponding - project object. If the mount_point or app_config_id is - specified, then `c.app` will be set to the corresponding tool application - object. - - -Artifacts ---------- - -We've mentioned artifacts a couple of times now without definition. An artifact, -as used in Allura, is some object that a tool needs to store in the -forge. The platform provides facilities for controlling access to individual -artifacts if that's what the tool designer favors. For instance, the Discussion -tool allows a user to edit or delete their own posts, but not to edit or delete -others (unless the user has the 'moderate' permission on the forum itself). -Some examples of artifacts in the current tools: - -- Discussion: Forum, Thread, Post, Attachment -- Wiki: Page, Comment, Attachment -- Tracker: Ticket, Comment, Attachment -- SCM: Repository, Commit, Patch - -In order to implement your own artifact, you should override at least a few of -the methods of the `allura.model.artifact.Artifact` class:: - - from ming.orm.property import FieldProperty - from allura.model import Artifact - - class NewArtifact(Artifact): - class __mongometa__: - name='my_new_artifact' # collection where this artifact is stored - type_s = 'My Artifact' # 'type' of the artifact used in search results - - # Add your own properties here (beyond those provided by Artifact) - shortname = FieldProperty(str) - - def url(self): - 'Each artifact should have its own URL ' - return self.app.url + self.shortname + '/' - - def index(self): - 'Return the fields you want indexed on this artifact' - result = Artifact.index(self) - result.update(type_s=self.type_s, - name_s=self.shortname, - text=self.shortname) - return result - - def shorthand_id(self): - 'Used in the generation of short links like [my_artifact]' - return self.shortname - -Platform services provided for artifacts ----------------------------------------- - -Whenever you create, modify, or delete an artifact, the platform does a couple of -things for you: - -- The artifact is added to the index and will appear in searches -- A shortlink is generated for the artifact (e.g. [MyWikiPage] or [#151]). This allows you - to reference the artifact from other artifacts. Whenever the commit message - is displayed in the SCM tool, any references to `[#151]` will be - automatically linked to that Ticket's page. - -Shortlinks work only within a project hierarchy (in order to link to some other project's -page, you'll have to use the full URL). Sometimes, a shortlink may need to be -differentiated based on its location in a subproject or in one of many tools of -the same type within a project. In order to do this, shortlinks may be prefixed -by either the tool mount point or a project ID and tool mount point. - -For instance, suppose we have an ticket tracker called 'features' and one called 'bugs'. -They both have many tickets in them. To distinguish, use the tracker mount point -within the reference. For example [features:#3] or [bugs:#3] - -Asynchronous Processing ------------------------ - -Much of the actual functionality of Allura comes from code that runs -*outside* the context of a web request, in the `taskd` server (invoked by -running `paster taskd development.ini`. Asynchronous processing is performed -by two types of functions, *tasks* and *events*, differentiated as follows: - -Task - Tasks are module-level global functions. They are annotated with the `@task` - decorator and are invoked with the `.post` method. For instance, to schedule - a task `foobar` to execute in the `taskd` context, you would write:: - - @task - def foobar(a,b,c=5): ... - - foobar.post(9,1,c=15) - -Event - Events are intended for "fan-out" types of events. Events have a string - name, and are "listened" for by using the `@event_handler` decorator. The - `g.post_event()` helper is provided to run the event handlers for a - particular event in the `taskd` context. Multiple event handlers can be - registered for each event:: - - @event_handler('event_name') - def handler1(topic, *args, **kwargs): ... - - @event_handler('event_name') - def handler2(topic, *args, **kwargs): ... - - g.post_event('event_name', 1,2,3, a=5) - - -Email Integration ------------------ - -The Allura platform provides easy-to-use email integration. Forge email addresses -are of the form -@[.]*..projects.sourceforge.net. -When a message is received on such an email address, the address is parsed and -the sending user is identified (if possible). Based on the parsed address, the -pylons context attributes `c.project` and `c.app` are set, and the application is -queried to determine whether the identified user has authority to send an email -to the given app/topic combination by calling `c.app.has_access(user, topic)`. -If the user has access, the message is decomposed into its component parts (if a -multipart MIME-encoded message) and `c.app.handle_message(topic, message)` is -called for each part with the following components to the `msg` dict: - -headers - The actual headers parsed from the body of the message -message_id - The `Message-ID` header (which should be universally - unique and is - generated by the email client), used for determining which messages are replies - to which other messages -in_reply_to - The `In-Reply-To` header, used for determining which messages are replies to - which other messages -references - The `References` header, used for determining which messages refer to - which other messages -filename - Optional, if the part is an attachment with a filename, this will be populated -content_type - The MIME content_type of the message part -payload - The actual content of the message part - -The Allura platform also provides full support for *sending* email without -worrying about the specifics of SMTP or sendmail handling. In order to send an -email, simply post a task for `allura.tasks.mail_tasks.sendmail` with the -following arguments: - -fromaddr - Return address on the message (usually the topic@tool_name that generated - it) -destinations - List of email addresses and/or :class:`bson.ObjectId` s for - :class:`allura.model.auth.User` objects -text - Markdown-formatted body of the message (If the user has requested html or - combined text+html messages in their preferences, the Markdown will be so - rendered. Otherwise a plain text message will be sent.) -reply_to - Address to which replies should be sent -subject - Subject of the message -message_id - Value to put in the `Message-ID` header (the `_id` field of a - :class:`allura.model.artifact.Message` is suitable for this) -in_reply_to (optional) - Value to put in the `In-Reply-To` header (the `parent_id` field of a - :class:`allura.model.artifact.Message` is suitable for this)