aurora-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From dles...@apache.org
Subject svn commit: r1632473 - in /incubator/aurora/site: publish/documentation/latest/configuration-reference/ publish/documentation/latest/cron-jobs/ publish/documentation/latest/images/ publish/documentation/latest/storage/ source/documentation/latest/ sour...
Date Fri, 17 Oct 2014 04:38:27 GMT
Author: dlester
Date: Fri Oct 17 04:38:27 2014
New Revision: 1632473

URL: http://svn.apache.org/r1632473
Log:
Updates Aurora docs to be in sync with git.

Added:
    incubator/aurora/site/publish/documentation/latest/cron-jobs/
    incubator/aurora/site/publish/documentation/latest/cron-jobs/index.html
    incubator/aurora/site/publish/documentation/latest/images/storage_hierarchy.png   (with
props)
    incubator/aurora/site/publish/documentation/latest/storage/
    incubator/aurora/site/publish/documentation/latest/storage/index.html
    incubator/aurora/site/source/documentation/latest/cron-jobs.md
    incubator/aurora/site/source/documentation/latest/images/storage_hierarchy.png   (with
props)
    incubator/aurora/site/source/documentation/latest/storage.md
Modified:
    incubator/aurora/site/publish/documentation/latest/configuration-reference/index.html
    incubator/aurora/site/source/documentation/latest/configuration-reference.md

Modified: incubator/aurora/site/publish/documentation/latest/configuration-reference/index.html
URL: http://svn.apache.org/viewvc/incubator/aurora/site/publish/documentation/latest/configuration-reference/index.html?rev=1632473&r1=1632472&r2=1632473&view=diff
==============================================================================
--- incubator/aurora/site/publish/documentation/latest/configuration-reference/index.html
(original)
+++ incubator/aurora/site/publish/documentation/latest/configuration-reference/index.html
Fri Oct 17 04:38:27 2014
@@ -527,14 +527,14 @@ resources are allocated.</p>
 <td>Number of instances (sometimes referred to as replicas or shards) of the task to
create. (Default: 1)</td>
 </tr>
 <tr>
-<td><code>cron_schedule</code> <strong>(Present, but not supported
and a no-op)</strong></td>
+<td><code>cron_schedule</code></td>
 <td style="text-align: center">String</td>
-<td>UTC Cron schedule in cron format. May only be used with non-service jobs. Default:
None (not a cron job.)</td>
+<td>Cron schedule in cron format. May only be used with non-service jobs. See <a
href="/documentation/latest/cron-jobs/">Cron Jobs</a> for more information. Default:
None (not a cron job.)</td>
 </tr>
 <tr>
-<td><code>cron_collision_policy</code> <strong>(Present, but not
supported and a no-op)</strong></td>
+<td><code>cron_collision_policy</code></td>
 <td style="text-align: center">String</td>
-<td>Policy to use when a cron job is triggered while a previous run is still active.
KILL<em>EXISTING Kill the previous run, and schedule the new run CANCEL</em>NEW
Let the previous run continue, and cancel the new run. RUN<em>OVERLAP Let the previous
run continue, and schedule the new run. (Default: KILL</em>EXISTING)</td>
+<td>Policy to use when a cron job is triggered while a previous run is still active.
KILL<em>EXISTING Kill the previous run, and schedule the new run CANCEL</em>NEW
Let the previous run continue, and cancel the new run. (Default: KILL_EXISTING)</td>
 </tr>
 <tr>
 <td><code>update_config</code></td>
@@ -552,11 +552,6 @@ resources are allocated.</p>
 <td>If True, restart tasks regardless of success or failure. (Default: False)</td>
 </tr>
 <tr>
-<td><code>daemon</code></td>
-<td style="text-align: center">Boolean</td>
-<td>A DEPRECATED alias for &ldquo;service&rdquo;. (Default: False)</td>
-</tr>
-<tr>
 <td><code>max_task_failures</code></td>
 <td style="text-align: center">Integer</td>
 <td>Maximum number of failures after which the task is considered to have failed (Default:
1) Set to -1 to allow for infinite failures</td>

Added: incubator/aurora/site/publish/documentation/latest/cron-jobs/index.html
URL: http://svn.apache.org/viewvc/incubator/aurora/site/publish/documentation/latest/cron-jobs/index.html?rev=1632473&view=auto
==============================================================================
--- incubator/aurora/site/publish/documentation/latest/cron-jobs/index.html (added)
+++ incubator/aurora/site/publish/documentation/latest/cron-jobs/index.html Fri Oct 17 04:38:27
2014
@@ -0,0 +1,251 @@
+<html>
+    <head>
+        <meta charset="utf-8">
+        <title>Apache Aurora</title>
+		    <meta name="viewport" content="width=device-width, initial-scale=1.0">
+		    <meta name="description" content="">
+		    <meta name="author" content="">
+
+		    <link href="/assets/css/bootstrap.css" rel="stylesheet">
+		    <link href="/assets/css/bootstrap-responsive.min.css" rel="stylesheet">
+		    <link href="/assets/css/main.css" rel="stylesheet">
+				
+		    <!-- JS -->
+		    <script type="text/javascript" src="/assets/js/jquery-1.10.1.min.js"></script>
+		    <script type="text/javascript" src="/assets/js/bootstrap-dropdown.js"></script>
+		
+				<!-- Analytics -->
+				<script type="text/javascript">
+					  var _gaq = _gaq || [];
+					  _gaq.push(['_setAccount', 'UA-45879646-1']);
+					  _gaq.push(['_setDomainName', 'apache.org']);
+					  _gaq.push(['_trackPageview']);
+
+					  (function() {
+					    var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async
= true;
+					    ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www')
+ '.google-analytics.com/ga.js';
+					    var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga,
s);
+					  })();
+				</script>
+	</head>
+    <body>	
+      <div class="navbar navbar-static-top">
+  <div class="navbar-inner">
+    <div class="container">
+	    <a href="/" class="logo"><img src="/assets/img/aurora_logo.png" alt="Apache
Aurora logo" /></a>
+      <ul class="nav">
+				<li><a href="/documentation/latest/">Documentation</a></li>
+        <li><a href="/downloads/">Download</a></li>
+        <li><a href="/community">Community</a></li>
+      </ul>
+    </div>
+  </div>
+</div>
+
+<div class="container">
+<!-- magical breadcrumbs -->
+<ul class="breadcrumb">
+  <li>
+    <div class="dropdown">
+      <a class="dropdown-toggle" data-toggle="dropdown" href="#">Apache Software Foundation
<b class="caret"></b></a>
+      <ul class="dropdown-menu" role="menu">
+        <li><a href="http://www.apache.org">Apache Homepage</a></li>
+        <li><a href="http://www.apache.org/licenses/">Apache License</a></li>
+        <li><a href="http://www.apache.org/foundation/sponsorship.html">Sponsorship</a></li>
 
+        <li><a href="http://www.apache.org/foundation/thanks.html">Thanks</a></li>
+        <li><a href="http://www.apache.org/security/">Security</a></li>
+      </ul>
+    </div>
+  </li>
+  <li><span class="divider">&bull;</span></li>
+  <li><a href="http://incubator.apache.org">Apache Incubator</a></li>
+  <li><span class="divider">&bull;</span></li>
+  <li><a href="http://aurora.incubator.apache.org">Apache Aurora</a></li>
+</ul>
+<!-- /breadcrumb -->
+	
+      <div class="container">
+        <h1 id="cron-jobs">Cron Jobs</h1>
+
+<p>Aurora supports execution of scheduled jobs on a Mesos cluster using cron-style
syntax.</p>
+
+<ul>
+<li><a href="#overview">Overview</a></li>
+<li><a href="#collision-policies">Collision Policies</a>
+
+<ul>
+<li><a href="#kill_existing">KILL_EXISTING</a></li>
+<li><a href="#cancel_new">CANCEL_NEW</a></li>
+</ul></li>
+<li><a href="#failure-recovery">Failure recovery</a></li>
+<li><a href="#interacting-with-cron-jobs-via-the-aurora-cli">Interacting with
cron jobs via the Aurora CLI</a>
+
+<ul>
+<li><a href="#cron-schedule">cron schedule</a></li>
+<li><a href="#cron-deschedule">cron deschedule</a></li>
+<li><a href="#cron-start">cron start</a></li>
+<li><a href="#job-killall-job-restart-job-kill">job killall, job restart, job
kill</a></li>
+</ul></li>
+<li><a href="#technical-note-about-syntax">Technical Note About Syntax</a></li>
+<li><a href="#caveats">Caveats</a>
+
+<ul>
+<li><a href="#failovers">Failovers</a></li>
+<li><a href="#collision-policy-is-best-effort">Collision policy is best-effort</a></li>
+<li><a href="#timezone-configuration">Timezone Configuration</a></li>
+</ul></li>
+</ul>
+
+<h2 id="overview">Overview</h2>
+
+<p>A job is identified as a cron job by the presence of a
+<code>cron_schedule</code> attribute containing a cron-style schedule in the
+<a href="configuration-reference.md#job-objects"><code>Job</code></a>
object. Examples of cron schedules
+include &ldquo;every 5 minutes&rdquo; (<code>*/5 * * * *</code>), &ldquo;Fridays
at 17:00&rdquo; (<code>* 17 * * FRI</code>), and
+&ldquo;the 1st and 15th day of the month at 03:00&rdquo; (<code>0 3 1,15 *</code>).</p>
+
+<p>Example (available in the <a href="/documentation/latest/vagrant/">Vagrant
environment</a>):</p>
+<pre class="highlight text">$ cat /vagrant/examples/job/cron_hello_world.aurora
+# cron_hello_world.aurora
+# A cron job that runs every 5 minutes.
+jobs = [
+  Job(
+    cluster = &#39;devcluster&#39;,
+    role = &#39;www-data&#39;,
+    environment = &#39;test&#39;,
+    name = &#39;cron_hello_world&#39;,
+    cron_schedule = &#39;*/5 * * * *&#39;,
+    task = SimpleTask(
+      &#39;cron_hello_world&#39;,
+      &#39;echo &quot;Hello world from cron, the time is now $(date --rfc-822)&quot;&#39;),
+  ),
+]
+</pre>
+<h2 id="collision-policies">Collision Policies</h2>
+
+<p>The <code>cron_collision_policy</code> field specifies the scheduler&rsquo;s
behavior when a new cron job is
+triggered while an older run hasn&rsquo;t finished. The scheduler has two policies available,
+<a href="#kill_existing">KILL_EXISTING</a> and <a href="#cancel_new">CANCEL_NEW</a>.</p>
+
+<h3 id="kill_existing">KILL_EXISTING</h3>
+
+<p>The default policy - on a collision the old instances are killed and a instances
with the current
+configuration are started.</p>
+
+<h3 id="cancel_new">CANCEL_NEW</h3>
+
+<p>On a collision the new run is cancelled.</p>
+
+<p>Note that the use of this flag is likely a code smell - interrupted cron jobs should
be able
+to recover their progress on a subsequent invocation, otherwise they risk having their work
queue
+grow faster than they can process it.</p>
+
+<h2 id="failure-recovery">Failure recovery</h2>
+
+<p>Unlike with services, which aurora will always re-execute regardless of exit status,
instances of
+cron jobs retry according to the <code>max_task_failures</code> attribute of
the
+<a href="configuration-reference.md#task-objects">Task</a> object. To get &ldquo;run-until-failure&rdquo;
semantics,
+set <code>max_task_failures</code> to <code>-1</code>.</p>
+
+<h2 id="interacting-with-cron-jobs-via-the-aurora-cli">Interacting with cron jobs via
the Aurora CLI</h2>
+
+<p>Most interaction with cron jobs takes place using the <code>cron</code>
subcommand. See <code>aurora2 help cron</code>
+for up-to-date usage instructions.</p>
+
+<h3 id="cron-schedule">cron schedule</h3>
+
+<p>Schedules a new cron job on the Aurora cluster for later runs, or updates an existing
job.</p>
+<pre class="highlight text">$ aurora2 cron schedule devcluster/www-data/test/cron_hello_world
/vagrant/examples/jobs/cron_hello_world.aurora
+</pre>
+<h3 id="cron-deschedule">cron deschedule</h3>
+
+<p>Deschedules a cron job, preventing future runs but allowing current runs to complete.</p>
+<pre class="highlight text">$ aurora2 cron deschedule devcluster/www-data/test/cron_hello_world
+</pre>
+<h3 id="cron-start">cron start</h3>
+
+<p>Start a cron job immediately, outside of its normal cron schedule.</p>
+<pre class="highlight text">$ aurora2 cron start devcluster/www-data/test/cron_hello_world
+</pre>
+<h3 id="job-killall,-job-restart,-job-kill">job killall, job restart, job kill</h3>
+
+<p>Cron jobs create instances running on the cluster that you can interact with like
normal Aurora
+tasks with <code>job kill</code> and <code>job restart</code>.</p>
+
+<h2 id="technical-note-about-syntax">Technical Note About Syntax</h2>
+
+<p><code>cron_schedule</code> uses a restricted subset of BSD crontab syntax.
While the
+execution engine currently uses Quartz, the schedule parsing is custom, a subset of FreeBSD
+<a href="http://www.freebsd.org/cgi/man.cgi?crontab(5)">crontab(5)</a> syntax.
See
+<a href="https://github.com/apache/incubator-aurora/blob/master/src/main/java/org/apache/aurora/scheduler/cron/CrontabEntry.java#L106-L124">the
source</a>
+for details.</p>
+
+<h2 id="caveats">Caveats</h2>
+
+<h3 id="failovers">Failovers</h3>
+
+<p>No failover recovery. Aurora does not record the latest minute it fired
+triggers for across failovers. Therefore it&rsquo;s possible to miss triggers
+on failover. Note that this behavior may change in the future.</p>
+
+<p>It&rsquo;s necessary to sync time between schedulers with something like <code>ntpd</code>.
+Clock skew could cause double or missed triggers in the case of a failover.</p>
+
+<h3 id="collision-policy-is-best-effort">Collision policy is best-effort</h3>
+
+<p>Aurora aims to always have <em>at least one copy</em> of a given instance
running at a time - it&rsquo;s
+an AP system, meaning it chooses Availability and Partition Tolerance at the expense of
+Consistency.</p>
+
+<p>If your collision policy was <code>CANCEL_NEW</code> and a task has
terminated but
+Aurora has not noticed this Aurora will go ahead and create your new
+task.</p>
+
+<p>If your collision policy was <code>KILL_EXISTING</code> and a task was
marked <code>LOST</code>
+but not yet GCed Aurora will go ahead and create your new task without
+attempting to kill the old one (outside the GC interval).</p>
+
+<h3 id="timezone-configuration">Timezone Configuration</h3>
+
+<p>Cron timezone is configured indepdendently of JVM timezone with the <code>-cron_timezone</code>
flag and
+defaults to UTC.</p>
+
+	  </div>
+      <div class="container">
+    <hr>
+    <footer class="footer">
+        <div class="row-fluid">
+            <div class="span2 text-left">
+                <h3>Links</h3>
+                <ul class="unstyled">
+                    <li><a href="/downloads/">Downloads</a></li>
+                    <li><a href="/developers/">Developers</a></li>
                   
+                </ul>
+            </div>
+            <div class="span3 text-left">
+                <h3>Community</h3>
+                <ul class="unstyled">
+                    <li><a href="/community/">Mailing Lists</a></li>
+                    <li><a href="http://issues.apache.org/jira/browse/aurora">Issue
Tracking</a></li>
+                    <li><a href="/docs/howtocontribute/">How To Contribute</a></li>
+                </ul>
+            </div>
+            <div class="span7 text-left">
+            	<h3>Apache Software Foundation</h3>
+
+							<div class="span8">
+                Copyright 2014 <a href="http://www.apache.org/">Apache Software Foundation</a>.
Licensed under the <a href="http://www.apache.org/licenses/">Apache License v2.0</a>.
Apache, Apache Thrift, and the Apache feather logo are trademarks of The Apache Software Foundation.
Currently part of the <a href="http://incubator.apache.org">Apache Incubator</a>.
+							</div>
+							<div class=" pull-right">
+								<a href="http://incubator.apache.org" class="logo"><img src="/assets/img/apache_incubator_logo.png"
alt="Apache Incubator" class="pull-right"/></a>
+							</div>
+            </div>
+
+        </div>
+
+    </footer>
+</div>
+
+	</body>
+</html>
+

Added: incubator/aurora/site/publish/documentation/latest/images/storage_hierarchy.png
URL: http://svn.apache.org/viewvc/incubator/aurora/site/publish/documentation/latest/images/storage_hierarchy.png?rev=1632473&view=auto
==============================================================================
Binary file - no diff available.

Propchange: incubator/aurora/site/publish/documentation/latest/images/storage_hierarchy.png
------------------------------------------------------------------------------
    svn:mime-type = application/octet-stream

Added: incubator/aurora/site/publish/documentation/latest/storage/index.html
URL: http://svn.apache.org/viewvc/incubator/aurora/site/publish/documentation/latest/storage/index.html?rev=1632473&view=auto
==============================================================================
--- incubator/aurora/site/publish/documentation/latest/storage/index.html (added)
+++ incubator/aurora/site/publish/documentation/latest/storage/index.html Fri Oct 17 04:38:27
2014
@@ -0,0 +1,202 @@
+<html>
+    <head>
+        <meta charset="utf-8">
+        <title>Apache Aurora</title>
+		    <meta name="viewport" content="width=device-width, initial-scale=1.0">
+		    <meta name="description" content="">
+		    <meta name="author" content="">
+
+		    <link href="/assets/css/bootstrap.css" rel="stylesheet">
+		    <link href="/assets/css/bootstrap-responsive.min.css" rel="stylesheet">
+		    <link href="/assets/css/main.css" rel="stylesheet">
+				
+		    <!-- JS -->
+		    <script type="text/javascript" src="/assets/js/jquery-1.10.1.min.js"></script>
+		    <script type="text/javascript" src="/assets/js/bootstrap-dropdown.js"></script>
+		
+				<!-- Analytics -->
+				<script type="text/javascript">
+					  var _gaq = _gaq || [];
+					  _gaq.push(['_setAccount', 'UA-45879646-1']);
+					  _gaq.push(['_setDomainName', 'apache.org']);
+					  _gaq.push(['_trackPageview']);
+
+					  (function() {
+					    var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async
= true;
+					    ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www')
+ '.google-analytics.com/ga.js';
+					    var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga,
s);
+					  })();
+				</script>
+	</head>
+    <body>	
+      <div class="navbar navbar-static-top">
+  <div class="navbar-inner">
+    <div class="container">
+	    <a href="/" class="logo"><img src="/assets/img/aurora_logo.png" alt="Apache
Aurora logo" /></a>
+      <ul class="nav">
+				<li><a href="/documentation/latest/">Documentation</a></li>
+        <li><a href="/downloads/">Download</a></li>
+        <li><a href="/community">Community</a></li>
+      </ul>
+    </div>
+  </div>
+</div>
+
+<div class="container">
+<!-- magical breadcrumbs -->
+<ul class="breadcrumb">
+  <li>
+    <div class="dropdown">
+      <a class="dropdown-toggle" data-toggle="dropdown" href="#">Apache Software Foundation
<b class="caret"></b></a>
+      <ul class="dropdown-menu" role="menu">
+        <li><a href="http://www.apache.org">Apache Homepage</a></li>
+        <li><a href="http://www.apache.org/licenses/">Apache License</a></li>
+        <li><a href="http://www.apache.org/foundation/sponsorship.html">Sponsorship</a></li>
 
+        <li><a href="http://www.apache.org/foundation/thanks.html">Thanks</a></li>
+        <li><a href="http://www.apache.org/security/">Security</a></li>
+      </ul>
+    </div>
+  </li>
+  <li><span class="divider">&bull;</span></li>
+  <li><a href="http://incubator.apache.org">Apache Incubator</a></li>
+  <li><span class="divider">&bull;</span></li>
+  <li><a href="http://aurora.incubator.apache.org">Apache Aurora</a></li>
+</ul>
+<!-- /breadcrumb -->
+	
+      <div class="container">
+        <h1 id="aurora-scheduler-storage">Aurora Scheduler Storage</h1>
+
+<ul>
+<li><a href="#overview">Overview</a></li>
+<li><a href="#reads-writes-modifications">Reads, writes, modifications&hellip;</a>
+
+<ul>
+<li><a href="#read-lifecycle">Read lifecycle</a></li>
+<li><a href="#write-lifecycle">Write lifecycle</a></li>
+</ul></li>
+<li><a href="#atomicity-consistency-and-isolation">Atomicity, consistency and
isolation</a></li>
+<li><a href="#population-on-restart">Population on restart</a></li>
+</ul>
+
+<h2 id="overview">Overview</h2>
+
+<p>Aurora scheduler maintains data that need to be persisted to survive failovers and
restarts.
+For example:</p>
+
+<ul>
+<li>Task configurations and scheduled task instances</li>
+<li>Job update configurations and update progress</li>
+<li>Production resource quotas</li>
+<li>Mesos resource offer host attributes</li>
+</ul>
+
+<p>Aurora solves its persistence needs by leveraging the Mesos implementation of a
Paxos replicated
+log <a href="https://ramcloud.stanford.edu/~ongaro/userstudy/paxos.pdf">[1]</a>
+<a href="http://en.wikipedia.org/wiki/State_machine_replication">[2]</a> with
a key-value
+<a href="https://github.com/google/leveldb">LevelDB</a> storage as persistence
media.</p>
+
+<p>Conceptually, it can be represented by the following major components:</p>
+
+<ul>
+<li>Volatile storage: in-memory cache of all available data. Implemented via in-memory
+<a href="http://www.h2database.com/html/main.html">H2 Database</a> and accessed
via
+<a href="http://mybatis.github.io/mybatis-3/">MyBatis</a>.</li>
+<li>Log manager: interface between Aurora storage and Mesos replicated log. The default
schema format
+is <a href="https://github.com/apache/thrift">thrift</a>. Data is stored in serialized
binary form.</li>
+<li>Snapshot manager: all data is periodically persisted in Mesos replicated log in
a single snapshot.
+This helps establishing periodic recovery checkpoints and speeds up volatile storage recovery
on
+restart.</li>
+<li>Backup manager: as a precaution, snapshots are periodically written out into backup
files.
+This solves a disaster recovery problem in case of a complete loss or corruption of Mesos
log files.</li>
+</ul>
+
+<p><img alt="Storage hierarchy" src="../images/storage_hierarchy.png" /></p>
+
+<h2 id="reads,-writes,-modifications...">Reads, writes, modifications&hellip;</h2>
+
+<p>All services in Aurora access data via a set of predefined store interfaces (aka
stores) logically
+grouped by the type of data they serve. Every interface defines a specific set of operations
allowed
+on the data thus abstracting out the storage access and the actual persistence implementation.
The
+latter is especially important in view of a general immutability of persisted data. With
the Mesos
+replicated log as the underlying persistence solution, data can be read and written easily
but not
+modified. All modifications are simulated by saving new versions of modified objects. This
feature
+and general performance considerations justify the existence of the volatile in-memory store.</p>
+
+<h3 id="read-lifecycle">Read lifecycle</h3>
+
+<p>There are two types of reads available in Aurora: consistent and weakly-consistent.
The difference
+is explained <a href="#atomicity-and-isolation">below</a>.</p>
+
+<p>All reads are served from the volatile storage making reads generally cheap storage
operations
+from the performance standpoint. The majority of the volatile stores are represented by the
+in-memory H2 database. This allows for rich schema definitions, queries and relationships
that
+key-value storage is unable to match.</p>
+
+<h3 id="write-lifecycle">Write lifecycle</h3>
+
+<p>Writes are more involved operations since in addition to updating the volatile store
data has to be
+appended to the replicated log. Data is not available for reads until fully ack-ed by both
+replicated log and volatile storage.</p>
+
+<h2 id="atomicity,-consistency-and-isolation">Atomicity, consistency and isolation</h2>
+
+<p>Aurora uses <a href="http://en.wikipedia.org/wiki/Write-ahead_logging">write-ahead
logging</a> to ensure
+consistency between replicated and volatile storage. In Aurora, data is first written into
the
+replicated log and only then updated in the volatile store.</p>
+
+<p>Aurora storage uses read-write locks to serialize data mutations and provide consistent
view of the
+available data. The available <code>Storage</code> interface exposes 3 major
types of operations:
+* <code>consistentRead</code> - access is locked using reader&rsquo;s lock
and provides consistent view on read
+* <code>weaklyConsistentRead</code> - access is lock-less. Delivers best contention
performance but may result
+in stale reads
+* <code>write</code> - access is fully serialized by using writer&rsquo;s
lock. Operation success requires both
+volatile and replicated writes to succeed.</p>
+
+<p>The consistency of the volatile store is enforced via H2 transactional isolation.</p>
+
+<h2 id="population-on-restart">Population on restart</h2>
+
+<p>Any time a scheduler restarts, it restores its volatile state from the most recent
position recorded
+in the replicated log by restoring the snapshot and replaying individual log entries on top
to fully
+recover the state up to the last write.</p>
+
+	  </div>
+      <div class="container">
+    <hr>
+    <footer class="footer">
+        <div class="row-fluid">
+            <div class="span2 text-left">
+                <h3>Links</h3>
+                <ul class="unstyled">
+                    <li><a href="/downloads/">Downloads</a></li>
+                    <li><a href="/developers/">Developers</a></li>
                   
+                </ul>
+            </div>
+            <div class="span3 text-left">
+                <h3>Community</h3>
+                <ul class="unstyled">
+                    <li><a href="/community/">Mailing Lists</a></li>
+                    <li><a href="http://issues.apache.org/jira/browse/aurora">Issue
Tracking</a></li>
+                    <li><a href="/docs/howtocontribute/">How To Contribute</a></li>
+                </ul>
+            </div>
+            <div class="span7 text-left">
+            	<h3>Apache Software Foundation</h3>
+
+							<div class="span8">
+                Copyright 2014 <a href="http://www.apache.org/">Apache Software Foundation</a>.
Licensed under the <a href="http://www.apache.org/licenses/">Apache License v2.0</a>.
Apache, Apache Thrift, and the Apache feather logo are trademarks of The Apache Software Foundation.
Currently part of the <a href="http://incubator.apache.org">Apache Incubator</a>.
+							</div>
+							<div class=" pull-right">
+								<a href="http://incubator.apache.org" class="logo"><img src="/assets/img/apache_incubator_logo.png"
alt="Apache Incubator" class="pull-right"/></a>
+							</div>
+            </div>
+
+        </div>
+
+    </footer>
+</div>
+
+	</body>
+</html>
+

Modified: incubator/aurora/site/source/documentation/latest/configuration-reference.md
URL: http://svn.apache.org/viewvc/incubator/aurora/site/source/documentation/latest/configuration-reference.md?rev=1632473&r1=1632472&r2=1632473&view=diff
==============================================================================
--- incubator/aurora/site/source/documentation/latest/configuration-reference.md (original)
+++ incubator/aurora/site/source/documentation/latest/configuration-reference.md Fri Oct 17
04:38:27 2014
@@ -313,12 +313,11 @@ Job Schema
    ```environment``` | String | Job environment, default ```devel```. Must be one of ```prod```,
```devel```, ```test``` or ```staging<number>```.
   ```contact``` | String | Best email address to reach the owner of the job. For production
jobs, this is usually a team mailing list.
   ```instances```| Integer | Number of instances (sometimes referred to as replicas or shards)
of the task to create. (Default: 1)
-   ```cron_schedule``` **(Present, but not supported and a no-op)** | String | UTC Cron schedule
in cron format. May only be used with non-service jobs. Default: None (not a cron job.)
-  ```cron_collision_policy``` **(Present, but not supported and a no-op)** | String | Policy
to use when a cron job is triggered while a previous run is still active. KILL_EXISTING Kill
the previous run, and schedule the new run CANCEL_NEW Let the previous run continue, and cancel
the new run. RUN_OVERLAP Let the previous run continue, and schedule the new run. (Default:
KILL_EXISTING)
+   ```cron_schedule``` | String | Cron schedule in cron format. May only be used with non-service
jobs. See [Cron Jobs](/documentation/latest/cron-jobs/) for more information. Default: None
(not a cron job.)
+  ```cron_collision_policy``` | String | Policy to use when a cron job is triggered while
a previous run is still active. KILL_EXISTING Kill the previous run, and schedule the new
run CANCEL_NEW Let the previous run continue, and cancel the new run. (Default: KILL_EXISTING)
   ```update_config``` | ```update_config``` object | Parameters for controlling the rate
and policy of rolling updates.
   ```constraints``` | dict | Scheduling constraints for the tasks. See the section on the
[constraint specification language](#Specifying-Scheduling-Constraints)
   ```service``` | Boolean | If True, restart tasks regardless of success or failure. (Default:
False)
-  ```daemon``` | Boolean | A DEPRECATED alias for "service". (Default: False)
   ```max_task_failures``` | Integer | Maximum number of failures after which the task is
considered to have failed (Default: 1) Set to -1 to allow for infinite failures
   ```priority``` | Integer | Preemption priority to give the task (Default 0). Tasks with
higher priorities may preempt tasks at lower priorities.
   ```production``` | Boolean |  Whether or not this is a production task backed by quota
(Default: False). Production jobs may preempt any non-production job, and may only be preempted
by production jobs in the same role and of higher priority. To run jobs at this level, the
job role must have the appropriate quota.

Added: incubator/aurora/site/source/documentation/latest/cron-jobs.md
URL: http://svn.apache.org/viewvc/incubator/aurora/site/source/documentation/latest/cron-jobs.md?rev=1632473&view=auto
==============================================================================
--- incubator/aurora/site/source/documentation/latest/cron-jobs.md (added)
+++ incubator/aurora/site/source/documentation/latest/cron-jobs.md Fri Oct 17 04:38:27 2014
@@ -0,0 +1,130 @@
+# Cron Jobs
+
+Aurora supports execution of scheduled jobs on a Mesos cluster using cron-style syntax.
+
+- [Overview](#overview)
+- [Collision Policies](#collision-policies)
+	- [KILL_EXISTING](#kill_existing)
+	- [CANCEL_NEW](#cancel_new)
+- [Failure recovery](#failure-recovery)
+- [Interacting with cron jobs via the Aurora CLI](#interacting-with-cron-jobs-via-the-aurora-cli)
+	- [cron schedule](#cron-schedule)
+	- [cron deschedule](#cron-deschedule)
+	- [cron start](#cron-start)
+	- [job killall, job restart, job kill](#job-killall-job-restart-job-kill)
+- [Technical Note About Syntax](#technical-note-about-syntax)
+- [Caveats](#caveats)
+	- [Failovers](#failovers)
+	- [Collision policy is best-effort](#collision-policy-is-best-effort)
+	- [Timezone Configuration](#timezone-configuration)
+
+## Overview
+
+A job is identified as a cron job by the presence of a
+`cron_schedule` attribute containing a cron-style schedule in the
+[`Job`](configuration-reference.md#job-objects) object. Examples of cron schedules
+include "every 5 minutes" (`*/5 * * * *`), "Fridays at 17:00" (`* 17 * * FRI`), and
+"the 1st and 15th day of the month at 03:00" (`0 3 1,15 *`).
+
+Example (available in the [Vagrant environment](/documentation/latest/vagrant/)):
+
+    $ cat /vagrant/examples/job/cron_hello_world.aurora
+    # cron_hello_world.aurora
+    # A cron job that runs every 5 minutes.
+    jobs = [
+      Job(
+        cluster = 'devcluster',
+        role = 'www-data',
+        environment = 'test',
+        name = 'cron_hello_world',
+        cron_schedule = '*/5 * * * *',
+        task = SimpleTask(
+          'cron_hello_world',
+          'echo "Hello world from cron, the time is now $(date --rfc-822)"'),
+      ),
+    ]
+
+## Collision Policies
+
+The `cron_collision_policy` field specifies the scheduler's behavior when a new cron job
is
+triggered while an older run hasn't finished. The scheduler has two policies available,
+[KILL_EXISTING](#kill_existing) and [CANCEL_NEW](#cancel_new).
+
+### KILL_EXISTING
+
+The default policy - on a collision the old instances are killed and a instances with the
current
+configuration are started.
+
+### CANCEL_NEW
+
+On a collision the new run is cancelled.
+
+Note that the use of this flag is likely a code smell - interrupted cron jobs should be able
+to recover their progress on a subsequent invocation, otherwise they risk having their work
queue
+grow faster than they can process it.
+
+## Failure recovery
+
+Unlike with services, which aurora will always re-execute regardless of exit status, instances
of
+cron jobs retry according to the `max_task_failures` attribute of the
+[Task](configuration-reference.md#task-objects) object. To get "run-until-failure" semantics,
+set `max_task_failures` to `-1`.
+
+## Interacting with cron jobs via the Aurora CLI
+
+Most interaction with cron jobs takes place using the `cron` subcommand. See `aurora2 help
cron`
+for up-to-date usage instructions.
+
+### cron schedule
+Schedules a new cron job on the Aurora cluster for later runs, or updates an existing job.
+
+    $ aurora2 cron schedule devcluster/www-data/test/cron_hello_world /vagrant/examples/jobs/cron_hello_world.aurora
+
+### cron deschedule
+Deschedules a cron job, preventing future runs but allowing current runs to complete.
+
+    $ aurora2 cron deschedule devcluster/www-data/test/cron_hello_world
+
+### cron start
+Start a cron job immediately, outside of its normal cron schedule.
+
+    $ aurora2 cron start devcluster/www-data/test/cron_hello_world
+
+### job killall, job restart, job kill
+Cron jobs create instances running on the cluster that you can interact with like normal
Aurora
+tasks with `job kill` and `job restart`.
+
+## Technical Note About Syntax
+
+`cron_schedule` uses a restricted subset of BSD crontab syntax. While the
+execution engine currently uses Quartz, the schedule parsing is custom, a subset of FreeBSD
+[crontab(5)](http://www.freebsd.org/cgi/man.cgi?crontab(5)) syntax. See
+[the source](https://github.com/apache/incubator-aurora/blob/master/src/main/java/org/apache/aurora/scheduler/cron/CrontabEntry.java#L106-L124)
+for details.
+
+## Caveats
+
+### Failovers
+No failover recovery. Aurora does not record the latest minute it fired
+triggers for across failovers. Therefore it's possible to miss triggers
+on failover. Note that this behavior may change in the future.
+
+It's necessary to sync time between schedulers with something like `ntpd`.
+Clock skew could cause double or missed triggers in the case of a failover.
+
+### Collision policy is best-effort
+Aurora aims to always have *at least one copy* of a given instance running at a time - it's
+an AP system, meaning it chooses Availability and Partition Tolerance at the expense of
+Consistency.
+
+If your collision policy was `CANCEL_NEW` and a task has terminated but
+Aurora has not noticed this Aurora will go ahead and create your new
+task.
+
+If your collision policy was `KILL_EXISTING` and a task was marked `LOST`
+but not yet GCed Aurora will go ahead and create your new task without
+attempting to kill the old one (outside the GC interval).
+
+### Timezone Configuration
+Cron timezone is configured indepdendently of JVM timezone with the `-cron_timezone` flag
and
+defaults to UTC.

Added: incubator/aurora/site/source/documentation/latest/images/storage_hierarchy.png
URL: http://svn.apache.org/viewvc/incubator/aurora/site/source/documentation/latest/images/storage_hierarchy.png?rev=1632473&view=auto
==============================================================================
Binary file - no diff available.

Propchange: incubator/aurora/site/source/documentation/latest/images/storage_hierarchy.png
------------------------------------------------------------------------------
    svn:mime-type = application/octet-stream

Added: incubator/aurora/site/source/documentation/latest/storage.md
URL: http://svn.apache.org/viewvc/incubator/aurora/site/source/documentation/latest/storage.md?rev=1632473&view=auto
==============================================================================
--- incubator/aurora/site/source/documentation/latest/storage.md (added)
+++ incubator/aurora/site/source/documentation/latest/storage.md Fri Oct 17 04:38:27 2014
@@ -0,0 +1,87 @@
+#Aurora Scheduler Storage
+
+- [Overview](#overview)
+- [Reads, writes, modifications...](#reads-writes-modifications)
+  - [Read lifecycle](#read-lifecycle)
+  - [Write lifecycle](#write-lifecycle)
+- [Atomicity, consistency and isolation](#atomicity-consistency-and-isolation)
+- [Population on restart](#population-on-restart)
+
+## Overview
+
+Aurora scheduler maintains data that need to be persisted to survive failovers and restarts.
+For example:
+
+* Task configurations and scheduled task instances
+* Job update configurations and update progress
+* Production resource quotas
+* Mesos resource offer host attributes
+
+Aurora solves its persistence needs by leveraging the Mesos implementation of a Paxos replicated
+log [[1]](https://ramcloud.stanford.edu/~ongaro/userstudy/paxos.pdf)
+[[2]](http://en.wikipedia.org/wiki/State_machine_replication) with a key-value
+[LevelDB](https://github.com/google/leveldb) storage as persistence media.
+
+Conceptually, it can be represented by the following major components:
+
+* Volatile storage: in-memory cache of all available data. Implemented via in-memory
+[H2 Database](http://www.h2database.com/html/main.html) and accessed via
+[MyBatis](http://mybatis.github.io/mybatis-3/).
+* Log manager: interface between Aurora storage and Mesos replicated log. The default schema
format
+is [thrift](https://github.com/apache/thrift). Data is stored in serialized binary form.
+* Snapshot manager: all data is periodically persisted in Mesos replicated log in a single
snapshot.
+This helps establishing periodic recovery checkpoints and speeds up volatile storage recovery
on
+restart.
+* Backup manager: as a precaution, snapshots are periodically written out into backup files.
+This solves a disaster recovery problem in case of a complete loss or corruption of Mesos
log files.
+
+![Storage hierarchy](images/storage_hierarchy.png)
+
+## Reads, writes, modifications...
+
+All services in Aurora access data via a set of predefined store interfaces (aka stores)
logically
+grouped by the type of data they serve. Every interface defines a specific set of operations
allowed
+on the data thus abstracting out the storage access and the actual persistence implementation.
The
+latter is especially important in view of a general immutability of persisted data. With
the Mesos
+replicated log as the underlying persistence solution, data can be read and written easily
but not
+modified. All modifications are simulated by saving new versions of modified objects. This
feature
+and general performance considerations justify the existence of the volatile in-memory store.
+
+### Read lifecycle
+
+There are two types of reads available in Aurora: consistent and weakly-consistent. The difference
+is explained [below](#atomicity-and-isolation).
+
+All reads are served from the volatile storage making reads generally cheap storage operations
+from the performance standpoint. The majority of the volatile stores are represented by the
+in-memory H2 database. This allows for rich schema definitions, queries and relationships
that
+key-value storage is unable to match.
+
+### Write lifecycle
+
+Writes are more involved operations since in addition to updating the volatile store data
has to be
+appended to the replicated log. Data is not available for reads until fully ack-ed by both
+replicated log and volatile storage.
+
+## Atomicity, consistency and isolation
+
+Aurora uses [write-ahead logging](http://en.wikipedia.org/wiki/Write-ahead_logging) to ensure
+consistency between replicated and volatile storage. In Aurora, data is first written into
the
+replicated log and only then updated in the volatile store.
+
+Aurora storage uses read-write locks to serialize data mutations and provide consistent view
of the
+available data. The available `Storage` interface exposes 3 major types of operations:
+* `consistentRead` - access is locked using reader's lock and provides consistent view on
read
+* `weaklyConsistentRead` - access is lock-less. Delivers best contention performance but
may result
+in stale reads
+* `write` - access is fully serialized by using writer's lock. Operation success requires
both
+volatile and replicated writes to succeed.
+
+The consistency of the volatile store is enforced via H2 transactional isolation.
+
+## Population on restart
+
+Any time a scheduler restarts, it restores its volatile state from the most recent position
recorded
+in the replicated log by restoring the snapshot and replaying individual log entries on top
to fully
+recover the state up to the last write.
+



Mime
View raw message