fluo-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From mwa...@apache.org
Subject [incubator-fluo-website] branch gh-pages updated: Created Fluo 1.1.0 release notes
Date Tue, 13 Jun 2017 16:44:04 GMT
This is an automated email from the ASF dual-hosted git repository.

mwalch pushed a commit to branch gh-pages
in repository https://gitbox.apache.org/repos/asf/incubator-fluo-website.git


The following commit(s) were added to refs/heads/gh-pages by this push:
     new 05e3c67  Created Fluo 1.1.0 release notes
05e3c67 is described below

commit 05e3c6797b5115c0d2e65c520f0c432c78529c4a
Author: Keith Turner <keith@deenlo.com>
AuthorDate: Tue Jun 13 12:44:03 2017 -0400

    Created Fluo 1.1.0 release notes
---
 _config.yml                                        |   2 +-
 _posts/release/2017-06-12-fluo-1.1.0-incubating.md | 182 +++++++++++++++++++++
 2 files changed, 183 insertions(+), 1 deletion(-)

diff --git a/_config.yml b/_config.yml
index 578863d..ff2dfc3 100644
--- a/_config.yml
+++ b/_config.yml
@@ -42,7 +42,7 @@ num_home_posts: 5
 color: default
 
 # Fluo specific settings
-latest_fluo_release: "1.0.0-incubating"
+latest_fluo_release: "1.1.0-incubating"
 latest_recipes_release: "1.0.0-incubating"
 
 # Sets links to external API
diff --git a/_posts/release/2017-06-12-fluo-1.1.0-incubating.md b/_posts/release/2017-06-12-fluo-1.1.0-incubating.md
new file mode 100644
index 0000000..624e5c6
--- /dev/null
+++ b/_posts/release/2017-06-12-fluo-1.1.0-incubating.md
@@ -0,0 +1,182 @@
+---
+title: Apache Fluo 1.1.0-incubating released
+date: 2017-06-12 15:00:00 +0000
+version: fluo-1.1.0-incubating
+---
+
+Below are resources for this release:
+
+ * Download a release tarball and verify by these [procedures] using these [KEYS]
+ 
+   | [fluo-1.1.0-incubating-bin.tar.gz][bin-release]            | [ASC][bin-asc] [MD5][bin-md5]
[SHA][bin-sha] |
+   | [fluo-1.1.0-incubating-source-release.tar.gz][src-release] | [ASC][src-asc] [MD5][src-md5]
[SHA][src-sha] |
+ * View the [documentation][docs] for this release.
+ * Read the [Javadocs][javadocs].
+ 
+Apache Fluo follows [semver](http://semver.org/) for its API . The API consists
+of everything under the `org.apache.fluo.api` package. Code outside of this
+package can change at any time. If your project is using Fluo code that falls
+outside of the API, then consider [initiating a discussion](/getinvolved/)
+about adding it to the API.
+
+
+## Significant changes
+
+The major changes in 1.1.0 are highlighted here, for the complete list of changes, see the
[1.1.0
+Milestone] on Github.
+
+### New API for configuring observers.
+
+The 1.0.0 API for providing Observers required configuring an Observer class for each observed
+column.  This API was cumbersome to use and made using lambdas impossible.  For [#816] a
better API
+was introduced.   The new API only requires configuring a single class that provides all
Observers.
+This single class can register lambdas to observe a column.  This new API makes writing Fluo
+applications faster and easier.  Below is an example of using the new API to register two
observers
+that compute the number of URLs that reference a document.
+
+```java
+public class AppObserverProvider implements ObserverProvider {
+
+  private static final Column DOC_CURR_COL = new Column("doc", "curr");
+  private static final Column DOC_NEW_COL = new Column("doc", "new");
+  private static final Column DOC_URL_CHANGE = new Column("doc", "urlChange");
+  private static final Column DOC_REF_COUNT_COL = new Column("doc", "refCount");
+
+  // Each Fluo worker will call this method to create the observers it needs.
+  @Override
+  public void provide(Registry registry, Context ctx) {
+    // This could be used to pass application specific configuration to observers. Its not
used in
+    // this example.
+    SimpleConfiguration appConfig = ctx.getAppConfiguration();
+
+    // Register an observer that processes new content for a document.
+    registry.forColumn(DOC_NEW_COL, STRONG).useObserver(new ContentObserver());
+
+    // Register a lambda that processes notifications for the column DOC_URL_CHANGE.
+    registry.forColumn(DOC_URL_CHANGE, WEAK).useStrObserver((tx, myUrl, col) -> {
+
+      // Compute the number of URLs that refer to this URL.
+      CellScanner refScanner = tx.scanner().over(Span.exact(myUrl, new Column("ref"))).build();
+      int numRefs = Iterables.size(refScanner);
+
+      // Do something interesting with count.  This is not interesting, but keeps the example
short.
+      tx.set(myUrl, DOC_REF_COUNT_COL, numRefs + "");
+    });
+  }
+
+  /**
+   * Compute the change in a documents URLs and propagate those to other documents.
+   */
+  private static class ContentObserver implements StringObserver {
+    @Override
+    public void process(TransactionBase tx, String myUrl, Column col) throws Exception {
+
+      // Retrieve the new and current document content.
+      Map<Column, String> colVals = tx.gets(myUrl, DOC_CURR_COL, DOC_NEW_COL);
+
+      String newContent = colVals.getOrDefault(DOC_NEW_COL, "");
+      String currContent = colVals.getOrDefault(DOC_CURR_COL, "");
+
+      // Extract the URLs in the new and current document content.
+      Set<String> newUrls = extractUrls(newContent);
+      Set<String> currUrls = extractUrls(currContent);
+
+      // For URLs that only exist in new content, update the document they reference.
+      Sets.difference(newUrls, currUrls).forEach(url -> {
+        tx.set(url, new Column("ref", myUrl), "");
+        tx.setWeakNotification(url, DOC_URL_CHANGE);
+      });
+
+      // For URLs that are no longer present, update the document they reference.
+      Sets.difference(currUrls, newUrls).forEach(url -> {
+        tx.delete(url, new Column("ref", myUrl));
+        tx.setWeakNotification(url, DOC_URL_CHANGE);
+      });
+
+      // Update the current document content.
+      tx.set(myUrl, DOC_CURR_COL, newContent);
+      tx.delete(myUrl, DOC_NEW_COL);
+    }
+
+    private Set<String> extractUrls(String newContent) {
+      // TODO implement extracting URLs from content
+      return null;
+    }
+  }
+}
+```
+
+Before initializing a Fluo App, the ObserverProvider above would be added to configuration
as follows.
+
+```java
+FluoConfiguration fluoConf = ...
+fluoConf.setObserverProvider(AppObserverProvider.class);
+
+// initialize Fluo app using fluoConf
+```
+
+The old API is still present but has been deprecated and may be removed in the future.
+
+### Improved Fluo scalability
+
+In the previous release each worker scanned the entire table looking for notifications that
hashed
+to it.  This strategy for finding notifications is O(N*W) where  N is the number of notification
and
+W is the number of workers.
+
+For [#500] a different strategy was used to find notifications.  Workers divide themselves
into
+groups and each group scans a subset of the table for notifications.  Every worker in a group
scans
+the groups entire subset of a table  looking for notifications that hash to it. The new strategy
results
+in O(N*G) work where N is the number of notifications and G is the group size.  By default
the group
+size is 7, but this is configurable.  The group size may need to be increased if portion
of a table
+is popular and assigned to one group that can not processes it.
+
+To compare the old and new ways assume we have 10<sup>9</sup> notifications and
100 workers.  The
+old method would have scanned 10<sup>11</sup> entries to to find all notifications.
 Assuming a group
+size of 7, the new strategy scans 7 * 10<sup>9</sup> entries to find all notifications.
 A
+nice feature of the new strategy is that the amount of scanning is independent of the number
of workers.
+For the old strategy if the number of workers increases by factor of 10, then the amount
scanning
+will increase by a factor of 10.  So for 1,000 workers the old strategy would scan
+10<sup>12</sup> entries to find all notifications.  The new strategy will still
only scan 7 *
+10<sup>9</sup> entries with 1,000 workers.
+
+### Improved Bytes
+
+Fluo's API has an immutable wrapper for a byte array.  Multiple improvements were made to
this byte
+wrapper.
+
+  * `startsWith(Bytes)` and `endsWith(Bytes)` methods were added in [#823]
+  * A `copyTo(byte[])` method was added for [#827]
+  * Internal performance improvements were made in [#826] and [#799]
+
+### Improved Spark integration
+
+For [#813] improvements were made that allow easy passing of FluoConfiguration objects to
remote Spark
+processes.
+
+## Testing
+
+Long runs of [Stresso][stress test] and [Webindex][webindex test] were successfully completed
on EC2 using multiple nodes.
+ 
+[procedures]: https://www.apache.org/info/verification
+[KEYS]: https://www.apache.org/dist/incubator/fluo/KEYS
+[bin-release]: https://www.apache.org/dyn/closer.lua/incubator/fluo/fluo/1.1.0-incubating/fluo-1.1.0-incubating-bin.tar.gz
+[bin-asc]: https://www.apache.org/dist/incubator/fluo/fluo/1.1.0-incubating/fluo-1.1.0-incubating-bin.tar.gz.asc
+[bin-md5]: https://www.apache.org/dist/incubator/fluo/fluo/1.1.0-incubating/fluo-1.1.0-incubating-bin.tar.gz.md5
+[bin-sha]: https://www.apache.org/dist/incubator/fluo/fluo/1.1.0-incubating/fluo-1.1.0-incubating-bin.tar.gz.sha
+[src-release]: https://www.apache.org/dyn/closer.lua/incubator/fluo/fluo/1.1.0-incubating/fluo-1.1.0-incubating-source-release.tar.gz
+[src-asc]: https://www.apache.org/dist/incubator/fluo/fluo/1.1.0-incubating/fluo-1.1.0-incubating-source-release.tar.gz.asc
+[src-md5]: https://www.apache.org/dist/incubator/fluo/fluo/1.1.0-incubating/fluo-1.1.0-incubating-source-release.tar.gz.md5
+[src-sha]: https://www.apache.org/dist/incubator/fluo/fluo/1.1.0-incubating/fluo-1.1.0-incubating-source-release.tar.gz.sha
+[javadocs]: {{ site.fluo_api_base }}/1.1.0-incubating/
+[docs]: /docs/fluo/1.1.0-incubating/
+[semver]: http://semver.org/
+[#500]: https://github.com/apache/incubator-fluo/issues/500
+[#799]: https://github.com/apache/incubator-fluo/issues/799
+[#813]: https://github.com/apache/incubator-fluo/issues/813
+[#816]: https://github.com/apache/incubator-fluo/issues/816
+[#823]: https://github.com/apache/incubator-fluo/issues/823
+[#826]: https://github.com/apache/incubator-fluo/issues/826
+[#827]: https://github.com/apache/incubator-fluo/issues/827
+[stress test]: https://lists.apache.org/thread.html/2381a0b315e8e81871b6caf41e2f1462aa513b9cb24d3ddd92724674@%3Cdev.fluo.apache.org%3E
+[webindex test]: https://twitter.com/ApacheFluo/status/870045298424512512 
+[1.1.0 Milestone]: https://github.com/apache/incubator-fluo/milestone/5?closed=1

-- 
To stop receiving notification emails like this one, please contact
['"commits@fluo.apache.org" <commits@fluo.apache.org>'].

Mime
View raw message