zeppelin-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From zjf...@apache.org
Subject zeppelin git commit: [Zeppelin 3388] Refactor documentation for ZeppelinContext
Date Wed, 02 May 2018 07:03:19 GMT
Repository: zeppelin
Updated Branches:
  refs/heads/master e503623de -> 0c3260e91


[Zeppelin 3388] Refactor documentation for ZeppelinContext

### What is this PR for?
This PR is a refactoring of the `ZepplinContext` documentation.

The description of `ZepplinContext` is now almost entirely contained within the _Spark_ interpreter's
documentation ([spark.md](https://github.com/apache/zeppelin/blob/master/docs/interpreter/spark.md)).
But `ZepplinContext` has many generic features that are available to all interpreters, and
it is important for `ZepplinContext` to have a more visible and independent presence in the
Zeppelin documentation.

### What type of PR is it?
[Documentation]

### Todos

### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-3388

### How should this be tested?
Only documentation.

### Screenshots (if appropriate)

### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No (This _is_ documentation)

Author: Sanjay Dasgupta <sanjay.dasgupta@gmail.com>

Closes #2953 from sanjaydasgupta/zeppelin-3388 and squashes the following commits:

bb8aa0c [Sanjay Dasgupta] Changes to address Felix Cheung's review https://github.com/apache/zeppelin/pull/2953#pullrequestreview-116196552
7360b50 [Sanjay Dasgupta] Correcting deleted hyperlink urls
64e4d7e [Sanjay Dasgupta] Initial version of files for 3388


Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo
Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/0c3260e9
Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/0c3260e9
Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/0c3260e9

Branch: refs/heads/master
Commit: 0c3260e91f4c1ec58e67856d466ea9cba89d0085
Parents: e503623
Author: Sanjay Dasgupta <sanjay.dasgupta@gmail.com>
Authored: Mon Apr 30 16:11:26 2018 +0530
Committer: Jeff Zhang <zjffdu@apache.org>
Committed: Wed May 2 15:03:07 2018 +0800

----------------------------------------------------------------------
 docs/_includes/themes/zeppelin/_navigation.html |   2 +
 docs/index.md                                   |   2 +
 docs/interpreter/shell.md                       |   4 +-
 docs/interpreter/spark.md                       | 154 +-----------
 docs/usage/other_features/zeppelin_context.md   | 233 +++++++++++++++++++
 5 files changed, 241 insertions(+), 154 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/zeppelin/blob/0c3260e9/docs/_includes/themes/zeppelin/_navigation.html
----------------------------------------------------------------------
diff --git a/docs/_includes/themes/zeppelin/_navigation.html b/docs/_includes/themes/zeppelin/_navigation.html
index 071e754..99e0140 100644
--- a/docs/_includes/themes/zeppelin/_navigation.html
+++ b/docs/_includes/themes/zeppelin/_navigation.html
@@ -63,6 +63,7 @@
                 <li><a href="{{BASE_PATH}}/usage/other_features/customizing_homepage.html">Customizing
Zeppelin Homepage</a></li>
                 <li><a href="{{BASE_PATH}}/usage/other_features/notebook_actions.html">Notebook
Actions</a></li>
                 <li><a href="{{BASE_PATH}}/usage/other_features/cron_scheduler.html">Cron
Scheduler</a></li>
+                <li><a href="{{BASE_PATH}}/usage/other_features/zeppelin_context.html">Zeppelin
Context</a></li>
                 <li role="separator" class="divider"></li>
                 <li class="title"><span>REST API</span></li>
                 <li><a href="{{BASE_PATH}}/usage/rest_api/interpreter.html">Interpreter
API</a></li>
@@ -182,3 +183,4 @@
         </nav><!--/.navbar-collapse -->
       </div>
     </div>
+

http://git-wip-us.apache.org/repos/asf/zeppelin/blob/0c3260e9/docs/index.md
----------------------------------------------------------------------
diff --git a/docs/index.md b/docs/index.md
index f00571e..788a979 100644
--- a/docs/index.md
+++ b/docs/index.md
@@ -74,6 +74,7 @@ limitations under the License.
   * [Personalized Mode](./usage/other_features/personalized_mode.html) 
   * [Customizing Zeppelin Homepage](./usage/other_features/customizing_homepage.html) with
one of your notebooks
   * [Notebook actions](./usage/other_features/notebook_actions.html)
+  * [Zeppelin-Context](./usage/other_features/zeppelin_context.html)
 * REST API: available REST API list in Apache Zeppelin
   * [Interpreter API](./usage/rest_api/interpreter.html)
   * [Zeppelin Server API](./usage/rest_api/zeppelin_server.html)
@@ -160,3 +161,4 @@ limitations under the License.
   * [Mailing List](https://zeppelin.apache.org/community.html)
   * [Apache Zeppelin Wiki](https://cwiki.apache.org/confluence/display/ZEPPELIN/Zeppelin+Home)
   * [Stackoverflow Questions about Zeppelin (tag: `apache-zeppelin`)](http://stackoverflow.com/questions/tagged/apache-zeppelin)
+

http://git-wip-us.apache.org/repos/asf/zeppelin/blob/0c3260e9/docs/interpreter/shell.md
----------------------------------------------------------------------
diff --git a/docs/interpreter/shell.md b/docs/interpreter/shell.md
index 25349aa..9ab4036 100644
--- a/docs/interpreter/shell.md
+++ b/docs/interpreter/shell.md
@@ -107,5 +107,5 @@ val members = spark.read.parquet(z.get("dataFileName"))
 
 Object interpolation is disabled by default, and can be enabled (for the Shell interpreter)
by 
 setting the value of the property `zeppelin.shell.interpolation` to `true` (see _Configuration_
above). 
-More details of this feature can be found in the Spark interpreter documentation under 
-[Object Interpolation](spark.html#object-interpolation)
+More details of this feature can be found in [Zeppelin-Context](../usage/other_features/zeppelin_context.html)
+

http://git-wip-us.apache.org/repos/asf/zeppelin/blob/0c3260e9/docs/interpreter/spark.md
----------------------------------------------------------------------
diff --git a/docs/interpreter/spark.md b/docs/interpreter/spark.md
index 3999e3a..85b2981 100644
--- a/docs/interpreter/spark.md
+++ b/docs/interpreter/spark.md
@@ -323,158 +323,7 @@ z.load("groupId:artifactId:version").local()
 
 ## ZeppelinContext
 Zeppelin automatically injects `ZeppelinContext` as variable `z` in your Scala/Python environment.
`ZeppelinContext` provides some additional functions and utilities.
-
-### Exploring Spark DataFrames
-`ZeppelinContext` provides a `show` method, which, using Zeppelin's `table` feature, can
be used to nicely display a Spark DataFrame:
-
-```
-df = spark.read.csv('/path/to/csv')
-z.show(df)
-```
-
-### Object Exchange
-`ZeppelinContext` extends map and it's shared between Scala and Python environment.
-So you can put some objects from Scala and read it from Python, vice versa.
-
-<div class="codetabs">
-  <div data-lang="scala" markdown="1">
-
-{% highlight scala %}
-// Put object from scala
-%spark
-val myObject = ...
-z.put("objName", myObject)
-
-// Exchanging data frames
-myScalaDataFrame = ...
-z.put("myScalaDataFrame", myScalaDataFrame)
-
-val myPythonDataFrame = z.get("myPythonDataFrame").asInstanceOf[DataFrame]
-{% endhighlight %}
-
-  </div>
-  <div data-lang="python" markdown="1">
-
-{% highlight python %}
-# Get object from python
-%spark.pyspark
-myObject = z.get("objName")
-
-# Exchanging data frames
-myPythonDataFrame = ...
-z.put("myPythonDataFrame", postsDf._jdf)
-
-myScalaDataFrame = DataFrame(z.get("myScalaDataFrame"), sqlContext)
-{% endhighlight %}
-
-  </div>
-</div>
-
-### Object Interpolation
-Some interpreters can interpolate object values from `z` into the paragraph text by using
the 
-`{variable-name}` syntax. The value of any object previously `put` into `z` can be 
-interpolated into a paragraph text by using such a pattern containing the object's name.

-The following example shows one use of this facility:
-
-####In Scala cell:
-```
-z.put("minAge", 35)
-```
-
-####In later SQL cell:
-```
-%sql select * from members where age >= {minAge}
-```
-
-The interpolation of a `{var-name}` pattern is performed only when `z` contains an object
with the specified name.
-But the pattern is left unchanged if the named object does not exist in `z`.
-Further, all `{var-name}` patterns within the paragraph text must must be translatable for
any interpolation to occur -- 
-translation of only some of the patterns in a paragraph text is never done.
-
-In some situations, it is necessary to use { and } characters in a paragraph text without
invoking the 
-object interpolation mechanism. For these cases an escaping mechanism is available -- 
-doubled braces {{ and }} should be used. The following example shows the use of {{ and }}
for passing a 
-regular expression containing just { and } into the paragraph text.
-
-```
-%sql select * from members where name rlike '[aeiou]{{3}}'
-```
-
-To summarize, patterns of the form `{var-name}` within the paragraph text will be interpolated
only if a predefined 
-object of the specified name exists. Additionally, all such patterns within the paragraph
text should also 
-be translatable for any interpolation to occur. Patterns of the form `{{any-text}}` are translated
into `{any-text}`. 
-These translations are performed only when all occurrences of `{`, `}`, `{{`, and `}}` in
the paragraph text conform 
-to one of the two forms described above. Paragraph text containing `{` and/or `}` characters
used in any other way 
-(than `{var-name}` and `{{any-text}}`) is used as-is without any changes. 
-No error is flagged in any case. This behavior is identical to the implementation of a similar
feature in 
-Jupyter's shell invocation using the `!` magic command.
-
-This feature is disabled by default, and must be explicitly turned on for each interpreter
independently 
-by setting the value of an interpreter-specific property to `true`. 
-Consult the _Configuration_ section of each interpreter's documentation 
-to find out if object interpolation is implemented, and the name of the parameter that must
be set to `true` to 
-enable the feature. The name of the parameter used to enable this feature it is different
for each interpreter. 
-For example, the SparkSQL and Shell interpreters use the parameter names `zeppelin.spark.sql.interpolation`
and 
-`zeppelin.shell.interpolation` respectively.
-
-At present only the SparkSQL and Shell interpreters support object interpolation. 
-
-### Form Creation
-
-`ZeppelinContext` provides functions for creating forms.
-In Scala and Python environments, you can create forms programmatically.
-<div class="codetabs">
-  <div data-lang="scala" markdown="1">
-
-{% highlight scala %}
-%spark
-/* Create text input form */
-z.input("formName")
-
-/* Create text input form with default value */
-z.input("formName", "defaultValue")
-
-/* Create select form */
-z.select("formName", Seq(("option1", "option1DisplayName"),
-                         ("option2", "option2DisplayName")))
-
-/* Create select form with default value*/
-z.select("formName", "option1", Seq(("option1", "option1DisplayName"),
-                                    ("option2", "option2DisplayName")))
-{% endhighlight %}
-
-  </div>
-  <div data-lang="python" markdown="1">
-
-{% highlight python %}
-%spark.pyspark
-# Create text input form
-z.input("formName")
-
-# Create text input form with default value
-z.input("formName", "defaultValue")
-
-# Create select form
-z.select("formName", [("option1", "option1DisplayName"),
-                      ("option2", "option2DisplayName")])
-
-# Create select form with default value
-z.select("formName", [("option1", "option1DisplayName"),
-                      ("option2", "option2DisplayName")], "option1")
-{% endhighlight %}
-
-  </div>
-</div>
-
-In sql environment, you can create form in simple template.
-
-```sql
-%spark.sql
-select * from ${table=defaultTableName} where text like '%${search}%'
-```
-
-To learn more about dynamic form, checkout [Dynamic Form](../usage/dynamic_form/intro.html).
-
+See [Zeppelin-Context](../usage/other_features/zeppelin_context.html) for more details.
 
 ## Matplotlib Integration (pyspark)
 Both the `python` and `pyspark` interpreters have built-in support for inline visualization
using `matplotlib`,
@@ -518,3 +367,4 @@ This is to make the server communicate with KDC.
   > **NOTE:** If you do not have permission to access for the above spark-defaults.conf
file, optionally, you can add the above lines to the Spark Interpreter setting through the
Interpreter tab in the Zeppelin UI.
 
 4. That's it. Play with Zeppelin!
+

http://git-wip-us.apache.org/repos/asf/zeppelin/blob/0c3260e9/docs/usage/other_features/zeppelin_context.md
----------------------------------------------------------------------
diff --git a/docs/usage/other_features/zeppelin_context.md b/docs/usage/other_features/zeppelin_context.md
new file mode 100644
index 0000000..5a77727
--- /dev/null
+++ b/docs/usage/other_features/zeppelin_context.md
@@ -0,0 +1,233 @@
+---
+layout: page
+title: "Zeppelin-Context"
+description: "The Zeppelin-Context is a system-wide container for a variety of user-specific
settings and parameters that are accessible across notebooks, cells, and interpreters."
+group: usage/other_features
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+{% include JB/setup %}
+
+# Zeppelin-Context
+
+<div id="toc"></div>
+
+The zeppelin-context is a system-wide container for common utility functions and
+user-specific data. It implements functions for data input, data display, etc. that are
+often needed but are not uniformly available in all interpreters.
+Its single per-user instance is accessible across all of the user's notebooks and cells,
+enabling data exchange between cells - even in different notebooks.
+But the way in which the zeppelin-context is used, and the functionality available differs
+depending on whether or not the associated interpreter is based on a programming language.
+Details of how the zeppelin-context is used for different purposes and in different
+environments is described below.
+
+## Usage in Programming Language Cells
+
+In many programming-language interpreters (e.g. Apache Spark, Python, R) the zeppelin-context
is available
+as a predefined variable `z` that can be used by directly invoking its methods.
+The methods available on the `z` object are described below.
+Other interpreters based on programming languages like spark.dep, Apache Beam, etc. also
provide the
+predefined variable `z`.
+
+### Exploring Spark DataFrames
+In the Apache Spark interpreter, the zeppelin-context provides a `show` method, which, 
+using Zeppelin's `table` feature, can be used to nicely display a Spark DataFrame:
+
+```
+df = spark.read.csv('/path/to/csv')
+z.show(df)
+```
+
+This display functionality using the `show` method is planned to be extended uniformly to

+other interpreters that can access the `z` object.
+
+### Object Exchange
+`ZeppelinContext` extends map and it's shared between the Apache Spark and Python environments.
+So you can put some objects using Scala (in an Apache Spark cell) and read it from Python,
and vice versa.
+
+<div class="codetabs">
+  <div data-lang="scala" markdown="1">
+
+{% highlight scala %}
+// Put object from scala
+%spark
+val myObject = ...
+z.put("objName", myObject)
+
+// Exchanging data frames
+myScalaDataFrame = ...
+z.put("myScalaDataFrame", myScalaDataFrame)
+
+val myPythonDataFrame = z.get("myPythonDataFrame").asInstanceOf[DataFrame]
+{% endhighlight %}
+
+  </div>
+  <div data-lang="python" markdown="1">
+
+{% highlight python %}
+# Get object from python
+%spark.pyspark
+myObject = z.get("objName")
+
+# Exchanging data frames
+myPythonDataFrame = ...
+z.put("myPythonDataFrame", postsDf._jdf)
+
+myScalaDataFrame = DataFrame(z.get("myScalaDataFrame"), sqlContext)
+{% endhighlight %}
+
+  </div>
+</div>
+
+### Form Creation
+
+`ZeppelinContext` provides functions for creating forms.
+In Scala and Python environments, you can create forms programmatically.
+<div class="codetabs">
+  <div data-lang="scala" markdown="1">
+
+{% highlight scala %}
+%spark
+/* Create text input form */
+z.input("formName")
+
+/* Create text input form with default value */
+z.input("formName", "defaultValue")
+
+/* Create select form */
+z.select("formName", Seq(("option1", "option1DisplayName"),
+                         ("option2", "option2DisplayName")))
+
+/* Create select form with default value*/
+z.select("formName", "option1", Seq(("option1", "option1DisplayName"),
+                                    ("option2", "option2DisplayName")))
+{% endhighlight %}
+
+  </div>
+  <div data-lang="python" markdown="1">
+
+{% highlight python %}
+%spark.pyspark
+# Create text input form
+z.input("formName")
+
+# Create text input form with default value
+z.input("formName", "defaultValue")
+
+# Create select form
+z.select("formName", [("option1", "option1DisplayName"),
+                      ("option2", "option2DisplayName")])
+
+# Create select form with default value
+z.select("formName", [("option1", "option1DisplayName"),
+                      ("option2", "option2DisplayName")], "option1")
+{% endhighlight %}
+
+  </div>
+</div>
+
+In sql environment, you can create form in simple template.
+
+```sql
+%spark.sql
+select * from ${table=defaultTableName} where text like '%${search}%'
+```
+
+To learn more about dynamic form, checkout [Dynamic Form](../usage/dynamic_form/intro.html).
+
+### Interpreter-Specific Functions
+
+Some interpreters use a subclass of `BaseZepplinContext` augmented with interpreter-specific
functions. 
+For example functions of the dependency loader (%spark.dep) can be invoked as `z.addRepo()`,
`z.load()`, etc. 
+Such interpreter-specific functions are described within each interpreter's documentation.
+
+## Usage with Embedded Commands
+
+In certain interpreters (see table below) zeppelin-context features may be invoked by embedding

+command strings into the paragraph text. Such embedded command strings are used to invoke

+dynamic-forms and object-interpolation as described below. 
+
+|             Interpreters that use Embedded Commands               |
+|-------------------------------------------------------------------|
+|spark.sql (\*), bigquery, cassandra, elasticsearch, file, hbase, ignite, jdbc (\*), kylin,
livy, markdown, neo4j, pig, python, shell (\*), zengine |
+
+Dynamic forms are available in all of the interpreters in the table above, 
+but object interpolation is only available in a small, but growing, list of interpreters

+(marked with an asterisk in the table above).
+Both these zeppelin-context features are described below.
+
+### Dynamic Forms
+
+Patterns of the form ${ ... } are used to dynamically create additional HTML elements
+for requesting user input (that replaces the corresponding pattern in the paragraph text).

+Currently only [text](https://developer.mozilla.org/en-US/docs/Web/HTML/Element/input/text),

+[select](https://developer.mozilla.org/en-US/docs/Web/HTML/Element/select) with 
+[options](https://developer.mozilla.org/en-US/docs/Web/HTML/Element/option), and 
+[checkbox](https://developer.mozilla.org/en-US/docs/Web/HTML/Element/input/checkbox) are
supported.
+
+Dynamic forms are described in detail here: [Dynamic Form](../usage/dynamic_form/intro.html).
+
+### Object Interpolation
+Some interpreters can interpolate object values from `z` into the paragraph text by using
the
+`{variable-name}` syntax. The value of any object previously `put` into `z` can be
+interpolated into a paragraph text by using such a pattern containing the object's name.
+The following example shows one use of this facility:
+
+####In Scala cell:
+```
+z.put("minAge", 35)
+```
+
+####In later SQL cell:
+```
+%sql select * from members where age >= {minAge}
+```
+
+The interpolation of a `{var-name}` pattern is performed only when `z` contains an object
with the specified name.
+But the pattern is left unchanged if the named object does not exist in `z`.
+Further, all `{var-name}` patterns within the paragraph text must must be translatable for
any interpolation to occur --
+translation of only some of the patterns in a paragraph text is never done.
+
+In some situations, it is necessary to use { and } characters in a paragraph text without
invoking the
+object interpolation mechanism. For these cases an escaping mechanism is available --
+doubled braces {{ and }} should be used. The following example shows the use of {{ and }}
for passing a
+regular expression containing just { and } into the paragraph text.
+
+```
+%sql select * from members where name rlike '[aeiou]{{3}}'
+```
+
+To summarize, patterns of the form `{var-name}` within the paragraph text will be interpolated
only if a predefined
+object of the specified name exists. Additionally, all such patterns within the paragraph
text should also
+be translatable for any interpolation to occur. Patterns of the form `{{any-text}}` are translated
into `{any-text}`.
+These translations are performed only when all occurrences of `{`, `}`, `{{`, and `}}` in
the paragraph text conform
+to one of the two forms described above. Paragraph text containing `{` and/or `}` characters
used in any other way
+(than `{var-name}` and `{{any-text}}`) is used as-is without any changes.
+No error is flagged in any case. This behavior is identical to the implementation of a similar
feature in
+Jupyter's shell invocation using the `!` magic command.
+
+This feature is disabled by default, and must be explicitly turned on for each interpreter
independently
+by setting the value of an interpreter-specific property to `true`.
+Consult the _Configuration_ section of each interpreter's documentation
+to find out if object interpolation is implemented, and the name of the parameter that must
be set to `true` to
+enable the feature. The name of the parameter used to enable this feature it is different
for each interpreter.
+For example, the SparkSQL and Shell interpreters use the parameter names `zeppelin.spark.sql.interpolation`
and
+`zeppelin.shell.interpolation` respectively.
+
+At present only the SparkSQL, JDBC, and Shell interpreters support object interpolation.
+
+
+
+


Mime
View raw message