+ + + +

+ + +

Lens Interpreter for Apache Zeppelin

+ +

Overview

+ +

Apache Lens provides an Unified Analytics interface. Lens aims to cut the Data Analytics silos by providing a single view of data across multiple tiered data stores and optimal execution environment for the analytical query. It seamlessly integrates Hadoop with traditional data warehouses to appear like one.

+ +

Installing and Running Lens

+ +

In order to use Lens interpreters, you may install Apache Lens in some simple steps:

+ +

Download Lens for latest version from the ASF. Or the older release can be found in the Archives.
Before running Lens, you have to set HIVEHOME and HADOOPHOME. If you want to get more information about this, please refer to here. Lens also provides Pseudo Distributed mode. Lens pseudo-distributed setup is done by using docker. Hive server and hadoop daemons are run as separate processes in lens pseudo-distributed setup.
Now, you can start lens server (or stop).

./bin/lens-ctl start # (or stop)
+

Configuring Lens Interpreter

+ +

At the "Interpreters" menu, you can edit Lens interpreter or create new one. Zeppelin provides these properties for Lens.

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +

Property Name	value	Description
lens.client.dbname	default	The database schema name
lens.query.enable.persistent.resultset	false	Whether to enable persistent resultset for queries. When enabled, server will fetch results from driver, custom format them if any and store in a configured location. The file name of query output is queryhandle-id, with configured extensions
lens.server.base.url	http://hostname:port/lensapi	The base url for the lens server. you have to edit "hostname" and "port" that you may use(ex. http://0.0.0.0:9999/lensapi)
lens.session.cluster.user	default	Hadoop cluster username
zeppelin.lens.maxResult	1000	Max number of rows to display
zeppelin.lens.maxThreads	10	If concurrency is true then how many threads?
zeppelin.lens.run.concurrent	true	Run concurrent Lens Sessions
xxx	yyy	anything else from [Configuring lens server](https://lens.apache.org/admin/config-server.html)

+ +

Apache Lens Interpreter Setting

+ +

Interpreter Binding for Zeppelin Notebook

+ +

After configuring Lens interpreter, create your own notebook, then you can bind interpreters like below image.

+ +

Zeppelin Notebook Interpreter Binding

+ +

For more interpreter binding information see here.

+ +

How to use

+ +

You can analyze your data by using OLAP Cube QL which is a high level SQL like language to query and describe data sets organized in data cubes. +You may experience OLAP Cube like this Video tutorial. +As you can see in this video, they are using Lens Client Shell(./bin/lens-cli.sh). All of these functions also can be used on Zeppelin by using Lens interpreter.

+ +

Create and Use (Switch) Databases.

create database newDb
+

use newDb
+

Create Storage.

create storage your/path/to/lens/client/examples/resources/db-storage.xml
+

Create Dimensions, Show fields and join-chains of them.

create dimension your/path/to/lens/client/examples/resources/customer.xml
+

dimension show fields customer
+

dimension show joinchains customer
+

Create Caches, Show fields and join-chains of them.

create cube your/path/to/lens/client/examples/resources/sales-cube.xml
+

cube show fields sales
+

cube show joinchains sales
+

Create Dimtables and Fact.

create dimtable your/path/to/lens/client/examples/resources/customer_table.xml
+

create fact your/path/to/lens/client/examples/resources/sales-raw-fact.xml
+

Add partitions to Dimtable and Fact.

dimtable add single-partition --dimtable_name customer_table --storage_name local 
+--path your/path/to/lens/client/examples/resources/customer-local-part.xml
+

fact add partitions --fact_name sales_raw_fact --storage_name local 
+--path your/path/to/lens/client/examples/resources/sales-raw-local-parts.xml
+

Now, you can run queries on cubes.

query execute cube select customer_city_name, product_details.description, 
+product_details.category, product_details.color, store_sales from sales 
+where time_range_in(delivery_time, '2015-04-11-00', '2015-04-13-00')
+

Lens Query Result

+ +

These are just examples that provided in advance by Lens. If you want to explore whole tutorials of Lens, see the tutorial video.

+ +

Lens UI Service

+ +

Lens also provides web UI service. Once the server starts up, you can open the service on http://serverhost:19999/index.html and browse. You may also check the structure that you made and use query easily here.

+ +

Lens UI Service

+ +

+ + +

+ +

+ + + +

+ + +

Livy Interpreter for Apache Zeppelin

+ +

Overview

+ +

Livy is an open source REST interface for interacting with Spark from anywhere. It supports executing snippets of code or programs in a Spark context that runs locally or in YARN.

+ +

Interactive Scala, Python and R shells
Batch submissions in Scala, Java, Python
Multi users can share the same server (impersonation support)
Can be used for submitting jobs from anywhere with REST
Does not require any code change to your programs

+ +

Requirements

+ +

Additional requirements for the Livy interpreter are:

+ +

Spark 1.3 or above.
Livy server.

+ +

Configuration

+ +

We added some common configurations for spark, and you can set any configuration you want. +You can find all Spark configurations in here. +And instead of starting property with spark. it should be replaced with livy.spark.. +Example: spark.driver.memory to livy.spark.driver.memory

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +

Property	Default	Description
zeppelin.livy.url	http://localhost:8998	URL where livy server is running
zeppelin.livy.spark.sql.maxResult	1000	Max number of Spark SQL result to display.
zeppelin.livy.spark.sql.field.truncate	true	Whether to truncate field values longer than 20 characters or not
zeppelin.livy.session.create_timeout	120	Timeout in seconds for session creation
zeppelin.livy.displayAppInfo	true	Whether to display app info
zeppelin.livy.pull_status.interval.millis	1000	The interval for checking paragraph execution status
livy.spark.driver.cores		Driver cores. ex) 1, 2.
livy.spark.driver.memory		Driver memory. ex) 512m, 32g.
livy.spark.executor.instances		Executor instances. ex) 1, 4.
livy.spark.executor.cores		Num cores per executor. ex) 1, 4.
livy.spark.executor.memory		Executor memory per worker instance. ex) 512m, 32g.
livy.spark.dynamicAllocation.enabled		Use dynamic resource allocation. ex) True, False.
livy.spark.dynamicAllocation.cachedExecutorIdleTimeout		Remove an executor which has cached data blocks.
livy.spark.dynamicAllocation.minExecutors		Lower bound for the number of executors.
livy.spark.dynamicAllocation.initialExecutors		Initial number of executors to run.
livy.spark.dynamicAllocation.maxExecutors		Upper bound for the number of executors.
livy.spark.jars.packages		Adding extra libraries to livy interpreter
zeppelin.livy.ssl.trustStore		client trustStore file. Used when livy ssl is enabled
zeppelin.livy.ssl.trustStorePassword		password for trustStore file. Used when livy ssl is enabled
zeppelin.livy.http.headers	key_1: value_1; key_2: value_2	custom http headers when calling livy rest api. Each http header is separated by `;`, and each header is one key value pair where key value is separated by `:`

+ +

We remove livy.spark.master in zeppelin-0.7. Because we sugguest user to use livy 0.3 in zeppelin-0.7. And livy 0.3 don't allow to specify livy.spark.master, it enfornce yarn-cluster mode.

+ +

Adding External libraries

+ +

You can load dynamic library to livy interpreter by set livy.spark.jars.packages property to comma-separated list of maven coordinates of jars to include on the driver and executor classpaths. The format for the coordinates should be groupId:artifactId:version.

+ +

Example

+ + + + + + + + + + + + +

Property	Example	Description
livy.spark.jars.packages	io.spray:spray-json_2.10:1.3.1	Adding extra libraries to livy interpreter

+ +

How to use

+ +

Basically, you can use

+ +

spark

%livy.spark
+sc.version
+

pyspark

%livy.pyspark
+print "1"
+

sparkR

%livy.sparkr
+hello <- function( name ) {
+    sprintf( "Hello, %s", name );
+}
+
+hello("livy")
+

Impersonation

+ +

When Zeppelin server is running with authentication enabled, +then this interpreter utilizes Livyâs user impersonation feature +i.e. sends extra parameter for creating and running a session ("proxyUser": "${loggedInUser}"). +This is particularly useful when multi users are sharing a Notebook server.

+ +

Apply Zeppelin Dynamic Forms

+ +

You can leverage Zeppelin Dynamic Form. Form templates is only avalible for livy sql interpreter.

%livy.sql
+select * from products where ${product_id=1}
+

And creating dynamic formst programmatically is not feasible in livy interpreter, because ZeppelinContext is not available in livy interpreter.

+ +

Shared SparkContext

+ +

Starting from livy 0.5 which is supported by Zeppelin 0.8.0, SparkContext is shared between scala, python, r and sql. +That means you can query the table via %livy.sql when this table is registered in %livy.spark, %livy.pyspark, $livy.sparkr.

+ +

FAQ

+ +

Livy debugging: If you see any of these in error console

+ +

+
Connect to livyhost:8998 [livyhost/127.0.0.1, livyhost/0:0:0:0:0:0:0:1] failed: Connection refused
+

+ +

Looks like the livy server is not up yet or the config is wrong

+ +

+
Exception: Session not found, Livy server would have restarted, or lost session.
+

+ +

The session would have timed out, you may need to restart the interpreter.

+ +

+
Blacklisted configuration values in session config: spark.master
+

+ +

Edit conf/spark-blacklist.conf file in livy server and comment out #spark.master line.

+ +

If you choose to work on livy in apps/spark/java directory in https://github.com/cloudera/hue, +copy spark-user-configurable-options.template to spark-user-configurable-options.conf file in livy server and comment out #spark.master.

+ +

+ + +

+ +