Mailing-List: contact commits-help@zeppelin.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@zeppelin.apache.org
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
From: moon@apache.org
To: commits@zeppelin.apache.org
Message-Id: <3c90386dc2ff43f1a110d03b60898b53@git.apache.org>
Subject: zeppelin git commit: SSL Support for Groovy Interpreter HTTP requests
 [ZEPPELIN-2443]
Date: Fri, 28 Apr 2017 09:52:16 +0000 (UTC)
archived-at: Fri, 28 Apr 2017 09:52:19 -0000

Repository: zeppelin
Updated Branches:
  refs/heads/master ec8461365 -> 925d09cef


SSL Support for Groovy Interpreter HTTP requests [ZEPPELIN-2443]

### What is this PR for?

Target: Create ability to call http services with custom keystores in a groovy way.
The following should work:
```groovy
//connect to host xxx.yyy with special keystore
HTTP.get(
  url: "https://xxx.yyy/zzz",
  ssl: " HTTP.getKeystoreSSLContext('./xxx_yyy_keystore.jks', 'testpass') "
)
//take context initialization code from groovy interpreter properties
HTTP.get(
  url: "https://xxx.yyy/zzz",
  ssl: g.SSL_CONTEXT_FROM_GROOVY_INTERPRET_PROPERTIES
)
//connect to host xxx.yyy with trust all (do not check trust certificates - dev mode only)
HTTP.get(
  url: "https://xxx.yyy/zzz",
  ssl: " HTTP.getNaiveSSLContext() "
)
//
HTTP.get(
  url: "https://xxx.yyy/zzz",
  ssl: " MyCustomSSLBuilder.build() "
)
```

### What type of PR is it?
Improvement

### Todos
* [ ] - Task

### What is the Jira issue?
[ZEPPELIN-2443]

### How should this be tested?
follow the samples above or in documentation

### Questions:
* Does the licenses files need update? NO
* Is there breaking changes for older versions? NO
* Does this needs documentation? YES

Author: dlukyanov <dlukyanov@ukr.net>

Closes #2287 from dlukyanov/master and squashes the following commits:

4baa22e [dlukyanov] ZEPPELIN-2443


Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo
Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/925d09ce
Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/925d09ce
Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/925d09ce

Branch: refs/heads/master
Commit: 925d09cef54e3056ae10ec0e547f46097e639158
Parents: ec84613
Author: dlukyanov <dlukyanov@ukr.net>
Authored: Wed Apr 26 09:38:05 2017 +0300
Committer: Lee moon soo <moon@apache.org>
Committed: Fri Apr 28 02:52:13 2017 -0700

----------------------------------------------------------------------
 docs/interpreter/groovy.md            |  56 +++++++++++-----
 groovy/src/main/resources/HTTP.groovy | 104 +++++++++++++++++++++++++++--
 2 files changed, 136 insertions(+), 24 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/zeppelin/blob/925d09ce/docs/interpreter/groovy.md
----------------------------------------------------------------------
diff --git a/docs/interpreter/groovy.md b/docs/interpreter/groovy.md
index 01074a3..f64cbde 100644
--- a/docs/interpreter/groovy.md
+++ b/docs/interpreter/groovy.md
@@ -39,7 +39,8 @@ def r = HTTP.get(
   headers: [
     'Accept':'application/json',
     //'Authorization:' : g.getProperty('search_auth'),
-  ] 
+  ],
+  ssl : g.getProperty('search_ssl') // assume groovy interpreter property search_ssl = HTTP.getNaiveSSLContext()
 )
 //check response code
 if( r.response.code==200 ) {
@@ -76,41 +77,62 @@ g.table(
 
 * `g.angular(String name)`
 
-Returns angular object by name. Look up notebook scope first and then global scope.
+   Returns angular object by name. Look up notebook scope first and then global scope.
 
 
 * `g.angularBind(String name, Object value)`
-
-Assign a new `value` into angular object `name`
+ 
+   Assign a new `value` into angular object `name`
 
 
 * `java.util.Properties g.getProperties()`
 
-returns all properties defined for this interpreter
+   returns all properties defined for this interpreter
 
 
 * `String g.getProperty('PROPERTY_NAME')` 
-```groovy 
-g.PROPERTY_NAME
-g.'PROPERTY_NAME'
-g['PROPERTY_NAME']
-g.getProperties().getProperty('PROPERTY_NAME')
-```
+   ```groovy 
+   g.PROPERTY_NAME
+   g.'PROPERTY_NAME'
+   g['PROPERTY_NAME']
+   g.getProperties().getProperty('PROPERTY_NAME')
+   ```
 
-All above the accessor to named property defined in groovy interpreter.
-In this case with name `PROPERTY_NAME`
+   All above the accessor to named property defined in groovy interpreter.
+   In this case with name `PROPERTY_NAME`
 
 
 * `groovy.xml.MarkupBuilder g.html()`
 
-Starts or continues rendering of `%angular` to output and returns [groovy.xml.MarkupBuilder](http://groovy-lang.org/processing-xml.html#_markupbuilder)
-MarkupBuilder is usefull to generate html (xml)
+   Starts or continues rendering of `%angular` to output and returns [groovy.xml.MarkupBuilder](http://groovy-lang.org/processing-xml.html#_markupbuilder)
+   MarkupBuilder is usefull to generate html (xml)
 
 * `void g.table(obj)`
 
-starts or continues rendering table rows.
+   starts or continues rendering table rows.
+
+   obj:  List(rows) of List(columns) where first line is a header 
+
+
+* `g.input(name, value )`
+
+   Creates `text` input with value specified. The parameter `value` is optional.
+   
+* `g.select(name, default, Map<Object, String> options)`
+
+   Creates `select` input with defined options. The parameter `default` is optional.
+
+   ```g.select('sex', 'm', ['m':'man', 'w':'woman'])```
+   
+* `g.checkbox(name, Collection checked, Map<Object, String> options)`
 
-obj:  List(rows) of List(columns) where first line is a header 
+   Creates `checkbox` input.
+   
+* `g.get(name, default)`
 
+   Returns interpreter-based variable. Visibility depends on interpreter scope. The parameter `default` is optional.
 
+* `g.put(name, value)`
 
+   Stores new value into interpreter-based variable. Visibility depends on interpreter scope.
+   

http://git-wip-us.apache.org/repos/asf/zeppelin/blob/925d09ce/groovy/src/main/resources/HTTP.groovy
----------------------------------------------------------------------
diff --git a/groovy/src/main/resources/HTTP.groovy b/groovy/src/main/resources/HTTP.groovy
index fe4eb36..63fdc04 100644
--- a/groovy/src/main/resources/HTTP.groovy
+++ b/groovy/src/main/resources/HTTP.groovy
@@ -17,6 +17,16 @@
 
 import groovy.json.JsonOutput
 
+import java.security.KeyStore;
+import javax.net.ssl.KeyManagerFactory;
+import javax.net.ssl.KeyManager;
+import javax.net.ssl.TrustManager;
+import javax.net.ssl.X509TrustManager;
+import java.security.cert.X509Certificate;
+import javax.net.ssl.SSLContext;
+import java.security.SecureRandom;
+import javax.net.ssl.HttpsURLConnection;
+
 /**
  * simple http rest client for groovy
  * by dlukyanov@ukr.net 
@@ -44,6 +54,11 @@ public class HTTP{
 		return send(ctx);
 	}
 	
+	public static Map<String,Object> head(Map<String,Object> ctx)throws IOException{
+		ctx.put('method','HEAD');
+		return send(ctx);
+	}
+	
 	public static Map<String,Object> post(Map<String,Object> ctx)throws IOException{
 		ctx.put('method','POST');
 		return send(ctx);
@@ -59,6 +74,16 @@ public class HTTP{
 		return send(ctx);
 	}
 	
+	/**
+	 * @param url string where to send request
+	 * @param query Map parameters to append to url
+	 * @param method http method to be used in request. standard methods: GET, POST, PUT, DELETE, HEAD
+	 * @param headers key-value map with headers that should be sent with request
+	 * @param body request body/data to send to url (InputStream, CharSequence, or Map for json and x-www-form-urlencoded context types)
+	 * @param encoding encoding name to use to send/receive data - default UTF-8
+	 * @param receiver Closure that will be called to receive data from server. Defaults: `HTTP.JSON_RECEIVER` for json content-type and `HTTP.TEXT_RECEIVER` otherwise. Available: `HTTP.FILE_RECEIVER(File)` - stores response to file.
+	 * @param ssl javax.net.ssl.SSLContext or String that evaluates the javax.net.ssl.SSLContext. example: send( url:..., ssl: "HTTP.getKeystoreSSLContext('./keystore.jks', 'testpass')" )
+	 */
 	public static Map<String,Object> send(Map<String,Object> ctx)throws IOException{
 		String             url      = ctx.url;
 		Map<String,String> headers  = (Map<String,String>)ctx.headers;
@@ -67,6 +92,7 @@ public class HTTP{
 		String             encoding = ctx.encoding?:"UTF-8";
 		Closure            receiver = (Closure)ctx.receiver;
 		Map<String,String> query    = (Map<String,String>)ctx.query;
+		Object             sslCtxObj= ctx.ssl;
 		
 		//copy context and set default values
 		ctx = [:] + ctx;
@@ -78,14 +104,28 @@ public class HTTP{
 		}
 		
 		HttpURLConnection connection = (HttpURLConnection) new URL(url).openConnection();
+		if(sslCtxObj!=null && connection instanceof HttpsURLConnection){
+			SSLContext         sslCtx   = null;
+			if(sslCtxObj instanceof SSLContext){
+				sslCtx = (SSLContext)sslCtxObj;
+			}else if(sslCtxObj instanceof CharSequence){
+				//assume this is a groovy code to get ssl context
+				sslCtx = evaluateSSLContext((CharSequence)sslCtxObj);
+			}else{
+				throw new IllegalArgumentException("Unsupported ssl parameter ${sslCtxObj.getClass()}")
+			}
+			((HttpsURLConnection)connection).setSSLSocketFactory(sslCtx.getSocketFactory());
+		}
 		
         connection.setDoOutput(true);
         connection.setRequestMethod(method);
 		if ( headers!=null && !headers.isEmpty() ) {
 			//add headers
 			for (Map.Entry<String, String> entry : headers.entrySet()) {
-				connection.addRequestProperty(entry.getKey(), entry.getValue());
-				if("content-type".equals(entry.getKey().toLowerCase()))contentType=entry.getValue();
+				if(entry.getValue()){
+					connection.addRequestProperty(entry.getKey(), entry.getValue());
+					if("content-type".equals(entry.getKey().toLowerCase()))contentType=entry.getValue();
+				}
 			}
 		}
 		
@@ -97,18 +137,20 @@ public class HTTP{
 			}else if(body instanceof InputStream){
 				out << (InputStream)body;
 			}else if(body instanceof Map){
-				if( contentType.matches("(?i)[^/]+/json") ){
+				if( contentType =~ "(?i)[^/]+/json" ) {
 					out.withWriter((String)ctx.encoding){
 						it.append( JsonOutput.toJson((Map)body) );
-						it.flush();
 					}
-				}else{
-					throw new IOException("Map body type supported only for */json content-type");
+				} else if( contentType =~ "(?i)[^/]+/x-www-form-urlencoded" ) {
+					out.withWriter((String)ctx.encoding) {
+						it.append( ((Map)body).collect{k,v-> ""+k+"="+URLEncoder.encode((String)v,'UTF-8') }.join('&') )
+					}
+				} else {
+					throw new IOException("Map body type supported only for */json of */x-www-form-urlencoded content-type");
 				}
 			}else if(body instanceof CharSequence){
 				out.withWriter((String)ctx.encoding){
 					it.append((CharSequence)body);
-					it.flush();
 				}
 			}else{
 				throw new IOException("Unsupported body type: "+body.getClass());
@@ -151,4 +193,52 @@ public class HTTP{
 		}
 		return ctx;
 	}
+	
+	@groovy.transform.Memoized
+	public static SSLContext getKeystoreSSLContext(String keystorePath, String keystorePass, String keystoreType="JKS", String keyPass = null){
+		if(keyPass == null) keyPass=keystorePass;
+		KeyStore clientStore = KeyStore.getInstance(keystoreType);
+		clientStore.load(new File( keystorePath ).newInputStream(), keystorePass.toCharArray());
+		KeyManagerFactory kmf = KeyManagerFactory.getInstance(KeyManagerFactory.getDefaultAlgorithm());
+		kmf.init(clientStore, keyPass.toCharArray());
+		KeyManager[] kms = kmf.getKeyManagers();
+		//init TrustCerts
+		TrustManager[] trustCerts = new TrustManager[1];                
+		trustCerts[0] = new X509TrustManager() {
+			public void checkClientTrusted( final X509Certificate[] chain, final String authType ) { }
+			public void checkServerTrusted( final X509Certificate[] chain, final String authType ) { }
+			public X509Certificate[] getAcceptedIssuers() {
+				return null;
+			}	
+		}
+		SSLContext sslContext = SSLContext.getInstance("TLS");
+		sslContext.init(kms, trustCerts, new SecureRandom());
+		return sslContext;
+	}
+    
+    @groovy.transform.Memoized
+	public static SSLContext getNaiveSSLContext(){
+		System.err.println("HTTP.getNaiveSSLContext() used. Must be disabled on prod!");
+        KeyManager[] kms = new KeyManager[0];
+		TrustManager[] trustCerts = new TrustManager[1];                
+		trustCerts[0] = new X509TrustManager() {
+			public void checkClientTrusted( final X509Certificate[] chain, final String authType ) { }
+			public void checkServerTrusted( final X509Certificate[] chain, final String authType ) { }
+			public X509Certificate[] getAcceptedIssuers() {
+				return null;
+			}	
+		}
+		SSLContext sslContext = SSLContext.getInstance("TLS");
+		sslContext.init(null, trustCerts, new SecureRandom());
+		return sslContext;
+	}
+	
+	/**
+	 * evaluates code that should return SSLContext
+	 */
+    @groovy.transform.Memoized
+	public static SSLContext evaluateSSLContext(CharSequence code) {
+		Object ssl = new GroovyShell( HTTP.class.getClassLoader() ).evaluate( code as String );
+		return (SSLContext) ssl;
+	}
 }