metron-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From l...@apache.org
Subject incubator-metron git commit: METRON-270 Add Zeppelin to the platform (justinleet) closes apache/incubator-metron#423
Date Fri, 27 Jan 2017 14:03:46 GMT
Repository: incubator-metron
Updated Branches:
  refs/heads/master fa6f3df51 -> 9890018af


METRON-270 Add Zeppelin to the platform (justinleet) closes apache/incubator-metron#423


Project: http://git-wip-us.apache.org/repos/asf/incubator-metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-metron/commit/9890018a
Tree: http://git-wip-us.apache.org/repos/asf/incubator-metron/tree/9890018a
Diff: http://git-wip-us.apache.org/repos/asf/incubator-metron/diff/9890018a

Branch: refs/heads/master
Commit: 9890018af0b6df3efba07927e72fc4aeb752cabb
Parents: fa6f3df
Author: justinleet <justinjleet@gmail.com>
Authored: Fri Jan 27 09:00:03 2017 -0500
Committer: leet <leet@apache.org>
Committed: Fri Jan 27 09:00:03 2017 -0500

----------------------------------------------------------------------
 metron-deployment/README.md                         |  2 ++
 .../METRON/CURRENT/configuration/metron-env.xml     |  6 ++++++
 .../common-services/METRON/CURRENT/metainfo.xml     | 10 ++++++++++
 .../CURRENT/package/scripts/indexing_master.py      | 12 ++++++++++++
 .../CURRENT/package/scripts/params/params_linux.py  |  6 ++++++
 .../CURRENT/package/scripts/params/status_params.py |  3 +++
 .../METRON/CURRENT/service_advisor.py               | 16 ++++++++++++----
 metron-platform/metron-indexing/README.md           |  8 ++++++++
 8 files changed, 59 insertions(+), 4 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/9890018a/metron-deployment/README.md
----------------------------------------------------------------------
diff --git a/metron-deployment/README.md b/metron-deployment/README.md
index 120f835..eb93df4 100644
--- a/metron-deployment/README.md
+++ b/metron-deployment/README.md
@@ -133,6 +133,8 @@ Notably, the URL for the GeoIP database that is preloaded (and is prefilled
by d
 
 After installation, a custom action is available in Ambari (where stop / start services are)
to install Elasticsearch templates.  Similar to this, a custom Kibana action to Load Template
is available.
 
+Another custom action is available in Ambari to import Zeppelin dashboards. See the [metron-indexing
README.md](../metron-platform/metron-indexing/README.md)
+
 #### Offline installation
 Currently there is only one point that would reach out to the internet during an install.
 This is the URL for the GeoIP database information.
 

http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/9890018a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-env.xml
----------------------------------------------------------------------
diff --git a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-env.xml
b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-env.xml
index 6015965..eeb2037 100644
--- a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-env.xml
+++ b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-env.xml
@@ -110,6 +110,12 @@
         <description>URL of Storm UI (storm.ui.hostname:8744)</description>
         <value></value>
     </property>
+    <property require-input = "true">
+        <name>zeppelin_server_url</name>
+        <display-name>Zeppelin Server Address</display-name>
+        <description>URL of Zeppelin (zeppelin.server.addr:zeppelin.server.port)</description>
+        <value></value>
+    </property>
     <property>
         <name>repo_type</name>
         <display-name>Repository Type</display-name>

http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/9890018a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/metainfo.xml
----------------------------------------------------------------------
diff --git a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/metainfo.xml
b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/metainfo.xml
index 3c71328..bbe7187 100644
--- a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/metainfo.xml
+++ b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/metainfo.xml
@@ -164,6 +164,14 @@
                   <timeout>600</timeout>
                 </commandScript>
               </customCommand>
+              <customCommand>
+                <name>ZEPPELIN_NOTEBOOK_IMPORT</name>
+                <commandScript>
+                  <script>scripts/indexing_master.py</script>
+                  <scriptType>PYTHON</scriptType>
+                  <timeout>600</timeout>
+                </commandScript>
+              </customCommand>
           </customCommands>
         </component>
       </components>
@@ -224,6 +232,7 @@
         <service>STORM</service>
         <service>ZOOKEEPER</service>
         <service>HBASE</service>
+        <service>ZEPPELIN</service>
       </requiredServices>
 
       <configuration-dependencies>
@@ -232,6 +241,7 @@
         <config-type>storm-site</config-type>
         <config-type>kafka-broker</config-type>
         <config-type>kafka-env</config-type>
+        <config-type>zeppelin-config</config-type>
       </configuration-dependencies>
       <restartRequiredAfterChange>true</restartRequiredAfterChange>
       <quickLinksConfigurations>

http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/9890018a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/indexing_master.py
----------------------------------------------------------------------
diff --git a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/indexing_master.py
b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/indexing_master.py
index 43a62f9..efc048d 100755
--- a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/indexing_master.py
+++ b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/indexing_master.py
@@ -14,6 +14,7 @@ See the License for the specific language governing permissions and
 limitations under the License.
 """
 
+import os
 from resource_management.core.exceptions import ComponentIsNotRunning
 from resource_management.core.logger import Logger
 from resource_management.core.resources.system import Execute
@@ -115,6 +116,17 @@ class Indexing(Script):
         yaf_cmd = ambari_format('curl -s -XDELETE "http://{es_http_url}/yaf_index*"')
         Execute(yaf_cmd, logoutput=True)
 
+    def zeppelin_notebook_import(self, env):
+        from params import params
+        env.set_params(params)
+
+        Logger.info(ambari_format('Searching for Zeppelin Notebooks in {metron_config_zeppelin_path}'))
+        for dirName, subdirList, files in os.walk(params.metron_config_zeppelin_path):
+            for fileName in files:
+                if fileName.endswith(".json"):
+                    zeppelin_cmd = ambari_format(
+                        'curl -s -XPOST http://{zeppelin_server_url}/api/notebook/import
-d "@' + os.path.join(dirName, fileName) + '"')
+                    Execute(zeppelin_cmd, logoutput=True)
 
 if __name__ == "__main__":
     Indexing().execute()

http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/9890018a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/params/params_linux.py
----------------------------------------------------------------------
diff --git a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/params/params_linux.py
b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/params/params_linux.py
index 4e00971..2427d25 100755
--- a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/params/params_linux.py
+++ b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/params/params_linux.py
@@ -96,6 +96,9 @@ if has_zk_host:
 # Storm
 storm_rest_addr = status_params.storm_rest_addr
 
+# Zeppelin
+zeppelin_server_url = status_params.zeppelin_server_url
+
 # Kafka
 kafka_hosts = default("/clusterHostInfo/kafka_broker_hosts", [])
 has_kafka_host = not len(kafka_hosts) == 0
@@ -162,3 +165,6 @@ metron_threat_intel_error_topic = status_params.metron_threat_intel_error_topic
 bro_index_path = tmp_dir + "/bro_index.template"
 snort_index_path = tmp_dir + "/snort_index.template"
 yaf_index_path = tmp_dir + "/yaf_index.template"
+
+# Zeppelin Notebooks
+metron_config_zeppelin_path = format("{metron_config_path}/zeppelin")

http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/9890018a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/params/status_params.py
----------------------------------------------------------------------
diff --git a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/params/status_params.py
b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/params/status_params.py
index 4880247..e8a8568 100644
--- a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/params/status_params.py
+++ b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/params/status_params.py
@@ -51,3 +51,6 @@ enrichment_configured_flag_file = metron_zookeeper_config_path + '/../metron_enr
 
 # Storm
 storm_rest_addr = config['configurations']['metron-env']['storm_rest_addr']
+
+# Zeppelin
+zeppelin_server_url = config['configurations']['metron-env']['zeppelin_server_url']

http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/9890018a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/service_advisor.py
----------------------------------------------------------------------
diff --git a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/service_advisor.py
b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/service_advisor.py
index 4c492ab..8aff88c 100644
--- a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/service_advisor.py
+++ b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/service_advisor.py
@@ -112,12 +112,20 @@ class METRON${metron.short.version}ServiceAdvisor(service_advisor.ServiceAdvisor
                 if property not in storm_site:
                     putStormSiteProperty(property, desired_value)
                 elif  property == "topology.classpath" and storm_site[property] != desired_value:
-                    topololgyClasspath = storm_site[property]
+                    topologyClasspath = storm_site[property]
                     #check that desired values exist in topology.classpath. append them if
they do not
                     for path in desired_value.split(':'):
-                        if path not in topololgyClasspath:
-                            topololgyClasspath += ":" + path
-                    putStormSiteProperty(property,topololgyClasspath)
+                        if path not in topologyClasspath:
+                            topologyClasspath += ":" + path
+                    putStormSiteProperty(property,topologyClasspath)
+
+        #Suggest Zeppelin Server URL
+        if "zeppelin-config" in services["configurations"]:
+            zeppelinServerHost = self.getComponentHostNames(services, "ZEPPELIN", "ZEPPELIN_MASTER")[0]
+            zeppelinServerPort = services["configurations"]["zeppelin-config"]["properties"]["zeppelin.server.port"]
+            zeppelinServerUrl = zeppelinServerHost + ":" + zeppelinServerPort
+            putMetronEnvProperty = self.putProperty(configurations, "metron-env", services)
+            putMetronEnvProperty("zeppelin_server_url", zeppelinServerUrl)
 
     def validateSTORMSiteConfigurations(self, properties, recommendedDefaults, configurations,
services, hosts):
 

http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/9890018a/metron-platform/metron-indexing/README.md
----------------------------------------------------------------------
diff --git a/metron-platform/metron-indexing/README.md b/metron-platform/metron-indexing/README.md
index 5611838..cbe7a76 100644
--- a/metron-platform/metron-indexing/README.md
+++ b/metron-platform/metron-indexing/README.md
@@ -140,3 +140,11 @@ Finally, if workers and executors are new to you or you don't know where
 to modify the flux file, the following might be of use to you:
 * [Understanding the Parallelism of a Storm Topology](http://www.michael-noll.com/blog/2012/10/16/understanding-the-parallelism-of-a-storm-topology/)
 * [Flux Docs](http://storm.apache.org/releases/current/flux.html)
+
+## Zeppelin Notebooks
+Zeppelin notebooks can be added to `/src/main/config/zeppelin/` (and subdirectories can be
created for organization).  The placed files must be .json files and be named appropriately.
+These files must be added to the metron.spec file and the RPMs rebuilt to be available to
be loaded into Ambari.
+
+The notebook files will be found on the server in `$METRON_HOME/config/zeppelin`
+
+The Ambari Management Pack has a custom action to load these templates, ZEPPELIN_DASHBOARD_INSTALL,
that will import them into Zeppelin.
\ No newline at end of file


Mime
View raw message