Return-Path: X-Original-To: apmail-ranger-commits-archive@www.apache.org Delivered-To: apmail-ranger-commits-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 2891D1817F for ; Thu, 1 Oct 2015 18:02:52 +0000 (UTC) Received: (qmail 4134 invoked by uid 500); 1 Oct 2015 18:02:52 -0000 Delivered-To: apmail-ranger-commits-archive@ranger.apache.org Received: (qmail 4108 invoked by uid 500); 1 Oct 2015 18:02:52 -0000 Mailing-List: contact commits-help@ranger.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@ranger.incubator.apache.org Delivered-To: mailing list commits@ranger.incubator.apache.org Received: (qmail 3988 invoked by uid 99); 1 Oct 2015 18:02:52 -0000 Received: from Unknown (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Oct 2015 18:02:51 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 626B5180970 for ; Thu, 1 Oct 2015 18:02:51 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.995 X-Spam-Level: ** X-Spam-Status: No, score=2.995 tagged_above=-999 required=6.31 tests=[KAM_ASCII_DIVIDERS=0.8, KAM_LAZY_DOMAIN_SECURITY=1, KAM_MANYCOMMENTS=1.2, RP_MATCHES_RCVD=-0.006, URIBL_BLOCKED=0.001] autolearn=disabled Received: from mx1-eu-west.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id yAjLzT-dQF6U for ; Thu, 1 Oct 2015 18:02:38 +0000 (UTC) Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx1-eu-west.apache.org (ASF Mail Server at mx1-eu-west.apache.org) with SMTP id 2C6692543A for ; Thu, 1 Oct 2015 18:02:36 +0000 (UTC) Received: (qmail 2785 invoked by uid 99); 1 Oct 2015 18:02:35 -0000 Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Oct 2015 18:02:35 +0000 Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33) id 42C04E1550; Thu, 1 Oct 2015 18:02:35 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: bosco@apache.org To: commits@ranger.incubator.apache.org Date: Thu, 01 Oct 2015 18:02:37 -0000 Message-Id: <00ebfc14199f43ad888a9b20c9250175@git.apache.org> In-Reply-To: <3c86cddcebe24bbbaa52b4d95d71f9c1@git.apache.org> References: <3c86cddcebe24bbbaa52b4d95d71f9c1@git.apache.org> X-Mailer: ASF-Git Admin Mailer Subject: [3/4] incubator-ranger git commit: RANGER-266: Solr configuration and setup files and documentation http://git-wip-us.apache.org/repos/asf/incubator-ranger/blob/c5ae2edd/security-admin/contrib/solr_for_audit_setup/conf/solrconfig.xml ---------------------------------------------------------------------- diff --git a/security-admin/contrib/solr_for_audit_setup/conf/solrconfig.xml b/security-admin/contrib/solr_for_audit_setup/conf/solrconfig.xml new file mode 100644 index 0000000..0991eba --- /dev/null +++ b/security-admin/contrib/solr_for_audit_setup/conf/solrconfig.xml @@ -0,0 +1,1865 @@ + + + + + + + + + 5.0.0 + + + + + + + + + + + + + + + + + + + + + + + ${solr.data.dir:} + + + + + + + + + ${solr.hdfs.home:} + + ${solr.hdfs.confdir:} + + ${solr.hdfs.blockcache.enabled:true} + + ${solr.hdfs.blockcache.global:true} + + + + + + + + + + true + managed-schema + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + ${solr.lock.type:native} + + + + + + + + + + + + + + + + true + + + + + + + + + + + + + + + + ${solr.ulog.dir:} + + + + + ${solr.autoCommit.maxTime:15000} + false + + + + + + ${solr.autoSoftCommit.maxTime:5000} + + + + + + + + + + + + + + + + 1024 + + + + + + + + + + + + + + + + + + + + + + + + + true + + + + + + 20 + + + 200 + + + + + + + + + + + + static firstSearcher warming in solrconfig.xml + + + + + + false + + + 2 + + + + + + + + + + + + + + + + + + + + + + solr-data-config.xml + + + + + + + + explicit + 10 + text + + + + + + + + + + + + + + explicit + json + true + text + + + + + + + + true + json + true + + + + + + + + explicit + + + velocity + browse + layout + + + edismax + *:* + 10 + *,score + + + on + 1 + + + + + + + text + add-unknown-fields-to-the-schema + + + + + + + + + + + + + true + ignored_ + + + true + links + ignored_ + + + + + + + + + + + + + + + + + + + + + + + + solrpingquery + + + all + + + + + + + + + explicit + true + + + + + + + + + + + + + + + + text_general + + + + + + default + text + solr.DirectSolrSpellChecker + + internal + + 0.5 + + 2 + + 1 + + 5 + + 4 + + 0.01 + + + + + + wordbreak + solr.WordBreakSolrSpellChecker + name + true + true + 10 + + + + + + + + + + + + + + + + text + + default + wordbreak + on + true + 10 + 5 + 5 + true + true + 10 + 5 + + + spellcheck + + + + + + mySuggester + FuzzyLookupFactory + DocumentDictionaryFactory + cat + price + string + + + + + + true + 10 + + + suggest + + + + + + + + + text + true + + + tvComponent + + + + + + + lingo + + + org.carrot2.clustering.lingo.LingoClusteringAlgorithm + + + clustering/carrot2 + + + + + stc + org.carrot2.clustering.stc.STCClusteringAlgorithm + + + + + kmeans + org.carrot2.clustering.kmeans.BisectingKMeansClusteringAlgorithm + + + + + + + true + true + + name + + id + + features + + true + + + + false + + + edismax + + text^0.5 features^1.0 name^1.2 sku^1.5 id^10.0 manu^1.1 cat^1.4 + + *:* + 10 + *,score + + + clustering + + + + + + + + + + true + false + + + terms + + + + + + + + string + elevate.xml + + + + + + explicit + text + + + elevator + + + + + + + + + + + 100 + + + + + + + + 70 + + 0.5 + + [-\w ,/\n\"']{20,200} + + + + + + + ]]> + ]]> + + + + + + + + + + + + + + + + + + + + + + + + ,, + ,, + ,, + ,, + ,]]> + ]]> + + + + + + 10 + .,!? + + + + + + + WORD + + + en + US + + + + + + + + + + + + + + + + yyyy-MM-dd'T'HH:mm:ss.SSSZ + yyyy-MM-dd'T'HH:mm:ss,SSSZ + yyyy-MM-dd'T'HH:mm:ss.SSS + yyyy-MM-dd'T'HH:mm:ss,SSS + yyyy-MM-dd'T'HH:mm:ssZ + yyyy-MM-dd'T'HH:mm:ss + yyyy-MM-dd'T'HH:mmZ + yyyy-MM-dd'T'HH:mm + yyyy-MM-dd HH:mm:ss.SSSZ + yyyy-MM-dd HH:mm:ss,SSSZ + yyyy-MM-dd HH:mm:ss.SSS + yyyy-MM-dd HH:mm:ss,SSS + yyyy-MM-dd HH:mm:ssZ + yyyy-MM-dd HH:mm:ss + yyyy-MM-dd HH:mmZ + yyyy-MM-dd HH:mm + yyyy-MM-dd + + + + text_general + + java.lang.Boolean + booleans + + + java.util.Date + tdates + + + java.lang.Long + java.lang.Integer + tlongs + + + java.lang.Number + tdoubles + + + + + + + + + + + + + + + + + + + + + + + text/plain; charset=UTF-8 + + + + + ${velocity.template.base.dir:} + + + + + 5 + + + + + + + + + + + + + + + + + + *:* + + + http://git-wip-us.apache.org/repos/asf/incubator-ranger/blob/c5ae2edd/security-admin/contrib/solr_for_audit_setup/install.properties ---------------------------------------------------------------------- diff --git a/security-admin/contrib/solr_for_audit_setup/install.properties b/security-admin/contrib/solr_for_audit_setup/install.properties new file mode 100644 index 0000000..92ec1e6 --- /dev/null +++ b/security-admin/contrib/solr_for_audit_setup/install.properties @@ -0,0 +1,96 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +#Note: +#1. This file is sourced from setup.sh, so make sure there are no spaces after the "=" +#2. For variable with file path, please provide full path + +#!/bin/bash + +#JAVA_HOME to be used by Solr. Solr only support JDK 1.7 and above. If JAVA_HOME is not set +#in the env, then please set it here +#JAVA_HOME= + +#The operating system (linux) user used by Solr process. You need to run Solr as the below user +SOLR_USER=solr + +#If you want this script to isntall Solr, set the value to true. If it is already installed, then set this to false +#If it is true, then it will download and install it. +#NOTE: If you want the script to install Solr, then this script needs to be executed as root. +SOLR_INSTALL=false + +### BEGIN: if SOLR_INSTALL==true ### +#Location to download Solr. If SOLR_INSTALL is true, then SOLR_DOWNLOAD_URL is mandatory + +#For open source version, pick a mirror from below. Recommended versions are Apache Solr 5.2.1 or above +#http://lucene.apache.org/solr/mirrors-solr-latest-redir.html + +#Note: If possible, use the link from one of the mirror site +#SOLR_DOWNLOAD_URL=http://archive.apache.org/dist/lucene/solr/5.2.1/solr-5.2.1.tgz +SOLR_DOWNLOAD_URL= + +### END: if SOLR_INSTALL==true ### + +#The folder where Solr is installed. If SOLR_INSTALL=false, then Solr need to be preinstalled, else the setup will +#install at the below location +#Note: If you are using RPM from LucidWorks in HDP, then Solr is by default installed in the following location: +#SOLR_INSTALL_FOLDER=/opt/lucidworks-hdpsearch/solr +SOLR_INSTALL_FOLDER=/opt/solr + +#The location for the Solr configuration for Ranger. This script copies required configuration and +#startup scripts to the $SOLR_RANGER_HOME folder. +#NOTE: In SolrCloud mode, the data folders are under this folder. So make sure this is on seperate drive +# with enough disk space. Have 1TB free disk space on this volume. Also regularly monitor available disk space +# for this volume +#SOLR_RANGER_HOME=/opt/solr/ranger_audit_server +SOLR_RANGER_HOME=/opt/solr/ranger_audit_server + +#Port for Solr instance to be used by Ranger. +SOLR_RANGER_PORT=6083 + +#Standalone or SolrCloud. Valid values are "standalone" or "solrcloud" +SOLR_DEPLOYMENT=standalone + +#### BEGIN: if SOLR_DEPLOYMENT=standalone ########################## +#Location for the data files. Make sure it has enough disk space. Since audits records can grow dramatically, +#please have 1TB free disk space for the data folder. Also regularly monitor available disk space for this volume +SOLR_RANGER_DATA_FOLDER=/opt/solr/ranger_audit_server/data +#### END: if SOLR_DEPLOYMENT=standalone ########################## + + +#### BEGIN: if SOLR_DEPLOYMENT=solrcloud ########################## +#Comma seperated list of of zookeeper host and path. Give fully qualified domain name for the host +#SOLR_ZK=localhost:2181/ranger_audits +SOLR_ZK= +#Base URL of the Solr. Used for creating collections +SOLR_HOST_URL=http://`hostname -f`:${SOLR_RANGER_PORT} +#Number of shards +SOLR_SHARDS=1 +#Number of replication +SOLR_REPLICATION=1 +#### END: if SOLR_DEPLOYMENT=solrcloud ########################## + +#Location for the log file. Please note that "solr" or the process owner should have write permission +#to log folder +#SOLR_LOG_FOLDER=logs +SOLR_LOG_FOLDER=/var/log/solr/ranger_audits + + +#Memory for Solr. Both min and max memory to the java process are set to this value. +#Note: In production, please assign enough memory. It is recommended to have at least 2GB RAM. +# Higher the RAM, the better. Solr core can take upto 32GB. For dev test you can use 512m +#SOLR_MAX_MEM=2g +#SOLR_MAX_MEM=512m +SOLR_MAX_MEM=2g http://git-wip-us.apache.org/repos/asf/incubator-ranger/blob/c5ae2edd/security-admin/contrib/solr_for_audit_setup/resources/log4j.properties.template ---------------------------------------------------------------------- diff --git a/security-admin/contrib/solr_for_audit_setup/resources/log4j.properties.template b/security-admin/contrib/solr_for_audit_setup/resources/log4j.properties.template new file mode 100644 index 0000000..61b25ff --- /dev/null +++ b/security-admin/contrib/solr_for_audit_setup/resources/log4j.properties.template @@ -0,0 +1,39 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Logging level +solr.log=__SOLR_LOG_FOLDER__ +log4j.rootLogger=INFO, file, CONSOLE + +log4j.appender.CONSOLE=org.apache.log4j.ConsoleAppender + +log4j.appender.CONSOLE.layout=org.apache.log4j.PatternLayout +log4j.appender.CONSOLE.layout.ConversionPattern=%-4r [%t] %-5p %c %x [%X{collection} %X{shard} %X{replica} %X{core}] \u2013 %m%n + +#- size rotation with log cleanup. +log4j.appender.file=org.apache.log4j.RollingFileAppender +log4j.appender.file.MaxFileSize=4MB +log4j.appender.file.MaxBackupIndex=9 + +#- File to log to and log format +log4j.appender.file.File=${solr.log}/solr.log +log4j.appender.file.layout=org.apache.log4j.PatternLayout +log4j.appender.file.layout.ConversionPattern=%d{ISO8601} [%t] %-5p [%X{collection} %X{shard} %X{replica} %X{core}] %C (%F:%L) - %m%n + +log4j.logger.org.apache.zookeeper=WARN +log4j.logger.org.apache.hadoop=WARN + +# set to INFO to enable infostream log messages +log4j.logger.org.apache.solr.update.LoggingInfoStream=OFF