Return-Path: X-Original-To: apmail-kylin-commits-archive@minotaur.apache.org Delivered-To: apmail-kylin-commits-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 7AB721005B for ; Thu, 15 Jan 2015 12:26:43 +0000 (UTC) Received: (qmail 59167 invoked by uid 500); 15 Jan 2015 12:26:45 -0000 Delivered-To: apmail-kylin-commits-archive@kylin.apache.org Received: (qmail 59135 invoked by uid 500); 15 Jan 2015 12:26:45 -0000 Mailing-List: contact commits-help@kylin.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@kylin.incubator.apache.org Delivered-To: mailing list commits@kylin.incubator.apache.org Received: (qmail 59123 invoked by uid 99); 15 Jan 2015 12:26:45 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Jan 2015 12:26:45 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=5.0 tests=ALL_TRUSTED,T_FILL_THIS_FORM_SHORT,T_RP_MATCHES_RCVD X-Spam-Check-By: apache.org Received: from [140.211.11.3] (HELO mail.apache.org) (140.211.11.3) by apache.org (qpsmtpd/0.29) with SMTP; Thu, 15 Jan 2015 12:26:39 +0000 Received: (qmail 57274 invoked by uid 99); 15 Jan 2015 12:26:19 -0000 Received: from tyr.zones.apache.org (HELO tyr.zones.apache.org) (140.211.11.114) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Jan 2015 12:26:19 +0000 Received: by tyr.zones.apache.org (Postfix, from userid 65534) id D8389A44113; Thu, 15 Jan 2015 12:26:18 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: lukehan@apache.org To: commits@kylin.incubator.apache.org Date: Thu, 15 Jan 2015 12:26:39 -0000 Message-Id: <58b10c34afd24fab9d9cc6b95b4efd47@git.apache.org> In-Reply-To: References: X-Mailer: ASF-Git Admin Mailer Subject: [23/50] [abbrv] incubator-kylin git commit: rename deploy.sh X-Virus-Checked: Checked by ClamAV on apache.org rename deploy.sh Project: http://git-wip-us.apache.org/repos/asf/incubator-kylin/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-kylin/commit/8ee64b50 Tree: http://git-wip-us.apache.org/repos/asf/incubator-kylin/tree/8ee64b50 Diff: http://git-wip-us.apache.org/repos/asf/incubator-kylin/diff/8ee64b50 Branch: refs/heads/master Commit: 8ee64b503ed6908822bcc6d9fa5df8954aecbc40 Parents: 7901501 Author: honma Authored: Wed Jan 14 14:21:34 2015 +0800 Committer: honma Committed: Wed Jan 14 14:21:34 2015 +0800 ---------------------------------------------------------------------- deploy.sh | 206 ------------------------- query/src/test/resources/query/.gitignore | 1 + sandbox_deploy.sh | 206 +++++++++++++++++++++++++ 3 files changed, 207 insertions(+), 206 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/8ee64b50/deploy.sh ---------------------------------------------------------------------- diff --git a/deploy.sh b/deploy.sh deleted file mode 100755 index 4dd8355..0000000 --- a/deploy.sh +++ /dev/null @@ -1,206 +0,0 @@ -#!/usr/bin/env bash - -set -o pipefail # trace ERR through pipes -set -o errtrace # trace ERR through 'time command' and other functions - -function error() { -SCRIPT="$0" # script name -LASTLINE="$1" # line of error occurrence -LASTERR="$2" # error code -echo "ERROR exit from ${SCRIPT} : line ${LASTLINE} with exit code ${LASTERR}" -exit 1 -} - -trap 'error ${LINENO} ${?}' ERR - -echo "" -echo "Welcome to use Kylin-Deploy script" -echo "This script will help you:" -echo "1. Check environment" -echo "2. Build Kylin artifacts" -echo "3. Prepare test cube related data" -echo "4. Lauch a web service to build cube and query with (at http://localhost:7070)" -echo "Please make sure you are running this script on a hadoop CLI machine, and you have enough permissions." -echo "Also, We assume you have installed: JAVA, TOMCAT, NPM and MAVEN." -echo "[Warning] The installation may break existing tomcat applications on this CLI" -echo "" - - -[[ "$SILENT" ]] || ( read -p "Are you sure you want to proceed?(press Y or y to confirm) " -n 1 -r -echo # (optional) move to a new line -if [[ ! $REPLY =~ ^[Yy]$ ]] -then - echo "Not going to proceed, quit without finishing! You can rerun the script to have another try." - exit 1 -fi ) - -echo "Checking JAVA status..." - -if [ -z "$JAVA_HOME" ] -then - echo "Please set JAVA_HOME so that Kylin-Deploy can proceed" - exit 1 -else - echo "JAVA_HOME is set to $JAVA_HOME" -fi - -if [ -d "$JAVA_HOME" ] -then - echo "$JAVA_HOME exists" -else - echo " $JAVA_HOME does not exist or is not a directory." - exit 1 -fi - -echo "Checking tomcat status..." - -if [ -z "$CATALINA_HOME" ] -then - echo "Please set CATALINA_HOME so that Kylin-Deploy knows where to start tomcat" - exit 1 -else - echo "CATALINA_HOME is set to $CATALINA_HOME" -fi - -if [ -d "$CATALINA_HOME" ] -then - echo "$CATALINA_HOME exists" -else - echo " $CATALINA_HOME does not exist or is not a directory." - exit 1 -fi - -echo "Checking maven..." - -if [ -z "$(command -v mvn)" ] -then - echo "Please install maven first so that Kylin-Deploy can proceed" - exit 1 -else - echo "maven check passed" -fi - -echo "Checking npm..." - -if [ -z "$(command -v npm)" ] -then - echo "Please install npm first so that Kylin-Deploy can proceed" - exit 1 -else - echo "npm check passed" -fi - -KYLIN_HOME="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )" -echo "Kylin home folder path is $KYLIN_HOME" -cd $KYLIN_HOME - - -echo "Building and packaging..." -source ./package.sh - -echo "retrieving classpath..." -cd $KYLIN_HOME/job/target -JOB_JAR_NAME="kylin-job-latest.jar" -#generate config variables -hbase org.apache.hadoop.util.RunJar $JOB_JAR_NAME com.kylinolap.job.deployment.HbaseConfigPrinter /tmp/kylin_retrieve.sh -#load config variables -source /tmp/kylin_retrieve.sh - -cd $KYLIN_HOME -mkdir -p /etc/kylin - -HOSTNAME=`hostname` -CLI_HOSTNAME_DEFAULT="kylin.job.remote.cli.hostname=sandbox.hortonworks.com" -CLI_PASSWORD_DEFAULT="kylin.job.remote.cli.password=hadoop" -METADATA_URL_DEFAULT="kylin.metadata.url=kylin_metadata_qa@hbase:sandbox.hortonworks.com:2181:/hbase-unsecure" -STORAGE_URL_DEFAULT="kylin.storage.url=hbase:sandbox.hortonworks.com:2181:/hbase-unsecure" -CHECK_URL_DEFAULT="kylin.job.yarn.app.rest.check.status.url=http://sandbox" - - -NEW_CLI_HOSTNAME_PREFIX="kylin.job.remote.cli.hostname=" -NEW_CLI_PASSWORD_PREFIX="kylin.job.remote.cli.password=" -NEW_METADATA_URL_PREFIX="kylin.metadata.url=kylin_metadata_qa@hbase:" -NEW_STORAGE_URL_PREFIX="kylin.storage.url=hbase:" -NEW_CHECK_URL_PREFIX="kylin.job.yarn.app.rest.check.status.url=http://" - -KYLIN_ZOOKEEPER_URL=${KYLIN_ZOOKEEPER_QUORUM}:${KYLIN_ZOOKEEPER_CLIENT_PORT}:${KYLIN_ZOOKEEPER_ZNODE_PARENT} - -echo "Kylin install script requires root password for ${HOSTNAME}" -echo "(The default root password for hortonworks VM is hadoop, and for cloudera VM is cloudera)" - -[[ "$SILENT" ]] || read -s -p "Enter Password for root: " ROOTPASS - -#deploy kylin.properties to /etc/kylin -cat examples/test_case_data/sandbox/kylin.properties | \ - sed -e "s,${CHECK_URL_DEFAULT},${NEW_CHECK_URL_PREFIX}${HOSTNAME}," | \ - sed -e "s,${CLI_HOSTNAME_DEFAULT},${NEW_CLI_HOSTNAME_PREFIX}${HOSTNAME}," | \ - sed -e "s,${CLI_PASSWORD_DEFAULT},${NEW_CLI_PASSWORD_PREFIX}${ROOTPASS}," | \ - sed -e "s,${METADATA_URL_DEFAULT},${NEW_METADATA_URL_PREFIX}${KYLIN_ZOOKEEPER_URL}," | \ - sed -e "s,${STORAGE_URL_DEFAULT},${NEW_STORAGE_URL_PREFIX}${KYLIN_ZOOKEEPER_URL}," > /etc/kylin/kylin.properties - - -echo "a copy of kylin config is generated at /etc/kylin/kylin.properties:" -echo "===================================================================" -cat /etc/kylin/kylin.properties -echo "" -echo "===================================================================" -echo "" - -[[ "$SILENT" ]] || ( read -p "please ensure the CLI address/username/password is correct, and press y to proceed: " -n 1 -r -echo # (optional) move to a new line -if [[ ! $REPLY =~ ^[Yy]$ ]] -then - echo "Not going to proceed, quit without finishing! You can rerun the script to have another try." - exit 1 -fi ) - -# 1. generate synthetic fact table(test_kylin_fact) data and dump it into hive -# 2. create empty cubes on these data, ready to be built -cd $KYLIN_HOME -mvn test -Dtest=com.kylinolap.job.SampleCubeSetupTest -DfailIfNoTests=false - -sudo -i "${CATALINA_HOME}/bin/shutdown.sh" || true # avoid trapping -cd $KYLIN_HOME/server/target -WAR_NAME="kylin.war" -rm -rf $CATALINA_HOME/webapps/kylin -rm -f $CATALINA_HOME/webapps/$WAR_NAME -cp $KYLIN_HOME/server/target/$WAR_NAME $CATALINA_HOME/webapps/ -cd $CATALINA_HOME/webapps; -chmod 644 $WAR_NAME; -echo "REST service deployed" - -rm -rf /var/www/html/kylin -mkdir -p /var/www/html/kylin -cd $KYLIN_HOME/ -tar -xf webapp/dist/Web.tar -C /var/www/html/kylin -echo "Web deployed" - -cd $KYLIN_HOME/ -#deploy setenv.sh -rm -rf $CATALINA_HOME/bin/setenv.sh -echo JAVA_OPTS=\"-Djava.library.path=${KYLIN_LD_LIBRARY_PATH}\" >> ${CATALINA_HOME}/bin/setenv.sh -echo CATALINA_OPTS=\"-Dorg.apache.tomcat.util.buf.UDecoder.ALLOW_ENCODED_SLASH=true -Dorg.apache.catalina.connector.CoyoteAdapter.ALLOW_BACKSLASH=true -Dspring.profiles.active=sandbox \" >> ${CATALINA_HOME}/bin/setenv.sh -echo CLASSPATH=\"${CATALINA_HOME}/lib/*:${KYLIN_HBASE_CLASSPATH}:/etc/kylin\" >> ${CATALINA_HOME}/bin/setenv.sh -echo "setenv.sh created" - -#deploy server.xml -rm -rf ${CATALINA_HOME}/conf/server.xml -cp deploy/server.xml ${CATALINA_HOME}/conf/server.xml -echo "server.xml copied" - -#deploy web.xml -rm -rf ${CATALINA_HOME}/conf/web.xml -cp deploy/web.xml ${CATALINA_HOME}/conf/web.xml -echo "web.xml copied" - -echo "Tomcat ready" - -# redeploy coprocessor -#hbase org.apache.hadoop.util.RunJar /usr/lib/kylin/kylin-job-latest.jar com.kylinolap.job.tools.DeployCoprocessorCLI /usr/lib/kylin/kylin-coprocessor-latest.jar - - -sudo -i "${CATALINA_HOME}/bin/startup.sh" - - -echo "Kylin-Deploy Success!" -echo "Please visit http://:7070 to play with the cubes! (Useranme: ADMIN, Password: KYLIN)" http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/8ee64b50/query/src/test/resources/query/.gitignore ---------------------------------------------------------------------- diff --git a/query/src/test/resources/query/.gitignore b/query/src/test/resources/query/.gitignore new file mode 100644 index 0000000..cb98934 --- /dev/null +++ b/query/src/test/resources/query/.gitignore @@ -0,0 +1 @@ +/sample.sql http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/8ee64b50/sandbox_deploy.sh ---------------------------------------------------------------------- diff --git a/sandbox_deploy.sh b/sandbox_deploy.sh new file mode 100755 index 0000000..4dd8355 --- /dev/null +++ b/sandbox_deploy.sh @@ -0,0 +1,206 @@ +#!/usr/bin/env bash + +set -o pipefail # trace ERR through pipes +set -o errtrace # trace ERR through 'time command' and other functions + +function error() { +SCRIPT="$0" # script name +LASTLINE="$1" # line of error occurrence +LASTERR="$2" # error code +echo "ERROR exit from ${SCRIPT} : line ${LASTLINE} with exit code ${LASTERR}" +exit 1 +} + +trap 'error ${LINENO} ${?}' ERR + +echo "" +echo "Welcome to use Kylin-Deploy script" +echo "This script will help you:" +echo "1. Check environment" +echo "2. Build Kylin artifacts" +echo "3. Prepare test cube related data" +echo "4. Lauch a web service to build cube and query with (at http://localhost:7070)" +echo "Please make sure you are running this script on a hadoop CLI machine, and you have enough permissions." +echo "Also, We assume you have installed: JAVA, TOMCAT, NPM and MAVEN." +echo "[Warning] The installation may break existing tomcat applications on this CLI" +echo "" + + +[[ "$SILENT" ]] || ( read -p "Are you sure you want to proceed?(press Y or y to confirm) " -n 1 -r +echo # (optional) move to a new line +if [[ ! $REPLY =~ ^[Yy]$ ]] +then + echo "Not going to proceed, quit without finishing! You can rerun the script to have another try." + exit 1 +fi ) + +echo "Checking JAVA status..." + +if [ -z "$JAVA_HOME" ] +then + echo "Please set JAVA_HOME so that Kylin-Deploy can proceed" + exit 1 +else + echo "JAVA_HOME is set to $JAVA_HOME" +fi + +if [ -d "$JAVA_HOME" ] +then + echo "$JAVA_HOME exists" +else + echo " $JAVA_HOME does not exist or is not a directory." + exit 1 +fi + +echo "Checking tomcat status..." + +if [ -z "$CATALINA_HOME" ] +then + echo "Please set CATALINA_HOME so that Kylin-Deploy knows where to start tomcat" + exit 1 +else + echo "CATALINA_HOME is set to $CATALINA_HOME" +fi + +if [ -d "$CATALINA_HOME" ] +then + echo "$CATALINA_HOME exists" +else + echo " $CATALINA_HOME does not exist or is not a directory." + exit 1 +fi + +echo "Checking maven..." + +if [ -z "$(command -v mvn)" ] +then + echo "Please install maven first so that Kylin-Deploy can proceed" + exit 1 +else + echo "maven check passed" +fi + +echo "Checking npm..." + +if [ -z "$(command -v npm)" ] +then + echo "Please install npm first so that Kylin-Deploy can proceed" + exit 1 +else + echo "npm check passed" +fi + +KYLIN_HOME="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )" +echo "Kylin home folder path is $KYLIN_HOME" +cd $KYLIN_HOME + + +echo "Building and packaging..." +source ./package.sh + +echo "retrieving classpath..." +cd $KYLIN_HOME/job/target +JOB_JAR_NAME="kylin-job-latest.jar" +#generate config variables +hbase org.apache.hadoop.util.RunJar $JOB_JAR_NAME com.kylinolap.job.deployment.HbaseConfigPrinter /tmp/kylin_retrieve.sh +#load config variables +source /tmp/kylin_retrieve.sh + +cd $KYLIN_HOME +mkdir -p /etc/kylin + +HOSTNAME=`hostname` +CLI_HOSTNAME_DEFAULT="kylin.job.remote.cli.hostname=sandbox.hortonworks.com" +CLI_PASSWORD_DEFAULT="kylin.job.remote.cli.password=hadoop" +METADATA_URL_DEFAULT="kylin.metadata.url=kylin_metadata_qa@hbase:sandbox.hortonworks.com:2181:/hbase-unsecure" +STORAGE_URL_DEFAULT="kylin.storage.url=hbase:sandbox.hortonworks.com:2181:/hbase-unsecure" +CHECK_URL_DEFAULT="kylin.job.yarn.app.rest.check.status.url=http://sandbox" + + +NEW_CLI_HOSTNAME_PREFIX="kylin.job.remote.cli.hostname=" +NEW_CLI_PASSWORD_PREFIX="kylin.job.remote.cli.password=" +NEW_METADATA_URL_PREFIX="kylin.metadata.url=kylin_metadata_qa@hbase:" +NEW_STORAGE_URL_PREFIX="kylin.storage.url=hbase:" +NEW_CHECK_URL_PREFIX="kylin.job.yarn.app.rest.check.status.url=http://" + +KYLIN_ZOOKEEPER_URL=${KYLIN_ZOOKEEPER_QUORUM}:${KYLIN_ZOOKEEPER_CLIENT_PORT}:${KYLIN_ZOOKEEPER_ZNODE_PARENT} + +echo "Kylin install script requires root password for ${HOSTNAME}" +echo "(The default root password for hortonworks VM is hadoop, and for cloudera VM is cloudera)" + +[[ "$SILENT" ]] || read -s -p "Enter Password for root: " ROOTPASS + +#deploy kylin.properties to /etc/kylin +cat examples/test_case_data/sandbox/kylin.properties | \ + sed -e "s,${CHECK_URL_DEFAULT},${NEW_CHECK_URL_PREFIX}${HOSTNAME}," | \ + sed -e "s,${CLI_HOSTNAME_DEFAULT},${NEW_CLI_HOSTNAME_PREFIX}${HOSTNAME}," | \ + sed -e "s,${CLI_PASSWORD_DEFAULT},${NEW_CLI_PASSWORD_PREFIX}${ROOTPASS}," | \ + sed -e "s,${METADATA_URL_DEFAULT},${NEW_METADATA_URL_PREFIX}${KYLIN_ZOOKEEPER_URL}," | \ + sed -e "s,${STORAGE_URL_DEFAULT},${NEW_STORAGE_URL_PREFIX}${KYLIN_ZOOKEEPER_URL}," > /etc/kylin/kylin.properties + + +echo "a copy of kylin config is generated at /etc/kylin/kylin.properties:" +echo "===================================================================" +cat /etc/kylin/kylin.properties +echo "" +echo "===================================================================" +echo "" + +[[ "$SILENT" ]] || ( read -p "please ensure the CLI address/username/password is correct, and press y to proceed: " -n 1 -r +echo # (optional) move to a new line +if [[ ! $REPLY =~ ^[Yy]$ ]] +then + echo "Not going to proceed, quit without finishing! You can rerun the script to have another try." + exit 1 +fi ) + +# 1. generate synthetic fact table(test_kylin_fact) data and dump it into hive +# 2. create empty cubes on these data, ready to be built +cd $KYLIN_HOME +mvn test -Dtest=com.kylinolap.job.SampleCubeSetupTest -DfailIfNoTests=false + +sudo -i "${CATALINA_HOME}/bin/shutdown.sh" || true # avoid trapping +cd $KYLIN_HOME/server/target +WAR_NAME="kylin.war" +rm -rf $CATALINA_HOME/webapps/kylin +rm -f $CATALINA_HOME/webapps/$WAR_NAME +cp $KYLIN_HOME/server/target/$WAR_NAME $CATALINA_HOME/webapps/ +cd $CATALINA_HOME/webapps; +chmod 644 $WAR_NAME; +echo "REST service deployed" + +rm -rf /var/www/html/kylin +mkdir -p /var/www/html/kylin +cd $KYLIN_HOME/ +tar -xf webapp/dist/Web.tar -C /var/www/html/kylin +echo "Web deployed" + +cd $KYLIN_HOME/ +#deploy setenv.sh +rm -rf $CATALINA_HOME/bin/setenv.sh +echo JAVA_OPTS=\"-Djava.library.path=${KYLIN_LD_LIBRARY_PATH}\" >> ${CATALINA_HOME}/bin/setenv.sh +echo CATALINA_OPTS=\"-Dorg.apache.tomcat.util.buf.UDecoder.ALLOW_ENCODED_SLASH=true -Dorg.apache.catalina.connector.CoyoteAdapter.ALLOW_BACKSLASH=true -Dspring.profiles.active=sandbox \" >> ${CATALINA_HOME}/bin/setenv.sh +echo CLASSPATH=\"${CATALINA_HOME}/lib/*:${KYLIN_HBASE_CLASSPATH}:/etc/kylin\" >> ${CATALINA_HOME}/bin/setenv.sh +echo "setenv.sh created" + +#deploy server.xml +rm -rf ${CATALINA_HOME}/conf/server.xml +cp deploy/server.xml ${CATALINA_HOME}/conf/server.xml +echo "server.xml copied" + +#deploy web.xml +rm -rf ${CATALINA_HOME}/conf/web.xml +cp deploy/web.xml ${CATALINA_HOME}/conf/web.xml +echo "web.xml copied" + +echo "Tomcat ready" + +# redeploy coprocessor +#hbase org.apache.hadoop.util.RunJar /usr/lib/kylin/kylin-job-latest.jar com.kylinolap.job.tools.DeployCoprocessorCLI /usr/lib/kylin/kylin-coprocessor-latest.jar + + +sudo -i "${CATALINA_HOME}/bin/startup.sh" + + +echo "Kylin-Deploy Success!" +echo "Please visit http://:7070 to play with the cubes! (Useranme: ADMIN, Password: KYLIN)"