Return-Path: X-Original-To: apmail-drill-commits-archive@www.apache.org Delivered-To: apmail-drill-commits-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id CE11B180C5 for ; Thu, 7 Jan 2016 00:40:58 +0000 (UTC) Received: (qmail 80015 invoked by uid 500); 7 Jan 2016 00:40:58 -0000 Delivered-To: apmail-drill-commits-archive@drill.apache.org Received: (qmail 79972 invoked by uid 500); 7 Jan 2016 00:40:58 -0000 Mailing-List: contact commits-help@drill.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: commits@drill.apache.org Delivered-To: mailing list commits@drill.apache.org Received: (qmail 79963 invoked by uid 99); 7 Jan 2016 00:40:58 -0000 Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 07 Jan 2016 00:40:58 +0000 Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33) id 5FF9ADFDC7; Thu, 7 Jan 2016 00:40:58 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: krishahn@apache.org To: commits@drill.apache.org Message-Id: <87972d11a59640ef9517c8abb88f509b@git.apache.org> X-Mailer: ASF-Git Admin Mailer Subject: drill git commit: redirect example Date: Thu, 7 Jan 2016 00:40:58 +0000 (UTC) Repository: drill Updated Branches: refs/heads/gh-pages 263c11334 -> 3f0a7d609 redirect example add config for redirect add instructions for redirect MD-643 Rahul's review Project: http://git-wip-us.apache.org/repos/asf/drill/repo Commit: http://git-wip-us.apache.org/repos/asf/drill/commit/3f0a7d60 Tree: http://git-wip-us.apache.org/repos/asf/drill/tree/3f0a7d60 Diff: http://git-wip-us.apache.org/repos/asf/drill/diff/3f0a7d60 Branch: refs/heads/gh-pages Commit: 3f0a7d609af0ef2848f102ca4e5d9f3fd66b143c Parents: 263c113 Author: Kris Hahn Authored: Wed Jan 6 15:45:23 2016 -0800 Committer: Kris Hahn Committed: Wed Jan 6 16:40:26 2016 -0800 ---------------------------------------------------------------------- README.md | 22 ++++++- _config.yml | 3 + .../plugins/070-hive-storage-plugin.md | 61 ++++++++++++++------ _docs/sample-datasets/001-aol.md | 6 +- 4 files changed, 71 insertions(+), 21 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/drill/blob/3f0a7d60/README.md ---------------------------------------------------------------------- diff --git a/README.md b/README.md index 49df1fa..2d3f0b7 100644 --- a/README.md +++ b/README.md @@ -11,6 +11,8 @@ jekyll serve --config _config.yml,_config-prod.yml ``` Note that you can skip the first two commands (and only run `jekyll serve`) if you haven't changed the title or path of any of the documentation pages. +## One Time Setup for Last-Modified-Date + To automatically add the last-modified-on date, a one-time local setup is required: 1. In your cloned directory of Drill, in drill/.git/hooks, create a file named pre-commit (no extension) that contains this script: @@ -28,7 +30,7 @@ To automatically add the last-modified-on date, a one-time local setup is requir chmod +x pre-commit -In addition to the title: and parent:, you now need to add date: to the front matter of any file you create. For example: +On any page you create, in addition to the title: and parent:, you now need to add date: to the front matter of any file you create. For example: --- title: "Configuring Multitenant Resources" @@ -36,7 +38,23 @@ In addition to the title: and parent:, you now need to add date: to the front ma date: --- -Do not fill in or alter the date: field. Jekyll and git take care of that when you commit the file. +Do not fill in or alter the date: field. Jekyll and git take care of that when you commit the file. + +## One Time Setup for Redirecting gh-pages + +Locally install the jekyll-redirect-from gem: + + gem install jekyll-redirect-from + +On any page you want to redirect, add the redirect_to: and the URL to the front matter. For example: + + --- + title: "Configuring Multitenant Resources" + parent: "Configuring a Multitenant Cluster" + date: + redirect_to: + - http:// + --- # Compiling the Website http://git-wip-us.apache.org/repos/asf/drill/blob/3f0a7d60/_config.yml ---------------------------------------------------------------------- diff --git a/_config.yml b/_config.yml index f6ffe66..9a6aa16 100644 --- a/_config.yml +++ b/_config.yml @@ -30,3 +30,6 @@ defaults: sass: style: :compressed + +gems: + - jekyll-redirect-from http://git-wip-us.apache.org/repos/asf/drill/blob/3f0a7d60/_docs/connect-a-data-source/plugins/070-hive-storage-plugin.md ---------------------------------------------------------------------- diff --git a/_docs/connect-a-data-source/plugins/070-hive-storage-plugin.md b/_docs/connect-a-data-source/plugins/070-hive-storage-plugin.md index 32643e5..4c6b699 100644 --- a/_docs/connect-a-data-source/plugins/070-hive-storage-plugin.md +++ b/_docs/connect-a-data-source/plugins/070-hive-storage-plugin.md @@ -1,6 +1,6 @@ --- title: "Hive Storage Plugin" -date: +date: 2016-01-07 parent: "Connect a Data Source" --- Drill 1.1 and later supports Hive 1.0. To access Hive tables @@ -8,22 +8,12 @@ using custom SerDes or InputFormat/OutputFormat, all nodes running Drillbits must have the SerDes or InputFormat/OutputFormat `JAR` files in the `/jars/3rdparty` folder. -## Hive Remote Metastore Configuration +You can run Hive queries in the following ways by configuring the Hive storage plugin as described in this document: -The Hive metastore runs as a separate service outside -of Hive. Drill communicates with the Hive metastore through Thrift. The -metastore service communicates with the Hive database over JDBC. Point Drill -to the Hive metastore service address, and provide the connection parameters -in a Hive storage plugin configuration to configure a connection to Drill. - -{% include startnote.html %}Verify that the Hive metastore service is running before you register the Hive metastore.{% include endnote.html %} +* [Connect Drill to the Hive remote metastore]({{site.baseurl}}/docs/hive-storage-plugin/#connect-drill-to-the-hive-remote-metastore-directly) +* [Connect to the Hive embedded metastore]({{site.baseurl}}/docs/hive-storage-plugin/#connect-to-the-hive-embedded-metastore) -To register a remote Hive metastore with Drill: - -1. Issue the following command to start the Hive metastore service on the system specified in the `hive.metastore.uris`: - `hive --service metastore` -2. In the [Drill Web Console]({{ site.baseurl }}/docs/plugin-configuration-basics/#using-the-drill-web-console), select the **Storage** tab. -3. In the list of disabled storage plugins in the Drill Web Console, click **Update** next to `hive`. The Hive storage plugin configuration appears: +You update the Hive storage plugin by selecting the **Storage tab** on the [Drill Web Console]({{ site.baseurl }}/docs/plugin-configuration-basics/#using-the-drill-web-console). From the list of disabled storage plugins in the Drill Web Console, click **Update** next to `hive`. The default Hive storage plugin configuration appears as follows: { "type": "hive", @@ -36,6 +26,27 @@ To register a remote Hive metastore with Drill: "hive.metastore.sasl.enabled": "false" } } + +## Connect Drill to the Hive Remote Metastore + +The Hive metastore runs as a separate service outside +of Hive. Drill can query the Hive metastore through Thrift. The +metastore service communicates with the Hive database over JDBC. + +Follow the steps in the next section to point Drill +to the Hive metastore service address. Provide the connection parameters +in a Hive storage plugin configuration to configure a connection to Drill. At this point, if you query data sources that Drill supports other than HBase (or MapR), you are finished configuring the Hive storage plugin. If you query HBase using Hive, you need to add ZooKeeper quorum and port properties. The HBaseStorageHandler requires these properties. Drill discovers HBase services using these properties. If you use the HBase storage plugin, the ZooKeeper quorum and port properties in the Hive storage plugin are the same as those in the HBase storage plugin, assuming you want to use the same HBase database. + +{% include startnote.html %}Verify that the Hive metastore service is running before you register the Hive metastore.{% include endnote.html %} + +### Hive Remote Metastore Configuration + +To connect Drill to a remote Hive metastore: + +1. Issue the following command to start the Hive metastore service on the system specified in the `hive.metastore.uris`: + `hive --service metastore` +2. In the [Drill Web Console]({{ site.baseurl }}/docs/plugin-configuration-basics/#using-the-drill-web-console), select the **Storage** tab. +3. In the list of disabled storage plugins in the Drill Web Console, click **Update** next to `hive`. 4. In the configuration window, add the `Thrift URI` and port to `hive.metastore.uris`. For example: ... @@ -55,15 +66,31 @@ To register a remote Hive metastore with Drill: } } -6. Click **Enable**. +6. If you do not query HBase, skip this step. If you query HBase, in the configuration window, add the names of the ZooKeeper quorum hosts and the ZooKeeper port, for example 2181. -## Hive Embedded Metastore Configuration + { + "type": "hive", + "enabled": false, + "configProps": { + . + . + . + "hbase.zookeeper.quorum": "zkhost1,zkhost2,zkhost3", + "hbase.zookeeper.property.clientPort:" "2181" + } + } + +7. Click **Enable**. + +## Connect to the Hive embedded metastore The Hive metastore configuration is embedded within the Drill process. Configure an embedded metastore only in a cluster that runs a single Drillbit and only for testing purposes. Do not embed the Hive metastore in production systems. Provide the metastore database configuration settings in the Drill Web Console. Before you configure an embedded Hive metastore, verify that the driver you use to connect to the Hive metastore is in the Drill classpath located in `//lib/.` If the driver is not there, copy the driver to `//lib` on the Drill node. For more information about storage types and configurations, refer to ["Hive Metastore Administration"](https://cwiki.apache.org/confluence/display/Hive/AdminManual+MetastoreAdmin). +### Hive Embedded Metastore Configuration + To configure an embedded Hive metastore, complete the following steps: http://git-wip-us.apache.org/repos/asf/drill/blob/3f0a7d60/_docs/sample-datasets/001-aol.md ---------------------------------------------------------------------- diff --git a/_docs/sample-datasets/001-aol.md b/_docs/sample-datasets/001-aol.md index 4948cbf..d9b8eed 100644 --- a/_docs/sample-datasets/001-aol.md +++ b/_docs/sample-datasets/001-aol.md @@ -1,6 +1,8 @@ --- title: "AOL Search" -date: +date: 2016-01-06 +redirect_to: + - http://gregsadetsky.com/aol-data parent: "Sample Datasets" --- ## Quick Stats @@ -12,7 +14,7 @@ a collection of real query log data that is based on real users. The dataset consists of 20M Web queries from 650k users over a period of three months, 440MB in total and available [for -download](http://zola.di.unipi.it/smalltext/datasets.html). The format used in +download](http://gregsadetsky.com/aol-data/). The format used in the dataset is: AnonID, Query, QueryTime, ItemRank, ClickURL