Drill 1.3 Released

Drill 1.2 was released on October 16, 2015.

Drill 1.3 was released on November 23, 2015.

Additional resources for Drill 1.2:

Additional resources for Drill 1.3:

If you're looking for an older release, see the release notes. @@ -169,6 +169,6 @@ m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m) ga('create', 'UA-53379651-1', 'auto'); ga('send', 'pageview'); - + http://git-wip-us.apache.org/repos/asf/drill-site/blob/9090a50b/faq/index.html ---------------------------------------------------------------------- diff --git a/faq/index.html b/faq/index.html index 9eaefb7..db5a1c7 100644 --- a/faq/index.html +++ b/faq/index.html @@ -310,6 +310,6 @@ m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m) ga('create', 'UA-53379651-1', 'auto'); ga('send', 'pageview'); - + http://git-wip-us.apache.org/repos/asf/drill-site/blob/9090a50b/feed.xml ---------------------------------------------------------------------- diff --git a/feed.xml b/feed.xml index 507a142..5124240 100644 --- a/feed.xml +++ b/feed.xml @@ -6,9 +6,78 @@ / - Wed, 18 Nov 2015 11:29:18 -0800 - Wed, 18 Nov 2015 11:29:18 -0800 - Jekyll v2.5.2 + Mon, 23 Nov 2015 13:49:19 -0800 + Mon, 23 Nov 2015 13:49:19 -0800 + Jekyll v2.5.3 + + + Drill 1.3 Released + <p>Today I'm happy to announce the availability of the Drill 1.3 release. This release addresses <a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12313820&version=12332946">58 JIRAs</a> on top of the 1.2 release. Highlights include:</p> + +<h2 id="enhanced-amazon-s3-support">Enhanced Amazon S3 Support</h2> + +<p>Drill 1.3 utilizes a new library, called s3a, for reading data from S3. The s3a library includes improvements over the previous s3n library, such as higher performance and the ability to read large files (over 5GB).</p> + +<p>In addition to the new s3a library, Drill 1.3 makes it easier to set up your AWS credentials. Simply edit the file <code>conf/core-site.xml</code> in the Drill install directory. For more information, check out the <a href="/docs/s3-storage-plugin/">step-by-step instructions</a> in the documentation.</p> + +<h2 id="heterogeneous-types">Heterogeneous Types</h2> + +<p>Drill 1.3 includes support for mixed-type columns, often found in systems like MongoDB and file formats like JSON. For example, Drill can now columns that evolve from one data type to another over time.</p> + +<p>Drill 1.3 provides a collection of functions that enable you to test the data type of a value. For example, if you have a column that has both lists (arrays) and numbers, you can use the following query to extract the first element from the array values:</p> + +<p><code>SELECT 1 + CASE WHEN is_list(a) THEN a[0] ELSE a END FROM table;</code></p> + +<h2 id="text-file-headers">Text File Headers</h2> + +<p>Drill is now able to parse the header row in a text file (CSV, TSV, etc.). Prior to Drill 1.3, data had to be accessed through the <code>columns</code> array:</p> +<div class="highlight"><pre><code class="language-text" data-lang="text">SELECT columns[0], columns[1] FROM dfs.`/path/to/users.csv` +</code></pre></div> +<p>With Drill 1.3, you can use the actual column names in the CSV file:</p> +<div class="highlight"><pre><code class="language-text" data-lang="text">SELECT name, address FROM dfs.`/path/to/users.csv` +</code></pre></div> +<p>Enabling header parsing is as simple as setting the <code>extractHeader</code> parameter in the storage plugin configuration for the desired file extensions. For more information, check out <a href="/docs/text-files-csv-tsv-psv/">the documentation</a>.</p> + +<h2 id="sequence-files">Sequence Files</h2> + +<p>Drill now <a href="/docs/querying-sequence-files/">supports sequence files</a>, a format commonly used in the Hadoop ecosystem. A sequence file contains a series of keys and values, and querying it with Drill is as easy as querying any other self-describing format:</p> +<div class="highlight"><pre><code class="language-text" data-lang="text">SELECT * +FROM dfs.tmp.`simple.seq` +LIMIT 1; ++--------------+---------------+ +| binary_key | binary_value | ++--------------+---------------+ +| [B@70828f46 | [B@b8c765f | ++--------------+---------------+ +</code></pre></div> +<p>Drill's <code>CONVERT_FROM</code> function makes it easy to decode the binary values:</p> +<div class="highlight"><pre><code class="language-text" data-lang="text">SELECT CONVERT_FROM(binary_key, 'UTF8'), CONVERT_FROM(binary_value, 'UTF8') +FROM dfs.tmp.`simple.seq` +LIMIT 1 +; ++-----------+-------------+ +| EXPR$0 | EXPR$1 | ++-----------+-------------+ +| key0 | value0 | ++-----------+-------------+ +</code></pre></div> +<h2 id="many-more-fixes">Many More Fixes</h2> + +<p>Drill 1.3 includes many other improvements, including enhancements related to querying Hive tables, MongoDB collections and Avro files. Check out the complete list of <a href="https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12313820&version=12332946">fixes and enhancements</a> for more information.</p> + +<p>Download the <a href="https://drill.apache.org/download/">Drill 1.3 release</a> now and let us know your thoughts.</p> + +<p>Drill On! +Jacques Nadeau</p> + + Mon, 23 Nov 2015 00:00:00 -0800 + /blog/2015/11/23/drill-1.3-released/ + /blog/2015/11/23/drill-1.3-released/ + + + blog + + Drill 1.2 Released @@ -66,8 +135,9 @@ Jacques Nadeau</p> Drill Tutorial at NoSQL Now! 2015 - <p><script type="text/javascript" src="//addthisevent.com/libs/1.5.8/ate.min.js"></script> -<a href="/blog/2015/07/23/drill-tutorial-at-nosql-now-2015/" title="Add to Calendar" class="addthisevent"> + <script type="text/javascript" src="//addthisevent.com/libs/1.5.8/ate.min.js"></script> + +<p><a href="/blog/2015/07/23/drill-tutorial-at-nosql-now-2015/" title="Add to Calendar" class="addthisevent"> Add to Calendar <span class="_start">08-20-2015 13:00:00</span> <span class="_end">08-20-2014 16:15:00</span> @@ -372,70 +442,5 @@ Tomer Shiran</p> - - Schema-free JSON Data Infrastructure - <p>JSON has emerged in recent years as the de-facto standard data exchange format. It is being used everywhere. Front-end Web applications use JSON to maintain data and communicate with back-end applications. Web APIs are JSON-based (eg, <a href="https://dev.twitter.com/rest/public">Twitter REST APIs</a>, <a href="http://developers.marketo.com/documentation/rest/">Marketo REST APIs</a>, <a href="https://developer.github.com/v3/">GitHub API</a>). It's the format of choice for public datasets, operational log files and more.</p> - -<h1 id="why-is-json-a-convenient-data-exchange-format?">Why is JSON a Convenient Data Exchange Format?</h1> - -<p>While I won't dive into the historical roots of JSON (JavaScript Object Notation, <a href="http://en.wikipedia.org/wiki/JSON#JavaScript_eval.28.29"><code>eval()</code></a>, etc.), I do want to highlight several attributes of JSON that make it a convenient data exchange format:</p> - -<ul> -<li><strong>JSON is self-describing</strong>. You can look at a JSON document and understand what it represents. The field names are included in the document. You don't need an external schema or definition to interpret JSON-encoded data. This makes life easier for anyone who wants to deal with the data, and it also means that a collection of JSON documents represents what many people call a "schema-less dataset" (where structure can evolve, and different records can have different fields).</li> -<li><strong>JSON is simple</strong>. Other self-describing formats such as XML are much more complicated. A JSON document is made up of arrays and maps (or objects, in JSON terminology), and that's about it.</li> -<li><strong>JSON can naturally represent real-world objects</strong>. Try representing your application's <code>Customer</code> object (with the person's address, order history, etc.) in a CSV file or a relational database. It's hard. In fact, ORM systems were invented to help alleviate this issue.</li> -<li><strong>JSON libraries are available in virtually every programming language</strong>. Take a look at <a href="http://www.json.org/">the list of supported languages on JSON.org</a>. I counted 15 languages that start with the letters A, B or C.</li> -<li><strong>JSON is idiomatic in loosely typed languages</strong>. Many loosely typed languages, such as Python, Ruby and JavaScript, have data structures that are similar to JSON objects, making it very natural to handle JSON data in those languages. For example, a Python dictionary looks just like a JSON object. This makes it easy for developers to utilize JSON in their applications.</li> -</ul> - -<h1 id="json-data-infrastructure">JSON Data Infrastructure</h1> - -<p>Traditional data infrastructure, such as relational databases, has some features that make it easier to store and process JSON-encoded data. For example, Oracle has <a href="https://docs.oracle.com/database/121/ADXDB/json.htm">a JSON data type and a set of functions for handling JSON data</a>.</p> - -<p>However, a new class of data infrastructure is providing a much more seamless experience via a full-fledged JSON data model. For example:</p> - -<ul> -<li>Drill is a SQL engine in which each record is conceptually a JSON document.</li> -<li>Elasticsearch is a search engine in which each indexed document is conceptually a JSON document.</li> -<li>MongoDB is an operational database in which each record is conceptually a JSON document.</li> -</ul> - -<p>These systems view JSON as a data model as opposed to one of many data types, realizing that JSON offers a simple way to represent real-world objects.</p> - -<table><thead> -<tr> -<th></th> -<th>Traditional Infrastructure</th> -<th>JSON Infrastructure</th> -</tr> -</thead><tbody> -<tr> -<td><strong>Examples:</strong></td> -<td>Oracle, SQL Server</td> -<td>Drill, Elasticsearch, MongoDB</td> -</tr> -<tr> -<td><strong>Record:</strong></td> -<td>Tuple</td> -<td>JSON document</td> -</tr> -<tr> -<td><strong>Variable schema:</strong></td> -<td>No</td> -<td>Yes</td> -</tr> -</tbody></table> - -<p>If you happen to be in the Bay Area tomorrow, please join Gaurav Gupta (VP Product Management, Elasticsearch), Paul Pedersen (Deputy CTO, MongoDB), Robert Greene (Senior Principal Product Manager, Oracle), Sukanta Ganguly (VP Solutions Architecture, Aerospike) and me for a panel moderated by Gartner's Nick Heudecker on this new world of schema-free JSON. Check out <a href="http://www.meetup.com/SF-Bay-Areas-Big-Data-Think-Tank/">The Hive Big Data Think Tank</a> for more information.</p> - - Tue, 27 Jan 2015 00:50:01 -0800 - /blog/2015/01/27/schema-free-json-data-infrastructure/ - /blog/2015/01/27/schema-free-json-data-infrastructure/ - - - blog - - - http://git-wip-us.apache.org/repos/asf/drill-site/blob/9090a50b/index.html ---------------------------------------------------------------------- diff --git a/index.html b/index.html index 537214a..269b48c 100644 --- a/index.html +++ b/index.html @@ -166,7 +166,7 @@ $(document).ready(function() {

@@ -282,6 +282,6 @@ m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m) ga('create', 'UA-53379651-1', 'auto'); ga('send', 'pageview'); - + http://git-wip-us.apache.org/repos/asf/drill-site/blob/9090a50b/js/script.js ---------------------------------------------------------------------- diff --git a/js/script.js b/js/script.js index 8fc7a54..ea318e2 100644 --- a/js/script.js +++ b/js/script.js @@ -1,107 +1,107 @@ -var reelPointer = null; -$(document).ready(function(e) { - - $(".aLeft").click(function() { - moveReel("prev"); - }); - $(".aRight").click(function() { - moveReel("next"); - }); - - if ($("#header .scroller .item").length == 1) { - - } else { - - $("#header .dots, .aLeft, .aRight").css({ display: 'block' }); - $("#header .scroller .item").each(function(i) { - $("#header .dots").append("

"); - $("#header .dots .dot").eq(i).click(function() { - var index = $(this).prevAll(".dot").length; - moveReel(index); - }); - }); - - reelPointer = setTimeout(function() { moveReel(1); },5000); - } - - $("#menu ul li").each(function(index, element) { - if ($(this).find("ul").length) { - $(this).addClass("parent"); - } - }); - - $("#header .dots .dot:eq(0)").addClass("sel"); - - resized(); - - $(window).scroll(onScroll); - - var pathname = window.location.pathname; - var pathSlashesReplaced = pathname.replace(/\//g, " "); - var pathSlashesReplacedNoFirstDash = pathSlashesReplaced.replace(" ",""); - var newClass = pathSlashesReplacedNoFirstDash.replace(/(\.[\s\S]+)/ig, ""); - $("body").addClass(newClass); - if ( $("body").attr("class") == "") - { - $("body").addClass("class"); - } -}); - -var reel_currentIndex = 0; -function resized() { - - var WW = parseInt($(window).width(),10); - var IW = (WW < 999) ? 999 : WW; - var IH = parseInt($("#header .scroller .item").css("height"),10); - var IN = $("#header .scroller .item").length; - - $("#header .scroller").css({ width: (IN * IW)+"px", marginLeft: -(reel_currentIndex * IW)+"px" }); - $("#header .scroller .item").css({ width: IW+"px" }); - - - $("#header .scroller .item").each(function(i) { - var th = parseInt($(this).find(".tc").height(),10); - var d = IH - th + 25; - $(this).find(".tc").css({ top: Math.round(d/2)+"px" }); - }); - - if (WW < 999) $("#menu").addClass("r"); - else $("#menu").removeClass("r"); - - onScroll(); - -} - -function moveReel(direction) { - - if (reelPointer) clearTimeout(reelPointer); - - var IN = $("#header .scroller .item").length; - var IW = $("#header .scroller .item").width(); - if (direction == "next") reel_currentIndex++; - else if (direction == "prev") reel_currentIndex--; - else reel_currentIndex = direction; - - if (reel_currentIndex >= IN) reel_currentIndex = 0; - if (reel_currentIndex < 0) reel_currentIndex = IN-1; - - $("#header .dots .dot").removeClass("sel"); - $("#header .dots .dot").eq(reel_currentIndex).addClass("sel"); - - $("#header .scroller").stop(false,true,false).animate({ marginLeft: -(reel_currentIndex * IW)+"px" }, 1000, "easeOutQuart"); - - reelPointer = setTimeout(function() { moveReel(1); },5000); - -} - -function onScroll() { - var ST = document.body.scrollTop || document.documentElement.scrollTop; - //if ($("#menu.r").length) { - // $("#menu.r").css({ top: ST+"px" }); - //} else { - // $("#menu").css({ top: "0px" }); - //} - - if (ST > 400) $("#subhead").addClass("show"); - else $("#subhead").removeClass("show"); -} +var reelPointer = null; +$(document).ready(function(e) { + + $(".aLeft").click(function() { + moveReel("prev"); + }); + $(".aRight").click(function() { + moveReel("next"); + }); + + if ($("#header .scroller .item").length == 1) { + + } else { + + $("#header .dots, .aLeft, .aRight").css({ display: 'block' }); + $("#header .scroller .item").each(function(i) { + $("#header .dots").append("

"); + $("#header .dots .dot").eq(i).click(function() { + var index = $(this).prevAll(".dot").length; + moveReel(index); + }); + }); + + reelPointer = setTimeout(function() { moveReel(1); },5000); + } + + $("#menu ul li").each(function(index, element) { + if ($(this).find("ul").length) { + $(this).addClass("parent"); + } + }); + + $("#header .dots .dot:eq(0)").addClass("sel"); + + resized(); + + $(window).scroll(onScroll); + + var pathname = window.location.pathname; + var pathSlashesReplaced = pathname.replace(/\//g, " "); + var pathSlashesReplacedNoFirstDash = pathSlashesReplaced.replace(" ",""); + var newClass = pathSlashesReplacedNoFirstDash.replace(/(\.[\s\S]+)/ig, ""); + $("body").addClass(newClass); + if ( $("body").attr("class") == "") + { + $("body").addClass("class"); + } +}); + +var reel_currentIndex = 0; +function resized() { + + var WW = parseInt($(window).width(),10); + var IW = (WW < 999) ? 999 : WW; + var IH = parseInt($("#header .scroller .item").css("height"),10); + var IN = $("#header .scroller .item").length; + + $("#header .scroller").css({ width: (IN * IW)+"px", marginLeft: -(reel_currentIndex * IW)+"px" }); + $("#header .scroller .item").css({ width: IW+"px" }); + + + $("#header .scroller .item").each(function(i) { + var th = parseInt($(this).find(".tc").height(),10); + var d = IH - th + 25; + $(this).find(".tc").css({ top: Math.round(d/2)+"px" }); + }); + + if (WW < 999) $("#menu").addClass("r"); + else $("#menu").removeClass("r"); + + onScroll(); + +} + +function moveReel(direction) { + + if (reelPointer) clearTimeout(reelPointer); + + var IN = $("#header .scroller .item").length; + var IW = $("#header .scroller .item").width(); + if (direction == "next") reel_currentIndex++; + else if (direction == "prev") reel_currentIndex--; + else reel_currentIndex = direction; + + if (reel_currentIndex >= IN) reel_currentIndex = 0; + if (reel_currentIndex < 0) reel_currentIndex = IN-1; + + $("#header .dots .dot").removeClass("sel"); + $("#header .dots .dot").eq(reel_currentIndex).addClass("sel"); + + $("#header .scroller").stop(false,true,false).animate({ marginLeft: -(reel_currentIndex * IW)+"px" }, 1000, "easeOutQuart"); + + reelPointer = setTimeout(function() { moveReel(1); },5000); + +} + +function onScroll() { + var ST = document.body.scrollTop || document.documentElement.scrollTop; + //if ($("#menu.r").length) { + // $("#menu.r").css({ top: ST+"px" }); + //} else { + // $("#menu").css({ top: "0px" }); + //} + + if (ST > 400) $("#subhead").addClass("show"); + else $("#subhead").removeClass("show"); +} http://git-wip-us.apache.org/repos/asf/drill-site/blob/9090a50b/mailinglists/index.html ---------------------------------------------------------------------- diff --git a/mailinglists/index.html b/mailinglists/index.html index a1eb84d..5b76986 100644 --- a/mailinglists/index.html +++ b/mailinglists/index.html @@ -153,6 +153,6 @@ m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m) ga('create', 'UA-53379651-1', 'auto'); ga('send', 'pageview'); - + http://git-wip-us.apache.org/repos/asf/drill-site/blob/9090a50b/overview/index.html ---------------------------------------------------------------------- diff --git a/overview/index.html b/overview/index.html index ba05e45..83872c2 100644 --- a/overview/index.html +++ b/overview/index.html @@ -161,6 +161,6 @@ m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m) ga('create', 'UA-53379651-1', 'auto'); ga('send', 'pageview'); - + http://git-wip-us.apache.org/repos/asf/drill-site/blob/9090a50b/search/index.html ---------------------------------------------------------------------- diff --git a/search/index.html b/search/index.html index 3cfd113..beb29cd 100644 --- a/search/index.html +++ b/search/index.html @@ -143,6 +143,6 @@ m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m) ga('create', 'UA-53379651-1', 'auto'); ga('send', 'pageview'); - + http://git-wip-us.apache.org/repos/asf/drill-site/blob/9090a50b/team/index.html ---------------------------------------------------------------------- diff --git a/team/index.html b/team/index.html index a006b30..22bbead 100644 --- a/team/index.html +++ b/team/index.html @@ -218,6 +218,6 @@ m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m) ga('create', 'UA-53379651-1', 'auto'); ga('send', 'pageview'); - + http://git-wip-us.apache.org/repos/asf/drill-site/blob/9090a50b/why/index.html ---------------------------------------------------------------------- diff --git a/why/index.html b/why/index.html index f06e60f..fb2d51e 100644 --- a/why/index.html +++ b/why/index.html @@ -196,6 +196,6 @@ m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m) ga('create', 'UA-53379651-1', 'auto'); ga('send', 'pageview'); - +