Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id EC350200AE2 for ; Fri, 13 May 2016 00:37:58 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id EA8C7160A10; Thu, 12 May 2016 22:37:58 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 77281160939 for ; Fri, 13 May 2016 00:37:57 +0200 (CEST) Received: (qmail 34324 invoked by uid 500); 12 May 2016 22:37:56 -0000 Mailing-List: contact user-help@kudu.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@kudu.incubator.apache.org Delivered-To: mailing list user@kudu.incubator.apache.org Received: (qmail 34316 invoked by uid 99); 12 May 2016 22:37:56 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 May 2016 22:37:56 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 3583B1A4E39 for ; Thu, 12 May 2016 22:37:56 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.198 X-Spam-Level: * X-Spam-Status: No, score=1.198 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd2-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id glDeIybsEJWV for ; Thu, 12 May 2016 22:37:52 +0000 (UTC) Received: from mail-oi0-f43.google.com (mail-oi0-f43.google.com [209.85.218.43]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 28C955F39D for ; Thu, 12 May 2016 22:37:51 +0000 (UTC) Received: by mail-oi0-f43.google.com with SMTP id k142so143799351oib.1 for ; Thu, 12 May 2016 15:37:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to; bh=t92BaGfHcIXt9HZTDDRV1UmR+q1c96xwMTD9gK2lkj0=; b=CuYiwuT6/0N4FvsSZwfov7CYwOY18LgbgruiOBJxImXqQDtSJvRxABXHgFUpfFx6bG LYetVmP8bYwhXm6XYI2M9nqv5t3CZRqLwpWhMi9+VewCn0KXwCfVmgn2jTA99aPLzv8c UxN+VdD8n7tXhefnvDJ2MkuqImuhkwh7b304XjaFudR/9tgVYLMkWQ73g8FBstxooRDc yPdWyAIYJUl4LbWn245PaqlgaHTj86VEOF98TrC8tjl+0FPI5Ep8cmvsVWaErza6JXHd X8lrcNDqhiq1DNfNkszdoaMIlHx9xaAUkGVqmWggFaRskDw1lsAH4w8sFQNyDoqZ/hRB CS4Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to; bh=t92BaGfHcIXt9HZTDDRV1UmR+q1c96xwMTD9gK2lkj0=; b=Mnk6A96Ozg/RQz/5MGlhMNiDBNmBUzp7f78bV0JegEklznILE5qjUTYOFZTMXoVumX xhAaNvaIkmBsXEplnnt331gbt/z8hk8CAszu9gxSIx+r9GlFajy4vX0JKW+ZL7W2LR04 DsXY2YkUQIrg+JJR3Ihc3ddXZeSO66FOrg28N44NRHCEgq/rD3AWDa5o7QC165tqfO7/ XklFw1W7XLmmMs/yUXuR857eusVLVGXTC2Kynt31SO+NvpWpQhZlgWROxV184nTx9tef WQlzNwLfKNjsvDY53544N7BBjjZHkegjnmL7P/t/r8jJlc1eSlcjUeiEjZdConOXAPFz daTA== X-Gm-Message-State: AOPr4FUnTWAlPCxu4b7lbD0L13Iba8ptdDdyRoRm2gtoVXhHk5d+9TFCMUh6kjSDU4hluwxdogBKlxQl8JmCyQ== MIME-Version: 1.0 X-Received: by 10.157.49.118 with SMTP id v51mr7179052otd.97.1463092664060; Thu, 12 May 2016 15:37:44 -0700 (PDT) Received: by 10.202.205.212 with HTTP; Thu, 12 May 2016 15:37:43 -0700 (PDT) In-Reply-To: References: Date: Thu, 12 May 2016 15:37:43 -0700 Message-ID: Subject: Re: Partition and Split rows From: Sand Stone To: user@kudu.incubator.apache.org Content-Type: multipart/alternative; boundary=001a113d05f083cffe0532acceff archived-at: Thu, 12 May 2016 22:37:59 -0000 --001a113d05f083cffe0532acceff Content-Type: text/plain; charset=UTF-8 Cool, I will study it. On Thu, May 12, 2016 at 2:17 PM, Dan Burkert wrote: > > > On Thu, May 12, 2016 at 2:04 PM, Sand Stone > wrote: > > >Instead, take advantage of the index capability of Primary Keys. >> Currently I did make the "5-min" field a part of the primary key as well. >> I am most likely overdoing it. I will play around with the schema and use >> cases around it. >> > > Definitely take a look at the data model in Kudu TS, it has extremely > efficient scan semantics (all scans retrieve only the necessary data by > using the primary key, no client *or* server side filtering), and it works > with arbitrarily large time range partitions. > > >> >since each tablet server should only have on the order of 10-20 tablets. >> How does this 10-20 heuristics come out? Is it based on certain machine >> profile? Or some default parameters in the code/config? >> > > 10-20 isn't a hard and fast rule, and it is very dependent on the dataset > and the machine size. We routinely run tablet servers with 300+ tablets, > so it's definitely not a hard limitation. My intention was to stress that > instead of pursuing fine grained partitions as a method of limiting the > size of scans, instead take advantage of the primary key indexing that Kudu > provides, just as you might in a traditional relational database. > Partitions are better suited for ensuring that inserts and large scans can > be parallelized across multiple machines. > > - Dan > > >> >> On Thu, May 12, 2016 at 1:45 PM, Dan Burkert wrote: >> >>> >>> On Thu, May 12, 2016 at 11:39 AM, Sand Stone >>> wrote: >>> >>> I don't know how Kudu load balance the data across the tablet servers. >>>> >>> >>> Individual tablets are replicated and balanced across all available >>> tablet servers, for more on that see >>> http://getkudu.io/docs/schema_design.html#data-distribution. >>> >>> >>> >>>> For example, do I need to pre-calculate every day, a list of 5 minutes >>>> apart timestamps at table creation? [assume I have to create a new table >>>> every day]. >>>> >>> >>> If you wish to range partition on the time column, then yes, currently >>> you must specify the splits upfront during table creation (but this will >>> change with the non-covering range partitions work). >>> >>> >>>> >>>> My hope, with the additional 5-min column, and use it as the range >>>> partition column, is that so I could spread the data evenly across the >>>> tablet servers. >>>> >>> >>> I don't think this is meaningfully different than range partitioning on >>> the full time column with splits every 5 minutes. >>> >>> >>>> Also, since 5-min interval data are always colocated together, the read >>>> query could be efficient too. >>>> >>> >>> Data colocation is a function of the partitioning and indexing. As I >>> mentioned before, if you have timestamp as part of your primary key then >>> you can guarantee that scans specifying a time range are efficient. Overall >>> it sounds like you are attempting to get fast scans by creating many fine >>> grained partitions, as you might with Parquet. This won't be an efficient >>> strategy in Kudu, since each tablet server should only have on the order of >>> 10-20 tablets. Instead, take advantage of the index capability of Primary >>> Keys. >>> >>> - Dan >>> >>> >>>> On Thu, May 12, 2016 at 11:13 AM, Dan Burkert wrote: >>>> >>>>> Forgot to add the PK specification to the CREATE TABLE, it should have >>>>> read as follows: >>>>> >>>>> CREATE TABLE metrics (metric STRING, time TIMESTAMP, value DOUBLE) >>>>> PRIMARY KEY (metric, time); >>>>> >>>>> - Dan >>>>> >>>>> >>>>> On Thu, May 12, 2016 at 11:12 AM, Dan Burkert >>>>> wrote: >>>>> >>>>>> >>>>>> On Thu, May 12, 2016 at 11:05 AM, Sand Stone >>>>>> wrote: >>>>>> >>>>>>> > Is the requirement to pre-aggregate by time window? >>>>>>> No, I am thinking to create a column say, "minute". It's basically >>>>>>> the minute field of the timestamp column(even round to 5-min bucket >>>>>>> depending on the needs). So it's a computed column being filled in on data >>>>>>> ingestion. My goal is that this field would help with data filtering at >>>>>>> read/query time, say select certain projection at minute 10-15, to speed up >>>>>>> the read queries. >>>>>>> >>>>>> >>>>>> In many cases, Kudu can do his for you without having to add special >>>>>> columns. The requirements are that the timestamp is part of the primary >>>>>> key, and any columns that come before the timestamp in the primary key (if >>>>>> it's a compound PK), have equality predicates. So for instance, if you >>>>>> create a table such as: >>>>>> >>>>>> CREATE TABLE metrics (metric STRING, time TIMESTAMP, value DOUBLE); >>>>>> >>>>>> then queries such as >>>>>> >>>>>> SELECT time, value FROM metrics WHERE metric = "my-metric" AND time > >>>>>> 2016-05-01T00:00 AND time < 2016-05-01T00:05 >>>>>> >>>>>> Then only the data for that 5 minute time window will be read from >>>>>> disk. If the query didn't have the equality predicate on the 'metric' >>>>>> column, then it would do a much bigger scan + filter operation. If you >>>>>> want more background on how this is achieved, check out the partition >>>>>> pruning design doc: >>>>>> https://github.com/apache/incubator-kudu/blob/master/docs/design-docs/scan-optimization-partition-pruning.md >>>>>> . >>>>>> >>>>>> - Dan >>>>>> >>>>>> >>>>>> >>>>>>> Thanks for the info., I will follow them. >>>>>>> >>>>>>> On Thu, May 12, 2016 at 10:50 AM, Dan Burkert >>>>>>> wrote: >>>>>>> >>>>>>>> Hey Sand, >>>>>>>> >>>>>>>> Sorry for the delayed response. I'm not quite following your use >>>>>>>> case. Is the requirement to pre-aggregate by time window? I don't think >>>>>>>> Kudu can help you directly with that (nothing built in), but you could >>>>>>>> always create a separate table to store the pre-aggregated values. As far >>>>>>>> as applying functions to do row splits, that is an interesting idea, but I >>>>>>>> think once Kudu has support for range bounds (the non-covering range >>>>>>>> partition design doc linked above), you can simply create the bounds where >>>>>>>> the function would have put them. For example, if you want a partition for >>>>>>>> every five minutes, you can create the bounds accordingly. >>>>>>>> >>>>>>>> Earlier this week I gave a talk on timeseries in Kudu, I've >>>>>>>> included some slides that may be interesting to you. Additionally, you may >>>>>>>> want to check out https://github.com/danburkert/kudu-ts, it's a >>>>>>>> very young (not feature complete) metrics layer on top of Kudu, it may >>>>>>>> give you some ideas. >>>>>>>> >>>>>>>> - Dan >>>>>>>> >>>>>>>> On Sat, May 7, 2016 at 1:28 PM, Sand Stone >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Thanks for sharing, Dan. The diagrams explained clearly how the >>>>>>>>> current system works. >>>>>>>>> >>>>>>>>> As for things in my mind. Take the schema of >>>>>>>>> , say, I am interested in data for the past 5 mins, >>>>>>>>> 10 mins, etc. Or, aggregate at 5 mins interval for the past 3 days, 7 days, >>>>>>>>> ... Looks like I need to introduce a special 5-min bar column, use that >>>>>>>>> column to do range partition to spread data across the tablet servers so >>>>>>>>> that I could leverage parallel filtering. >>>>>>>>> >>>>>>>>> The cost of this extra column (INT8) is not ideal but not too bad >>>>>>>>> either (storage cost wise, compression should do wonders). So I am thinking >>>>>>>>> whether it would be better to take "functions" as row split instead of only >>>>>>>>> constants. Of course if business requires to drop down to 1-min bar, the >>>>>>>>> data has to be re-sharded again. So a more cost effective way of doing this >>>>>>>>> on a production cluster would be good. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Sat, May 7, 2016 at 8:50 AM, Dan Burkert >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Hi Sand, >>>>>>>>>> >>>>>>>>>> I've been working on some diagrams to help explain some of the >>>>>>>>>> more advanced partitioning types, it's attached. Still pretty rough at >>>>>>>>>> this point, but the goal is to clean it up and move it into the Kudu >>>>>>>>>> documentation proper. I'm interested to hear what kind of time series you >>>>>>>>>> are interested in Kudu for. I'm tasked with improving Kudu for time >>>>>>>>>> series, you can follow progress here >>>>>>>>>> . If you have >>>>>>>>>> any additional ideas I'd love to hear them. You may also be interested in >>>>>>>>>> a small project that a JD and I have been working on in the past week to >>>>>>>>>> build an OpenTSDB style store on top of Kudu, you can find it >>>>>>>>>> here . Still quite >>>>>>>>>> feature limited at this point. >>>>>>>>>> >>>>>>>>>> - Dan >>>>>>>>>> >>>>>>>>>> On Fri, May 6, 2016 at 4:51 PM, Sand Stone < >>>>>>>>>> sand.m.stone@gmail.com> wrote: >>>>>>>>>> >>>>>>>>>>> Thanks. Will read. >>>>>>>>>>> >>>>>>>>>>> Given that I am researching time series data, row locality is >>>>>>>>>>> crucial :-) >>>>>>>>>>> >>>>>>>>>>> On Fri, May 6, 2016 at 3:57 PM, Jean-Daniel Cryans < >>>>>>>>>>> jdcryans@apache.org> wrote: >>>>>>>>>>> >>>>>>>>>>>> We do have non-covering range partitions coming in the next few >>>>>>>>>>>> months, here's the design (in review): >>>>>>>>>>>> http://gerrit.cloudera.org:8080/#/c/2772/9/docs/design-docs/non-covering-range-partitions.md >>>>>>>>>>>> >>>>>>>>>>>> The "Background & Motivation" section should give you a good >>>>>>>>>>>> idea of why I'm mentioning this. >>>>>>>>>>>> >>>>>>>>>>>> Meanwhile, if you don't need row locality, using hash >>>>>>>>>>>> partitioning could be good enough. >>>>>>>>>>>> >>>>>>>>>>>> J-D >>>>>>>>>>>> >>>>>>>>>>>> On Fri, May 6, 2016 at 3:53 PM, Sand Stone < >>>>>>>>>>>> sand.m.stone@gmail.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Makes sense. >>>>>>>>>>>>> >>>>>>>>>>>>> Yeah it would be cool if users could specify/control the split >>>>>>>>>>>>> rows after the table is created. Now, I have to "think ahead" to pre-create >>>>>>>>>>>>> the range buckets. >>>>>>>>>>>>> >>>>>>>>>>>>> On Fri, May 6, 2016 at 3:49 PM, Jean-Daniel Cryans < >>>>>>>>>>>>> jdcryans@apache.org> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> You will only get 1 tablet and no data distribution, which is >>>>>>>>>>>>>> bad. >>>>>>>>>>>>>> >>>>>>>>>>>>>> That's also how HBase works, but it will split regions as you >>>>>>>>>>>>>> insert data and eventually you'll get some data distribution even if it >>>>>>>>>>>>>> doesn't start in an ideal situation. Tablet splitting will come later for >>>>>>>>>>>>>> Kudu. >>>>>>>>>>>>>> >>>>>>>>>>>>>> J-D >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Fri, May 6, 2016 at 3:42 PM, Sand Stone < >>>>>>>>>>>>>> sand.m.stone@gmail.com> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> One more questions, how does the range partition work if I >>>>>>>>>>>>>>> don't specify the split rows? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks! >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Fri, May 6, 2016 at 3:37 PM, Sand Stone < >>>>>>>>>>>>>>> sand.m.stone@gmail.com> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thanks, Misty. The "advanced" impala example helped. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I was just reading the Java API,CreateTableOptions.java, >>>>>>>>>>>>>>>> it's unclear how the range partition column names associated with the >>>>>>>>>>>>>>>> partial rows params in the addSplitRow API. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Fri, May 6, 2016 at 3:08 PM, Misty Stanley-Jones < >>>>>>>>>>>>>>>> mstanleyjones@cloudera.com> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Hi Sand, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Please have a look at >>>>>>>>>>>>>>>>> http://getkudu.io/docs/kudu_impala_integration.html#partitioning_tables >>>>>>>>>>>>>>>>> and see if it is helpful to you. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>> Misty >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Fri, May 6, 2016 at 2:00 PM, Sand Stone < >>>>>>>>>>>>>>>>> sand.m.stone@gmail.com> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Hi, I am new to Kudu. I wonder how the split rows work. I >>>>>>>>>>>>>>>>>> know from some docs, this is currently for pre-creation the table. I am >>>>>>>>>>>>>>>>>> researching how to partition (hash+range) some time series test data. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Is there an example? or notes somewhere I could read >>>>>>>>>>>>>>>>>> upon. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Thanks much. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> > --001a113d05f083cffe0532acceff Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Cool, I will study it.
On Thu, May 12, 2016 at 2:17 PM, Dan Burkert <d= an@cloudera.com> wrote:


On Thu, May 12, 2016 at 2:04 PM, Sand Stone <sand.m= .stone@gmail.com> wrote:

>Instead, take advantage of the i= ndex capability of Primary Keys.
Currently I did ma= ke the "5-min"=C2=A0field a part of the primary key as well. I am= most likely overdoing it. I will play around with the schema and use cases= around it.=C2=A0
=C2=A0
Defi= nitely take a look at the data model in Kudu TS, it has extremely efficient= scan semantics (all scans retrieve only the necessary data by using the pr= imary key, no client *or* server side filtering), and it works with arbitra= rily large time range partitions.
=C2=A0
>since each tablet server should only have on th= e order of 10-20 tablets.
How does this 10-20 heuri= stics come out? Is it based on certain machine profile? Or some default par= ameters in the code/config?

<= div>10-20 isn't a hard and fast rule, and it is very dependent on the d= ataset and the machine size.=C2=A0 We routinely run tablet servers with 300= + tablets, so it's definitely not a hard limitation.=C2=A0 My intention= was to stress that instead of pursuing fine grained partitions as a method= of limiting the size of scans, instead take advantage of the primary key i= ndexing that Kudu provides, just as you might in a traditional relational d= atabase.=C2=A0 Partitions are better suited for ensuring that inserts and l= arge scans can be parallelized across multiple machines.

- Dan
=C2=A0

On Thu, May 12, 2016 at 1:45 PM, Dan Burkert <<= a href=3D"mailto:dan@cloudera.com" target=3D"_blank">dan@cloudera.com&g= t; wrote:

On Thu, May 12, 20= 16 at 11:39 AM, Sand Stone <sand.m.stone@gmail.com> wro= te:

I don't know how Kudu load balance the data across the= tablet servers.

Individ= ual tablets are replicated and balanced across all available tablet servers= , for more on that see=C2=A0http://getkudu.io/docs/schema_de= sign.html#data-distribution.

=C2=A0
For example, do I need to pre-calc= ulate every day, a list of 5 minutes apart timestamps at table creation? [a= ssume I have to create a new table every day].

If you wish to range partition on the time column, t= hen yes, currently you must specify the splits upfront during table creatio= n (but this will change with the non-covering range partitions work).
=
=C2=A0

My hope, with the additional 5-min column, and use it as the range partit= ion column, is that so I could spread the data evenly across the tablet ser= vers.

I don't think = this is meaningfully different than range partitioning on the full time col= umn with splits every 5 minutes.
=C2=A0=C2=A0
Also, since 5-min interval data are always c= olocated together, the read query could be efficient too.=C2=A0
=

Data colocation is a function of th= e partitioning and indexing.=C2=A0 As I mentioned before, if you have times= tamp as part of your primary key then you can guarantee that scans specifyi= ng a time range are efficient. Overall it sounds like you are attempting to= get fast scans by creating many fine grained partitions, as you might with= Parquet.=C2=A0 This won't be an efficient strategy in Kudu, since each= tablet server should only have on the order of 10-20 tablets.=C2=A0 Instea= d, take advantage of the index capability of Primary Keys.
<= div>=C2=A0
- Dan
=C2=A0
On Thu, May 12, 2016= at 11:13 AM, Dan Burkert <dan@cloudera.com> wrote:
Forgot to add the PK specification to the CREAT= E TABLE, it should have read as follows:

CREATE TABLE metrics (metric STRING, time TIMESTAMP,= value DOUBLE)
PRIMARY KEY (metric, time);

- Dan
=


On Thu, May 12, 2016 at 11:12 AM, Dan Burkert <dan@cloudera.c= om> wrote:

On Thu, May 12, 2016 at = 11:05 AM, Sand Stone <sand.m.stone@gmail.com> wrote:
>=C2=A0Is the requirement to pre-aggregate by time window?No, I am thinking to create a column say, "minute". It's=C2= =A0basically the minute field of the timestamp column(even round to 5-min b= ucket depending on the needs). So it's a computed column being filled i= n on data ingestion. My goal is that this field would help with data filter= ing at read/query time, say select certain projection at minute 10-15, to s= peed up the read queries.=C2=A0

In many cases, Kudu can do his for you without having to add specia= l columns.=C2=A0 The requirements are that the timestamp is part of the pri= mary key, and any columns that come before the timestamp in the primary key= (if it's a compound PK), have equality predicates.=C2=A0 So for instan= ce, if you create a table such as:

CREATE TABLE me= trics (metric STRING, time TIMESTAMP, value DOUBLE);

then queries such as=C2=A0

SELECT time, value F= ROM metrics WHERE metric =3D "my-metric" AND time > 2016-05-01= T00:00 AND time < 2016-05-01T00:05

Then only th= e data for that 5 minute time window will be read from disk.=C2=A0 If the q= uery didn't have the equality predicate on the 'metric' column,= then it would do a much bigger scan + filter operation.=C2=A0 If you want = more background on how this is achieved, check out the partition pruning de= sign doc:=C2=A0https://github.com/apache/incubator-kudu/blob/master/docs/design-docs/s= can-optimization-partition-pruning.md.

- Dan

=C2=A0
Thanks for= the info., I will follow them.=C2=A0

On Thu, May 12, 2016 at 10:50 AM,= Dan Burkert <dan@cloudera.com> wrote:
=
Hey Sand,

Sorry for the delayed respons= e.=C2=A0 I'm not quite following your use case.=C2=A0 Is the requiremen= t to pre-aggregate by time window? I don't think Kudu can help you dire= ctly with that (nothing built in), but you could always create a separate t= able to store the pre-aggregated values.=C2=A0 As far as applying functions= to do row splits, that is an interesting idea, but I think once Kudu has s= upport for range bounds (the non-covering range partition design doc linked= above), you can simply create the bounds where the function would have put= them.=C2=A0 For example, if you want a partition for every five minutes, y= ou can create the bounds accordingly.

Earlier this= week I gave a talk on timeseries in Kudu, I've included some slides th= at may be interesting to you.=C2=A0 Additionally, you may want to check out= =C2=A0h= ttps://github.com/danburkert/kudu-ts, it's a very young =C2=A0(not = feature complete) metrics layer on top of Kudu, it may give you some ideas.=

- Dan
<= /span>

On Sat, May 7, 2016 at 1:28 PM, Sand Stone <sand.m.stone@gmail.c= om> wrote:
Thanks for sh= aring, Dan. The diagrams explained clearly how the current system works.=C2= =A0

As for things in my mind. Take the schema of <hos= t,metric,time,...>, say, I am interested in data for the past 5 mins, 10= mins, etc. Or, aggregate at 5 mins interval for the past 3 days, 7 days, .= .. Looks like I need to introduce a special 5-min bar column, use that colu= mn to do range partition to spread data across the tablet servers so that I= could leverage parallel filtering.=C2=A0

The cost of th= is extra column (INT8) is not ideal but not too bad either (storage cost wi= se, compression should do wonders). So I am thinking whether it would be be= tter to take "functions" as row split instead of only constants. = Of course if business requires to drop down to 1-min bar, the data has to b= e re-sharded again. So a more cost effective way of doing this on a product= ion cluster would be good.=C2=A0



On Sat, May 7, 2016 at 8:50 AM, Dan Burkert = <dan@cloudera.com<= /a>> wrote:
Hi Sand,
I've been working on some diagrams to help explain some of = the more advanced partitioning types, it's attached. =C2=A0 Still prett= y rough at this point, but the goal is to clean it up and move it into the = Kudu documentation proper.=C2=A0 I'm interested to hear what kind of ti= me series you are interested in Kudu for.=C2=A0 I'm tasked with improvi= ng Kudu for time series, you can follow progress here. If you have a= ny additional ideas I'd love to hear them.=C2=A0 You may also be intere= sted in a small project that a JD and I have been working on in the past we= ek to build an OpenTSDB style store on top of Kudu, you can find it=C2=A0here.= =C2=A0 Still quite feature limited at this point.

- Dan

On Fri, May 6, 2016 a= t 4:51 PM, Sand Stone <sand.m.stone@gmail.com> wrote:
Thanks. Will read.=C2=A0

=
Given that I am researching time series data, row locality is crucial = :-) =C2=A0

On Fri, May 6, 2016 at 3:57 PM, Jean-Daniel Cryans <jdc= ryans@apache.org> wrote:
We do have non-covering range partitions coming in the next few months, he= re's the design (in review):=C2=A0http://gerrit.cloudera.org:8080/#/c/2772/9/docs/design-docs/non= -covering-range-partitions.md

The "Background &= amp; Motivation" section should give you a good idea of why I'm me= ntioning this.

Meanwhile, if you don't need ro= w locality, using hash partitioning could be good enough.

J-D

On Fri, May 6, = 2016 at 3:53 PM, Sand Stone <sand.m.stone@gmail.com> wr= ote:
Makes sense.=C2=A0

=
Yeah it would be cool if users could specify/control the split rows af= ter the table is created. Now, I have to "think ahead" to pre-cre= ate the range buckets.=C2=A0

On Fri, May 6, 2016 at 3:49 PM, Jean-Danie= l Cryans <jdcryans@apache.org> wrote:
=
You will only get 1 tablet and no data distribution, which= is bad.

That's also how HBase works, but it will sp= lit regions as you insert data and eventually you'll get some data dist= ribution even if it doesn't start in an ideal situation. Tablet splitti= ng will come later for Kudu.

J-D
=
On Fri, May 6, 2016 at 3:42 PM, Sand Stone <= span dir=3D"ltr"><sand.m.stone@gmail.com> wrote:
One more questions, how does the range partition work if I don= 't specify the split rows?=C2=A0

Thanks!=C2=A0
=

O= n Fri, May 6, 2016 at 3:37 PM, Sand Stone <sand.m.stone@gmail.com= > wrote:
Thanks, Misty. The "advanced" i= mpala example helped.=C2=A0

I was just reading the Jav= a API,CreateTableOptions.java, it's unclear how the range partition col= umn names associated with the partial rows params in the=C2=A0addSplitRow API.=

On Fri, May 6, 2016 at 3:08 PM, Misty Stanley-Jones <mstanleyjones@cloudera.com> wrote:
Hi Sand,

Please have a look at=C2=A0http://getkudu.io/docs/kudu_impala_integration.html#par= titioning_tables and see if it is helpful to you.

<= div>Thanks,
Misty

On Fri, May 6, 2016 at 2:00 PM, Sand Stone = <sand.m.stone@gmail.com> wrote:
Hi, I am new to Kudu. I wonder how the split rows work. I kno= w from some docs, this is currently for pre-creation the table. I am resear= ching how to partition (hash+range) some time series test data.=C2=A0
<= br>
Is there an example? or notes somewhere I could read upon.=C2= =A0

Thanks much.=C2=A0


















--001a113d05f083cffe0532acceff--