Mailing-List: contact user-help@accumulo.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@accumulo.apache.org
Date: Fri, 4 Sep 2015 15:57:57 +0000 (UTC)
From: dlmarion@comcast.net
To: user@accumulo.apache.org
Message-ID: <720171860.18313820.1441382277779.JavaMail.zimbra@comcast.net>
In-Reply-To: 
 <CAOiqz+ReYxOgsXnwnrObEzppSeWiFM_enmFvddxt8x1+TsOYLw@mail.gmail.com>
References: 
 <CAOiqz+ReYxOgsXnwnrObEzppSeWiFM_enmFvddxt8x1+TsOYLw@mail.gmail.com>
Subject: Re: Accumulo: "BigTable" vs. "Document Model"
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----=_Part_18313819_2146376538.1441382277778"
Thread-Topic: Accumulo: "BigTable" vs. "Document Model"
Thread-Index: L9ByHcyzlQceN+sc8IbEj+Qqz/0oyg==

------=_Part_18313819_2146376538.1441382277778
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


Both will work, but I think the answer depends on the amount of data that you will be querying over and your query latency requirements. I would include the wikisearch[1] storage scheme into your list as well (k/v table + indices). Then, personally, I would rate them in the following order as database size increases and query latency requirements decrease: 

1. Document Model 
2. K/V Model 
3. K/V Model with indices (wikisearch) 

[1] https://accumulo.apache.org/example/wikisearch.html 

----- Original Message -----

From: "Michael Moss" <michael.moss@gmail.com> 
To: user@accumulo.apache.org 
Sent: Friday, September 4, 2015 11:42:20 AM 
Subject: Accumulo: "BigTable" vs. "Document Model" 

Hello, everyone. 

I'd love to hear folks' input on using the "natural" data model of Accumulo ("BigTable" style) vs more of a Document Model. I'll try to succinctly describe with a contrived example. 

Let's say I have one domain object I'd like to model, "SensorReadings". A single entry might look something like the following with 4 distinct CF, CQ pairs. 

RowKey: DeviceID-YYYMMDD-ReadingID (i.e. - 1-20150101-1234) 
CF: "Meta", CQ: "Timestamp", Value: <Some timestamp> 
CF: "Sensor", CQ: "Temperature", Value: 80.4 
CF: "Sensor", CQ: "Humidity", Value: 40.2 
CF: "Sensor", CQ: "Barometer", Value: 29.1 

I might do queries like "get me all SensorReadings for 2015 for DeviceID = 1" and if I wanted to operate on each SensorReading as a single unit (and not as the 4 'rows' it returns for each one), I'd either have to aggregate the 4 CF, CQ pairs for each RowKey client side, or use something like the WholeRowIterator. 

In addition, if I wanted to write a query like, "for DeviceID = 1 in 2015, return me SensorReadings where Temperature > 90, Humidity < 40, Barometer > 31", I'd again have to either use the WholeRowIterator to 'see' each entire SensorReading in memory on the server for the compound query, or I could take the intersection of the results of 3 parallel, independent queries on the client side. 

Where I am going with this is, what are the thoughts around creating a Java, Protobuf, Avro (etc) object with these 4 CF, CQ pairs as fields and storing each SensorReading as a single 'Document'? 

RowKey: DeviceID-YYYMMDD 
CF: ReadingID Value: Protobuf(Timestamp=123, Temperature=80.4, Humidity=40.2, Barometer = 29.1) 

This way you avoid having to use the WholeRowIterator and unless you often have queries that only look at a tiny subset of your fields (let's say just "Temperature"), the serialization costs seem similar since Value is just bytes anyway. 

Appreciate folks' experience and wisdom here. Hope this makes sense, happy to clarify. 

Best. 

-Mike 


------=_Part_18313819_2146376538.1441382277778
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: quoted-printable

<html><body><div style=3D"font-family: Arial; font-size: 12pt; color: #0000=
00"><div><br></div><div>&nbsp;Both will work, but I think the answer depend=
s on the amount of data that you will be querying over and your query laten=
cy requirements. I would include the wikisearch[1] storage scheme into your=
 list as well (k/v table + indices). Then, personally, I would rate them in=
 the following order as database size increases and query latency requireme=
nts decrease:<br></div><div><br></div><div>&nbsp;1. Document Model<br></div=
><div>&nbsp;2. K/V Model<br></div><div>&nbsp;3. K/V Model with indices (wik=
isearch)<br></div><div><br></div><div>[1] https://accumulo.apache.org/examp=
le/wikisearch.html<br></div><div><br></div><hr id=3D"zwchr"><div style=3D"c=
olor:#000;font-weight:normal;font-style:normal;text-decoration:none;font-fa=
mily:Helvetica,Arial,sans-serif;font-size:12pt;"><b>From: </b>"Michael Moss=
" &lt;michael.moss@gmail.com&gt;<br><b>To: </b>user@accumulo.apache.org<br>=
<b>Sent: </b>Friday, September 4, 2015 11:42:20 AM<br><b>Subject: </b>Accum=
ulo: "BigTable" vs. "Document Model"<br><div><br></div><div dir=3D"ltr">Hel=
lo, everyone.<div><br></div><div>I'd love to hear folks' input on using the=
 "natural" data model of Accumulo ("BigTable" style) vs more of a Document =
Model. I'll try to succinctly describe with a contrived example.</div><div>=
<br></div><div>Let's say I have one domain object I'd like to model, "Senso=
rReadings". A single entry might look something like the following with 4 d=
istinct CF, CQ pairs.</div><div><br></div><div>RowKey: DeviceID-YYYMMDD-Rea=
dingID (i.e. - 1-20150101-1234)</div><div>CF: "Meta", CQ: "Timestamp", Valu=
e: &lt;Some timestamp&gt;</div><div>CF: "Sensor", CQ: "Temperature", Value:=
 80.4<br></div><div>CF: "Sensor", CQ: "Humidity", Value: 40.2</div><div>CF:=
 "Sensor", CQ: "Barometer", Value: 29.1</div><div><br></div><div>I might do=
 queries like "get me all SensorReadings for 2015 for DeviceID =3D 1" and i=
f I wanted to operate on each SensorReading as a single unit (and not as th=
e 4 'rows' it returns for each one), I'd either have to aggregate the 4 CF,=
 CQ pairs for each RowKey client side, or use something like the WholeRowIt=
erator.</div><div><br></div><div>In addition, if I wanted to write a query =
like, "for DeviceID =3D 1 in 2015, return me SensorReadings where Temperatu=
re &gt; 90, Humidity &lt; 40, Barometer &gt; 31", I'd again have to either =
use the WholeRowIterator to 'see' each entire SensorReading in memory on th=
e server for the compound query, or I could take the intersection of the re=
sults of 3 parallel, independent queries on the client side.</div><div><br>=
</div><div>Where I am going with this is, what are the thoughts around crea=
ting a Java, Protobuf, Avro (etc) object with these 4 CF, CQ pairs as field=
s and storing each SensorReading as a single 'Document'?&nbsp;</div><div><b=
r></div><div>RowKey: DeviceID-YYYMMDD</div><div>CF: ReadingID Value: Protob=
uf(Timestamp=3D123, Temperature=3D80.4, Humidity=3D40.2, Barometer =3D 29.1=
)</div><div><br></div><div>This way you avoid having to use the WholeRowIte=
rator and unless you often have queries that only look at a tiny subset of =
your fields (let's say just "Temperature"), the serialization costs seem si=
milar since Value is just bytes anyway.</div><div><br></div><div>Appreciate=
 folks' experience and wisdom here. Hope this makes sense, happy to clarify=
.</div><div><br></div><div>Best.</div><div><br></div><div>-Mike</div><div><=
br></div><div><br></div><div><br></div><div><br></div></div>
</div><div><br></div></div></body></html>
------=_Part_18313819_2146376538.1441382277778--