Return-Path: Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: (qmail 78823 invoked from network); 5 May 2010 10:06:13 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 5 May 2010 10:06:13 -0000 Received: (qmail 40910 invoked by uid 500); 5 May 2010 10:06:12 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 40828 invoked by uid 500); 5 May 2010 10:06:11 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 40818 invoked by uid 99); 5 May 2010 10:06:10 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 05 May 2010 10:06:10 +0000 X-ASF-Spam-Status: No, hits=2.2 required=10.0 tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of dsimeonov@gmail.com designates 209.85.211.190 as permitted sender) Received: from [209.85.211.190] (HELO mail-yw0-f190.google.com) (209.85.211.190) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 05 May 2010 10:06:03 +0000 Received: by ywh28 with SMTP id 28so2447711ywh.28 for ; Wed, 05 May 2010 03:05:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:received:in-reply-to :references:date:message-id:subject:from:to:content-type; bh=qPLx8gWS0UoHOjc+05bQGCPUJsPrGX0kCX00Af0rSkY=; b=Ig4NztjU7dq2J3gjhkWUQ495Yi4lnSzicG8JJh7YJGGNbiZPby6h+DOg1hJHhj6Wef 3vwOyf7Kx5nQ2DynzSkEaG2hsXF4iNOcpXitz4A+TUPrDQCxvYQe9Q7vP1UEd5g6kOkj YQrHtaiwMV1c8lz4sef2nqVKKYfjRRf2Aaoss= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; b=xz0uJ7d8NWN3hm0KaehRLKvuMWwLWr/fYeMdTMDPzeOupRWZjqWF13Lj34IckU73JI ybf75Md+amT4iv4iWJCcXO2XVv3lnWGlHOaBpheOI9qV0lWYSrD5DL7XVtMnte+wfjX5 Wc6l54+df8rpRjIpmT8yy9XI0g74T9CNgg2PQ= MIME-Version: 1.0 Received: by 10.101.218.9 with SMTP id v9mr5553394anq.83.1273053939212; Wed, 05 May 2010 03:05:39 -0700 (PDT) Received: by 10.100.152.11 with HTTP; Wed, 5 May 2010 03:05:39 -0700 (PDT) In-Reply-To: References: <10706384-61F5-48C1-B86C-C473AAF3089D@ucsfcti.org> <3FF7CD22-CEF6-47D6-89E5-B238EA9DE964@gmail.com> Date: Wed, 5 May 2010 13:05:39 +0300 Message-ID: Subject: Re: Best way to store millisecond-accurate data From: =?UTF-8?B?0JTQsNC90LjQtdC7INCh0LjQvNC10L7QvdC+0LI=?= To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=001636eee1f9d3d46a0485d5f75d X-Virus-Checked: Checked by ClamAV on apache.org --001636eee1f9d3d46a0485d5f75d Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Hi "In practice, one would want to model their data such that the 'row has too much columns' scenario is prevented." I am curious how really to prevent this, if the data is sharded with one day granularity, nothing stops the client to insert enormous amount of new columns (very often it is not possible to foreseen how much data clients would insert) then some functionality is needed prevent too much columns in a row (too much depends on the data), then such runtime sharding in necessary (to split the day granulary to two rows). I still think if this runtime sharding is possible in cassandra. Best regards, Daniel. 2010/5/4 Miguel Verde > One would use batch processes (e.g. through Hadoop) or client-side > aggregation, yes. In theory it would be possible to introduce runtime > sharding across rows into the Cassandra server side, but it's not part of > its design. > > In practice, one would want to model their data such that the 'row has to= o > much columns' scenario is prevented. > > On May 4, 2010, at 8:06 AM, =D0=94=D0=B0=D0=BD=D0=B8=D0=B5=D0=BB =D0=A1= =D0=B8=D0=BC=D0=B5=D0=BE=D0=BD=D0=BE=D0=B2 wrote: > > Hi Miguel, > I'd like to ask is it possible to have runtime sharding or rows in > cassandra, i.e. if the row has too much new columns inserted then create > another one row (let's say if the original timesharding is one day per ro= w, > then we would have two rows for that day). Maybe batch processes could do > that. > Best regards, Daniel. > > 2010/4/24 Miguel Verde < miguelitovert@gmail.com= > > >> TimeUUID's time component is measured in 100-nanosecond intervals. The >> library you use might calculate it with poorer accuracy or precision, bu= t >> from a storage/comparison standpoint in Cassandra millisecond data is ea= sily >> captured by it. >> >> One typical way of dealing with the data explosion of sampled time serie= s >> data is to bucket/shard rows (i.e. Bob-20100423-bloodpressure) so that y= ou >> put an upper bound on the row length. >> >> >> On Apr 23, 2010, at 7:01 PM, Andrew Nguyen < >> andrew-lists-cassandra@ucsfcti.org> wrote: >> >> Hello, >>> >>> I am looking to store patient physiologic data in Cassandra - it's bein= g >>> collected at rates of 1 to 125 Hz. I'm thinking of storing the timesta= mps >>> as the column names and the patient/parameter combo as the row key. Fo= r >>> example, Bob is in the ICU and is currently having his blood pressure, >>> intracranial pressure, and heart rate monitored. I'd like to collect t= his >>> with the following row keys: >>> >>> Bob-bloodpressure >>> Bob-intracranialpressure >>> Bob-heartrate >>> >>> The column names would be timestamps but that's where my questions star= t: >>> >>> I'm not sure what the best data type and CompareWith would be. From my >>> searching, it sounds like the TimeUUID may be suitable but isn't really >>> designed for millisecond accuracy. My other thought is just to store t= hem >>> as strings (2010-04-23 10:23:45.016). While I space isn't the foremost >>> concern, we will be collecting this data 24/7 so we'll be creating many >>> columns over the long-term. >>> >>> I found >>> https://issues.apache.org/jira/browse/CASSANDRA-16 which states that th= e >>> entire row must fit in memory. Does this include the values as well as= the >>> column names? >>> >>> In considering the limits of cassandra and the best way to model this, = we >>> would be adding 3.9 billion rows per year (assuming 125 Hz @ 24/7). >>> However, I can't really think of a better way to model this... So, am= I >>> thinking about this all wrong or am I on the right track? >>> >>> Thanks, >>> Andrew >>> >> > --001636eee1f9d3d46a0485d5f75d Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Hi
=C2=A0=C2=A0 =C2=A0"In practice, one would want to model their = data such that the 'row has too much columns' scenario is prevented= ."
=C2=A0=C2=A0 I am curious how really to prevent this, if = the data is sharded with one day granularity, nothing stops the client to i= nsert enormous amount of new columns (very often it is not possible to fore= seen how much data clients would insert) then some functionality is needed = prevent too much columns in a row (too much depends on the data), then such= runtime sharding in necessary (to split the day granulary to two rows). I = still think if this runtime sharding is possible in cassandra.
Best regards, Daniel.

2010/5/4 Miguel Verde <= miguelitovert@= gmail.com>
One would use batch processes (e.g. through H= adoop) or client-side aggregation, yes. In theory it would be possible to i= ntroduce runtime sharding across rows into the Cassandra server side, but i= t's not part of its design.

In practice, one would want to model their data such th= at the 'row has too much columns' scenario is prevented.

On May 4, 2010, at 8:06 AM, =D0=94=D0=B0=D0= =BD=D0=B8=D0=B5=D0=BB =D0=A1=D0=B8=D0=BC=D0=B5=D0=BE=D0=BD=D0=BE=D0=B2 <= dsimeonov@gmail.co= m> wrote:

Hi Miguel,
=C2=A0= =C2=A0I'd like to ask is it possible to have runtime sharding or rows i= n cassandra, i.e. if the row has too much new columns inserted then create = another one row (let's say if the original timesharding is one day per = row, then we would have two rows for that day). Maybe batch processes could= do that.=C2=A0
Best regards, Daniel.

2010/4/24 Migu= el Verde <miguelitovert@gmail.com>
TimeUUID's time component is measured in 100-nanosecond intervals. The = library you use might calculate it with poorer accuracy or precision, but f= rom a storage/comparison standpoint in Cassandra millisecond data is easily= captured by it.

One typical way of dealing with the data explosion of sampled time series d= ata is to bucket/shard rows (i.e. Bob-20100423-bloodpressure) so that you p= ut an upper bound on the row length.


On Apr 23, 2010, at 7:01 PM, Andrew Nguyen <andrew-lists-cassandra@ucsfcti= .org> wrote:

Hello,

I am looking to store patient physiologic data in Cassandra - it's bein= g collected at rates of 1 to 125 Hz. =C2=A0I'm thinking of storing the = timestamps as the column names and the patient/parameter combo as the row k= ey. =C2=A0For example, Bob is in the ICU and is currently having his blood = pressure, intracranial pressure, and heart rate monitored. =C2=A0I'd li= ke to collect this with the following row keys:

Bob-bloodpressure
Bob-intracranialpressure
Bob-heartrate

The column names would be timestamps but that's where my questions star= t:

I'm not sure what the best data type and CompareWith would be. =C2=A0Fr= om my searching, it sounds like the TimeUUID may be suitable but isn't = really designed for millisecond accuracy. =C2=A0My other thought is just to= store them as strings (2010-04-23 10:23:45.016). =C2=A0While I space isn&#= 39;t the foremost concern, we will be collecting this data 24/7 so we'l= l be creating many columns over the long-term.

I found https://issues.apache.org/jira/browse/CASSANDRA-16<= /a> which states that the entire row must fit in memory. =C2=A0Does this in= clude the values as well as the column names?

In considering the limits of cassandra and the best way to model this, we w= ould be adding 3.9 billion rows per year (assuming 125 Hz @ 24/7). =C2=A0Ho= wever, I can't really think of a better way to model this... =C2=A0So, = am I thinking about this all wrong or am I on the right track?

Thanks,
Andrew


--001636eee1f9d3d46a0485d5f75d--