From user-return-33253-apmail-cassandra-user-archive=cassandra.apache.org@cassandra.apache.org Fri Apr 5 18:38:25 2013 Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 2AD31FB65 for ; Fri, 5 Apr 2013 18:38:25 +0000 (UTC) Received: (qmail 39593 invoked by uid 500); 5 Apr 2013 18:38:22 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 39520 invoked by uid 500); 5 Apr 2013 18:38:22 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 39512 invoked by uid 99); 5 Apr 2013 18:38:22 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 05 Apr 2013 18:38:22 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: local policy) Received: from [66.111.4.221] (HELO new1-smtp.messagingengine.com) (66.111.4.221) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 05 Apr 2013 18:38:17 +0000 Received: from compute5.internal (compute5.nyi.mail.srv.osa [10.202.2.45]) by gateway1.nyi.mail.srv.osa (Postfix) with ESMTP id 7F19019C7 for ; Fri, 5 Apr 2013 14:37:56 -0400 (EDT) Received: from frontend2.nyi.mail.srv.osa ([10.202.2.161]) by compute5.internal (MEProxy); Fri, 05 Apr 2013 14:37:56 -0400 DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; d=venarc.com; h=from :content-type:message-id:mime-version:subject:date:references:to :in-reply-to; s=mesmtp; bh=HWA+0dtoCrqunAR6i9Qd4tVIPIU=; b=wVacA BFV6YQpXbPT4tvX9FiaYcIjxAUWn/zyk1nHuQ08QCp8GZLqG84cVqB6mx15q9oCp MOoz7o4BRc6D1jZPhqFhAH5yhCg3SN2N5I4XFT+/hdaaeKfSAzUisbD/bd3+shsO XhtxKA1oRtTBm8NJA2u/afa5B/sYw7zjHxUXz0= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; d= messagingengine.com; h=from:content-type:message-id:mime-version :subject:date:references:to:in-reply-to; s=smtpout; bh=HWA+0dtoC rqunAR6i9Qd4tVIPIU=; b=ZtoLeCePprBUtl8FVQflGXUNjTxfTbdU24jHx7nww 7/7xQudwXQ5F35JQwIEKosEGvOqb8VhrXLAxr/0fXS3wOG7i0k6k0laLr/mlPbRB hhltylNufIRtTv3P7St48xg2bO9ArfAmcqwZicF0qBJnSX+3KTfXUZYvDB2pllCB iI= X-Sasl-enc: Umw8Y7yFBtOlXJWUnXuBQ82/LWwBYZ0/TT/ymnNtJhc3 1365187075 Received: from [192.168.25.101] (unknown [24.176.202.50]) by mail.messagingengine.com (Postfix) with ESMTPA id 7474C200183 for ; Fri, 5 Apr 2013 14:37:55 -0400 (EDT) From: Drew Kutcharian Content-Type: multipart/alternative; boundary="Apple-Mail=_5879A812-D7D7-46F3-BC6F-251ACCD8EF0A" Message-Id: Mime-Version: 1.0 (Mac OS X Mail 6.2 \(1499\)) Subject: Re: Data Modeling: How to keep track of arbitrarily inserted column names? Date: Fri, 5 Apr 2013 11:37:54 -0700 References: <682A0397-D205-4A46-BBDD-1C0F27DE7762@venarc.com> <65958708-4EBB-4EB8-8C6A-C0E7EF81082D@venarc.com> To: user@cassandra.apache.org In-Reply-To: X-Mailer: Apple Mail (2.1499) X-Virus-Checked: Checked by ClamAV on apache.org --Apple-Mail=_5879A812-D7D7-46F3-BC6F-251ACCD8EF0A Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=windows-1252 One thing I can do is to have a client-side cache of the keys to reduce = the number of updates. On Apr 5, 2013, at 6:14 AM, Edward Capriolo = wrote: > Since there are few column names what you can do is this. Make a = reverse index, low read repair chance, Be aggressive with compaction. It = will be many extra writes but that is ok.=20 >=20 > Other option is turn on row cache and try read before write. It is a = good case for row cache because it is a very small data set. >=20 > On Thursday, April 4, 2013, Drew Kutcharian wrote: > > I don't really need to answer "what rows contain column named X", so = no need for a reverse index here. All I want is a distinct set of all = the column names, so I can answer "what are all the available column = names" > > > > On Apr 4, 2013, at 4:20 PM, Edward Capriolo = wrote: > > > > Your reverse index of "which rows contain a column named X" will = have very wide rows. You could look at cassandra's secondary indexing, = or possibly look at a solandra/solr approach. Another option is you can = shift the problem slightly, "which rows have column X that was added = between time y and time z". Remember with few distinct column names that = reverse index of column to row is going to be a very big list. > > > > > > On Thu, Apr 4, 2013 at 5:45 PM, Drew Kutcharian = wrote: > >> > >> Hi Edward, > >> I anticipate that the column names will be reused a lot. For = example, key1 will be in many rows. So I think the number of distinct = column names will be much much smaller than the number of rows. Is there = a way to have a separate CF that keeps track of the column names?=20 > >> What I was thinking was to have a separate CF that I write only the = column name with a null value in there every time I write a key/value to = the main CF. In this case if that column name exist, then it will just = be overridden. Now if I wanted to get all the column names, then I can = just query that CF. Not sure if that's the best approach at high load = (100k inserts a second). > >> -- Drew > >> > >> On Apr 4, 2013, at 12:02 PM, Edward Capriolo = wrote: > >> > >> You can not get only the column name (which you are calling a key) = you can use get_range_slice which returns all the columns. When you = specify an empty byte array (new byte[0]{}) as the start and finish you = get back all the columns. =46rom there you can return only the columns = to the user in a format that you like. > >> > >> > >> On Thu, Apr 4, 2013 at 2:18 PM, Drew Kutcharian = wrote: > >>> > >>> Hey Guys, > >>> > >>> I'm working on a project and one of the requirements is to have a = schema free CF where end users can insert arbitrary key/value pairs per = row. What would be the best way to know what are all the "keys" that = were inserted (preferably w/o any locking). For example, > >>> > >>> Row1 =3D> key1 -> XXX, key2 -> XXX > >>> Row2 =3D> key1 -> XXX, key3 -> XXX > >>> Row3 =3D> key4 -> XXX, key5 -> XXX > >>> Row4 =3D> key2 -> XXX, key5 -> XXX > >>> =85 > >>> > >>> The query would be give me all the inserted keys and the response = would be {key1, key2, key3, key4, key5} > >>> > >>> Thanks, > >>> > >>> Drew > >>> > >> > >> > > > > > > --Apple-Mail=_5879A812-D7D7-46F3-BC6F-251ACCD8EF0A Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=windows-1252 One = thing I can do is to have a client-side cache of the keys to reduce the = number of updates.


On Apr 5, 2013, at = 6:14 AM, Edward Capriolo <edlinuxguru@gmail.com> = wrote:

Since there are few column names what you can do is this. = Make a reverse index, low read repair chance, Be aggressive with = compaction. It will be many extra writes but that is ok.

Other = option is turn on row cache and try read before write. It is a good case = for row cache because it is a very small data set.

On Thursday, April 4, 2013, Drew Kutcharian <drew@venarc.com> wrote:
> I = don't really need to answer "what rows contain column named X", so no = need for a reverse index here. All I want is a distinct set of all the = column names, so I can answer "what are all the available column = names"
>
> On Apr 4, 2013, at 4:20 PM, Edward Capriolo <edlinuxguru@gmail.com> = wrote:
>
> Your reverse index of "which rows contain a = column named X" will have very wide rows. You could look at cassandra's = secondary indexing, or possibly look at a solandra/solr approach. = Another option is you can shift the problem slightly, "which rows have = column X that was added between time y and time z". Remember with few = distinct column names that reverse index of column to row is going to be = a very big list.
>
>
> On Thu, Apr 4, 2013 at 5:45 PM, Drew Kutcharian = <drew@venarc.com> = wrote:
>>
>> Hi Edward,
>> I anticipate that = the column names will be reused a lot. For example, key1 will be in many = rows. So I think the number of distinct column names will be much much = smaller than the number of rows. Is there a way to have a separate = CF that keeps track of the column names? 
>> What I was thinking was to have a separate CF that I write only = the column name with a null value in there every time I write a = key/value to the main CF. In this case if that column name exist, then = it will just be overridden. Now if I wanted to get all the column names, = then I can just query that CF. Not sure if that's the best approach at = high load (100k inserts a second).
>> -- Drew
>>
>> On Apr 4, 2013, at 12:02 PM, = Edward Capriolo <edlinuxguru@gmail.com> = wrote:
>>
>> You can not get only the column name = (which you are calling a key) you can use get_range_slice which returns = all the columns. When you specify an empty byte array (new byte[0]{}) as = the start and finish you get back all the columns. =46rom there you can = return only the columns to the user in a format that you like.
>>
>>
>> On Thu, Apr 4, 2013 at 2:18 PM, Drew = Kutcharian <drew@venarc.com> = wrote:
>>>
>>> Hey = Guys,
>>>
>>> I'm working on a project and one = of the requirements is to have a schema free CF where end users can = insert arbitrary key/value pairs per row. What would be the best way to = know what are all the "keys" that were inserted (preferably w/o any = locking). For example,
>>>
>>> Row1 =3D> key1 -> XXX, key2 -> = XXX
>>> Row2 =3D> key1 -> XXX, key3 -> = XXX
>>> Row3 =3D> key4 -> XXX, key5 -> = XXX
>>> Row4 =3D> key2 -> XXX, key5 -> XXX
>>> =85
>>>
>>> The query would be give = me all the inserted keys and the response would be {key1, key2, key3, = key4, key5}
>>>
>>> = Thanks,
>>>
>>> Drew
>>>
>>
>>
>
>
>

= --Apple-Mail=_5879A812-D7D7-46F3-BC6F-251ACCD8EF0A--