Mailing-List: contact dev-help@gora.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@gora.apache.org
Received-SPF: pass (nike.apache.org: domain of lewis.mcgibbney@gmail.com
 designates 209.85.223.176 as permitted sender)
MIME-Version: 1.0
Date: Thu, 26 Feb 2015 12:16:56 -0800
Message-ID: 
 <CAGaRif2FKLopQfJez4hMEXH2uWoDbuQw4oR4VFxOTh3CSRsv=g@mail.gmail.com>
Subject: Re: Dynamically generating HBase columns
From: Lewis John Mcgibbney <lewis.mcgibbney@gmail.com>
To: dev-digest@gora.apache.org
Cc: "<dev@gora.apache.org>" <dev@gora.apache.org>
Content-Type: multipart/alternative; boundary=001a11351bc6f8bc690510036e30

--001a11351bc6f8bc690510036e30
Content-Type: text/plain; charset=UTF-8

Hi Alfonso,

On Tue, Feb 24, 2015 at 10:27 PM, <dev-digest-help@gora.apache.org> wrote:

>
> In my use cases I always need a mix between static and dynamic columns.
> In my first week I tried to mix a Map over a column family overlapped with
> static columns. Didn't work because Gora was not prepared for that (and
> indeed needs thinking about it further).
>

Yeah. I've logged the following focus to deal with it
https://issues.apache.org/jira/browse/GORA-413


>
> What I do is separate the static columns in one column family (or serveral)
> from the dynamic stuff (that goes in a map). One Map is mapped to one
> column family in which each column:value is key=>value in the map.
> I have several maps depending on my needs, but can be just one big one with
> key=column.
>

Can you please show this graphically so I am absolutely clear on what you
are doing?


>
> What I don't fully understand is the timestamp you talk about, since we
> don't handle HBase timestamps. Do you specifically need it?
>

Yes, please read comment on GORA-413


>
> I'm not quite sure if I answer you :S
>

We will clarify it soon. Don;t worry ;)


>
> Something important to ask is much columns will you store in the column
> family?
>

Well dynamic columns will be added with every incoming chunk of data.


> Since we removed the StateManager, when you modify a map it deletes the
> column familiy and sends all the data again to be written (
>
> https://github.com/apache/gora/blob/master/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L289
> ),
> so adding/removing just one column can be quite killing when persisting
> several huge maps. About what volume and write pattern are we talking?
>

The volume of data will not be so large however it is concerning that
entire column families are deleted and re-written. It seems like a waste of
time and resources which we should address in an effort to make this a more
efficient process.
Thanks, lets take the discussion over to GORA-413

--001a11351bc6f8bc690510036e30--