gora-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Lewis John Mcgibbney <lewis.mcgibb...@gmail.com>
Subject Re: Dynamically generating HBase columns
Date Thu, 26 Feb 2015 20:16:56 GMT
Hi Alfonso,

On Tue, Feb 24, 2015 at 10:27 PM, <dev-digest-help@gora.apache.org> wrote:

>
> In my use cases I always need a mix between static and dynamic columns.
> In my first week I tried to mix a Map over a column family overlapped with
> static columns. Didn't work because Gora was not prepared for that (and
> indeed needs thinking about it further).
>

Yeah. I've logged the following focus to deal with it
https://issues.apache.org/jira/browse/GORA-413


>
> What I do is separate the static columns in one column family (or serveral)
> from the dynamic stuff (that goes in a map). One Map is mapped to one
> column family in which each column:value is key=>value in the map.
> I have several maps depending on my needs, but can be just one big one with
> key=column.
>

Can you please show this graphically so I am absolutely clear on what you
are doing?


>
> What I don't fully understand is the timestamp you talk about, since we
> don't handle HBase timestamps. Do you specifically need it?
>

Yes, please read comment on GORA-413


>
> I'm not quite sure if I answer you :S
>

We will clarify it soon. Don;t worry ;)


>
> Something important to ask is much columns will you store in the column
> family?
>

Well dynamic columns will be added with every incoming chunk of data.


> Since we removed the StateManager, when you modify a map it deletes the
> column familiy and sends all the data again to be written (
>
> https://github.com/apache/gora/blob/master/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L289
> ),
> so adding/removing just one column can be quite killing when persisting
> several huge maps. About what volume and write pattern are we talking?
>

The volume of data will not be so large however it is concerning that
entire column families are deleted and re-written. It seems like a waste of
time and resources which we should address in an effort to make this a more
efficient process.
Thanks, lets take the discussion over to GORA-413

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message