incubator-hama-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Edward J. Yoon" <edwardy...@apache.org>
Subject Re: how to deal with more and more tables in hbase?
Date Sat, 20 Sep 2008 16:15:59 GMT
> as we know , when we create a matrix,  a new hbase table will be added in
> hbase. after a lot of matrix operations, we will get more and more tables in
> hbase. And some tables may be never used again after its matrix operation
> finished. These tables will become orphans. Is it right? if not, plz let me
> know.

Yes. Some tables wont be opened and it may go to trash.

> And I guess that most of matrix operations will be executed just as below:
> load some matrices from files,  use hama to do mapred-style matrix
> operations , and get some other matrices or information, then dump them to
> files.

More detail, I guess hama can be used like this:

1) raw data (txt file on HDFS or DBMS data or Hbase table or ..)
2) put the data into the matrix structure
  - Matrix values can be declared by user arbitrary from the raw data
3) matrix operations using hama
4) save the results to file/table or .. for some applications

This schedule can be a loop job for specified business logic or a ad hoc job.

> Matrix in hbase just acts as a temp role. after we got the matrices in file,
> they will become useless.
>
> Should we do some job to solve it?

Almost yes...
Actually, hbase will be used for the network communication on Map/Reduce.

We thought about the table pre-allocation and gabarge collector for
the ad-hoc jobs - https://issues.apache.org/jira/browse/HAMA-4

If you have a good idea, please let us know. :)

/Edward

On Sat, Sep 20, 2008 at 10:54 PM, Samuel Guo <guosijie@gmail.com> wrote:
> Hi all,
>
> as we know , when we create a matrix,  a new hbase table will be added in
> hbase. after a lot of matrix operations, we will get more and more tables in
> hbase. And some tables may be never used again after its matrix operation
> finished. These tables will become orphans. Is it right? if not, plz let me
> know.
>
> And I guess that most of matrix operations will be executed just as below:
> load some matrices from files,  use hama to do mapred-style matrix
> operations , and get some other matrices or information , then dump them to
> files.
>
> Matrix in hbase just acts as a temp role. after we got the matrices in file,
> they will become useless.
>
>
> regards,
>
> samuel
>



-- 
Best regards, Edward J. Yoon
edwardyoon@apache.org
http://blog.udanax.org

Mime
View raw message