incubator-hama-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Samuel Guo" <guosi...@gmail.com>
Subject Re: how to deal with more and more tables in hbase?
Date Sun, 21 Sep 2008 08:09:09 GMT
On Sun, Sep 21, 2008 at 12:15 AM, Edward J. Yoon <edwardyoon@apache.org>wrote:

> > as we know , when we create a matrix,  a new hbase table will be added in
> > hbase. after a lot of matrix operations, we will get more and more tables
> in
> > hbase. And some tables may be never used again after its matrix operation
> > finished. These tables will become orphans. Is it right? if not, plz let
> me
> > know.
>
> Yes. Some tables wont be opened and it may go to trash.
>
> > And I guess that most of matrix operations will be executed just as
> below:
> > load some matrices from files,  use hama to do mapred-style matrix
> > operations , and get some other matrices or information, then dump them
> to
> > files.
>
> More detail, I guess hama can be used like this:
>
> 1) raw data (txt file on HDFS or DBMS data or Hbase table or ..)
> 2) put the data into the matrix structure
>  - Matrix values can be declared by user arbitrary from the raw data
> 3) matrix operations using hama
> 4) save the results to file/table or .. for some applications
>
> This schedule can be a loop job for specified business logic or a ad hoc
> job.
>
> > Matrix in hbase just acts as a temp role. after we got the matrices in
> file,
> > they will become useless.
> >
> > Should we do some job to solve it?
>
> Almost yes...
> Actually, hbase will be used for the network communication on Map/Reduce.
>
> We thought about the table pre-allocation and gabarge collector for
> the ad-hoc jobs - https://issues.apache.org/jira/browse/HAMA-4


does it mean that we need a daemon which does the pre-allocation and gc work
and like *Master* in HBase and *JobTracker* in Hadoop?
Every Hama Instance should connect to the daemon to get matrices before
matrix computing.

Or every hama instance has its own manager and gc?


>
> If you have a good idea, please let us know. :)
>
> /Edward
>
>  --
> Best regards, Edward J. Yoon
> edwardyoon@apache.org
> http://blog.udanax.org
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message