Return-Path: Delivered-To: apmail-incubator-hama-dev-archive@minotaur.apache.org Received: (qmail 97106 invoked from network); 19 Mar 2009 08:58:12 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2) by minotaur.apache.org with SMTP; 19 Mar 2009 08:58:12 -0000 Received: (qmail 74536 invoked by uid 500); 19 Mar 2009 08:58:11 -0000 Delivered-To: apmail-incubator-hama-dev-archive@incubator.apache.org Received: (qmail 74523 invoked by uid 500); 19 Mar 2009 08:58:11 -0000 Mailing-List: contact hama-dev-help@incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: hama-dev@incubator.apache.org Delivered-To: mailing list hama-dev@incubator.apache.org Received: (qmail 74511 invoked by uid 99); 19 Mar 2009 08:58:11 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 Mar 2009 01:58:11 -0700 X-ASF-Spam-Status: No, hits=1.2 required=10.0 tests=SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (nike.apache.org: local policy) Received: from [209.85.198.251] (HELO rv-out-0708.google.com) (209.85.198.251) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 Mar 2009 08:58:01 +0000 Received: by rv-out-0708.google.com with SMTP id b17so438222rvf.0 for ; Thu, 19 Mar 2009 01:57:39 -0700 (PDT) MIME-Version: 1.0 Sender: edward@udanax.org Received: by 10.141.45.16 with SMTP id x16mr676692rvj.290.1237453059157; Thu, 19 Mar 2009 01:57:39 -0700 (PDT) In-Reply-To: References: <25aacb800903180424k19dd2c39w933ebcf7c053dbcf@mail.gmail.com> Date: Thu, 19 Mar 2009 17:57:39 +0900 X-Google-Sender-Auth: f3f70424a73794d2 Message-ID: Subject: Re: Difference between sparse* and dense* From: "Edward J. Yoon" To: hama-dev@incubator.apache.org Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Virus-Checked: Checked by ClamAV on apache.org If you have any ideas, Pls let us know. On Wed, Mar 18, 2009 at 10:21 PM, Edward J. Yoon wr= ote: > Oh, good point. > > Hbase seems good fit for huge sparse matrcies. > > - Non-zero value > - Index for row and column > > However, It's too good for dense matrix. IMO, We can't store the huge > dense matrix to Hbase. When I store the 5000 * 5000 double matrix with > row/column/time index to Hbase, 15~16 GB was used for each nodes. > (replica =3D 3) So, I made a two implement. =C2=A0We should survey about = data > structures. > > And, There is also a difference of algorithms/benefits between Dense > and Sparse. > > - The blocking algorithm only work for Dense Matrix, And stores all. > - Sparse Matrix stores only non-zero value (storage efficient) but, If > sparsity is low, manipulations will have some overhead by irregular > access through network. > > I've start the work for documentation -- > http://wiki.apache.org/hama/Architecture -- Please also review this. > > On Wed, Mar 18, 2009 at 8:24 PM, Samuel Guo wrote: >> Hi all, >> >> It seems that DenseVector and SparseVector both use *MapWritable* as the >> container of vector data. And the methods' implementations of DenseVecto= r & >> SparseVector are similarly. so why we need two copies of the code? >> >> There are same issues in DenseMatrix and SparseMatrix. >> >> Regards, >> Samuel >> > > > > -- > Best Regards, Edward J. Yoon > edwardyoon@apache.org > http://blog.udanax.org > --=20 Best Regards, Edward J. Yoon edwardyoon@apache.org http://blog.udanax.org