Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (athena.apache.org: domain of kranthili2020@gmail.com
 designates 209.85.212.41 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <D15D98CC-3F15-4734-9227-C8EB48D857E0@gmail.com>
References: 
 <CADE5hJHx9xV8nd1teRG27Y2RfcbrPQVCqxyQSRUyG3RBuDWz6g@mail.gmail.com>
	<D15D98CC-3F15-4734-9227-C8EB48D857E0@gmail.com>
Date: Mon, 5 Dec 2011 10:53:16 +0530
Message-ID: 
 <CADE5hJEbF7YzN_WMGsbAd7z=q9Qe38886OU5sGQf0yBaVHne9g@mail.gmail.com>
Subject: Re: Unexpected Data insertion time and Data size explosion
From: kranthi reddy <kranthili2020@gmail.com>
To: yuzhihong@gmail.com
Cc: "user@hbase.apache.org" <user@hbase.apache.org>
Content-Type: multipart/alternative; boundary=bcaec546911f19cb7c04b35184c5

--bcaec546911f19cb7c04b35184c5
Content-Type: text/plain; charset=ISO-8859-1

No, I split the table on the fly. This I have done because converting my
table into Hbase format (rowID, family, qualifier, value) would result in
the input file being arnd 300GB. Hence, I had decided to do the splitting
and generating this format on the fly.

Will this effect the performance so heavily ???

On Mon, Dec 5, 2011 at 1:21 AM, <yuzhihong@gmail.com> wrote:

> May I ask whether you pre-split your table before loading ?
>
>
>
> On Dec 4, 2011, at 6:19 AM, kranthi reddy <kranthili2020@gmail.com> wrote:
>
> > Hi all,
> >
> >    I am a newbie to Hbase and Hadoop. I have setup a cluster of 4
> machines
> > and am trying to insert data. 3 of the machines are tasktrackers, with 4
> > map tasks each.
> >
> >    My data consists of about 1.3 billion rows with 4 columns each (100GB
> > txt file). The column structure is "rowID, word1, word2, word3".  My DFS
> > replication in hadoop and hbase is set to 3 each. I have put only one
> > column family and 3 qualifiers for each field (word*).
> >
> >    I am using the SampleUploader present in the HBase distribution. To
> > complete 40% of the insertion, it has taken around 21 hrs and it's still
> > running. I have 12 map tasks running.* I would like to know is the
> > insertion time taken here on expected lines ??? Because when I used
> lucene,
> > I was able to insert the entire data in about 8 hours.*
> >
> >    Also, there seems to be huge explosion of data size here. With a
> > replication factor of 3 for HBase, I was expecting the table size
> inserted
> > to be around 350-400GB. (350-400GB for an 100GB txt file I have, 300GB
> for
> > replicating the data 3 times and 50+ GB for additional storage
> > information). But even for 40% completion of data insertion, the space
> > occupied is around 550GB (Looks like it might take around 1.2TB for an
> > 100GB file).* I have used the rowID to be a String, instead of Long. Will
> > that account for such rapid increase in data storage???
> > *
> >
> > Regards,
> > Kranthi
>


-- 
Kranthi Reddy. B

http://www.setusoftware.com/setu/index.htm

--bcaec546911f19cb7c04b35184c5--