hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ashish Shinde <ash...@strandls.com>
Subject Re: Bulk upload an LZO compression
Date Thu, 27 Jan 2011 05:07:01 GMT
Hi Todd,

Thanks. The problem was I used the patch for multi column family bulk
upload first and then added the LZO modifications. The
code in the trunk for creating writers is different from this so my code
changes will not be equivalent for the trunk and the mutli column
family patch.

Any chances the multi column family patch will make it to the trunk
soon?

Anyways will generate a patch against the trunk as well and attach it
to the ticket.

Thanks and regards,
 - Ashish

On Wed, 26 Jan 2011 19:50:06 -0800
Todd Lipcon <todd@cloudera.com> wrote:

> On Wed, Jan 26, 2011 at 2:43 AM, Ashish Shinde <ashish@strandls.com>
> wrote:
> 
> > Hi,
> >
> > I am using 0.90.0 candidate_3 from
> > http://people.apache.org/~stack/hbase-0.90.0-candidate-3/
> >
> > and have patched it to run multi column family bulk upload from
> > ticket
> >
> > https://issues.apache.org/jira/browse/HBASE-1861
> >
> > Also created the ticket
> > https://issues.apache.org/jira/browse/HBASE-3474
> > to handle this.
> >
> > I modified the code but not sure how to generate a patch. The multi
> > column family code looks to be on 0.92.0 branch which I
> > can't find in hbase svn repo. how do I create the patch.
> >
> 
> 0.92 is just "trunk" in the svn repo - so you can generate your patch
> against that.
> 
> Or, if we want we can consider this a bug fix / compatible
> improvement, and we can apply it to both 0.90 (for 0.90.1) and trunk.
> But, we'll need a patch for trunk as well.
> 
> -Todd
> 
>  On Tue, 25
> > Jan 2011 10:00:00 +0530 Ashish Shinde <ashish@strandls.com> wrote:
> >
> > > Hi,
> > >
> > > Yup after some digging I got to HFileOutputFormat and was
> > > relieved to know that it does support compression. Was able to
> > > add code to set compression based on the column family's
> > > compression setting.
> > >
> > > Will create a ticket and submit the patch after some more testing
> > > and going over the coding guidelines. My code looks a little
> > > hacky because I am passing the family specific compression
> > > algorithm name as "," delimited single configuration item. I
> > > figure that Configuration should have a method to return all key
> > > values where key's match a pattern. Maybe there are better ways
> > > to do this. Will get this into the ticket.
> > >
> > > Thanks and regards,
> > >  - Ashish
> > >
> > >  On Mon, 24 Jan 2011 11:12:06 -0800
> > > Todd Lipcon <todd@cloudera.com> wrote:
> > >
> > > > On Mon, Jan 24, 2011 at 9:50 AM, Stack <stack@duboce.net> wrote:
> > > >
> > > > > In HFileOutputFormat it says this near top:
> > > > >
> > > > >    // Invented config.  Add to hbase-*.xml if other than
> > > > > default compression.
> > > > >    final String compression = conf.get("hfile.compression",
> > > > >      Compression.Algorithm.NONE.getName());
> > > > >
> > > > > You might try messing with this config?
> > > > >
> > > >
> > > > And would be great to file (and provide a patch for) a JIRA that
> > > > automatically sets this based on the HTableDescriptor when
> > > > you're loading into an existing table!
> > > >
> > > > -Todd
> > > >
> > > >
> > > > > On Sun, Jan 23, 2011 at 9:38 PM, Ashish Shinde
> > > > > <ashish@strandls.com> wrote:
> > > > > > Hi,
> > > > > >
> > > > > > I have been importing data to hbase 0.90.0 using the code
> > > > > > from the bulk uploader (ImportTsv.java). The table has LZO
> > > > > > compression set, however unless major compaction is run the
> > > > > > table it does not get compressed.
> > > > > >
> > > > > > Is there a way to compress the table as the bulk uploader
> > > > > > creates the HFile. This is important for us because we don't
> > > > > > want to have a burst increase in our disk usage.
> > > > > >
> > > > > > Thanks and regards,
> > > > > >  - Ashish
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> >
> >
> 
> 


Mime
View raw message