hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From shashwat shriparv <dwivedishash...@gmail.com>
Subject Re: Problem doing bulk load programmatically
Date Thu, 24 May 2012 12:02:55 GMT
Super, i m happy that it solve your problem :) Cheer :)

On Thu, May 24, 2012 at 3:10 PM, sakin cali <sakinnncali@gmail.com> wrote:

> Thank you,
>
> As said the problem was about hdfs,
> putting dfs configuration directly into configuration object solved my
> problem
>
> --------------
> Configuration conf = HBaseConfiguration.create();
> conf.set("fs.default.name", "hdfs://master:54310");
>
> ..
> ..
> FileSystem fs = FileSystem.get(conf);
>
> -------------
>
> I have one more question,
> I have to get the execution time for bulk load
> Simply I get the time before and after the doBulkLoad method,
> But doBulkLoad spawns threads inside, Although I get a time difference, it
> will not be real execution time...
> What do you think?
>
> -------------
> start = System.currentTimeMillis();
> loader.doBulkLoad(dir,table);
> end = System.currentTimeMillis();
> diff= end-start;
> -------------
>
> On 24 May 2012 11:13, shashwat shriparv <dwivedishashwat@gmail.com> wrote:
>
> > your hbase is not able to communicate to hadoop, so before starting hbase
> > put the core-site.xml, hdfs-site.xml  and mapred-site.xml from hadoop
> conf
> > dir to hbase conf dir and then start hbase and try,, also you can copy
> > hadoop core from hadoop to hbase library..
> >
> > Regards
> > ∞
> >
> > Shashwat Shriparv
> >
> >
> >
> >
> > On Thu, May 24, 2012 at 1:41 PM, shashwat shriparv <
> > dwivedishashwat@gmail.com> wrote:
> >
> > > Check if hdfs is running fine and you can access it, and haddop is able
> > to
> > > access the file try putting that file on hdfs and then try.
> > >
> > > Regards
> > >
> > > ∞
> > > Shashwat Shriparv
> > >
> > >
> > >
> > > On Thu, May 24, 2012 at 12:35 PM, sakin cali <sakinnncali@gmail.com
> > >wrote:
> > >
> > >> Something new...
> > >>
> > >> When I start MiniDfsCluster and use its filesystem in my code,
> > >> bulk load finishes successfully..
> > >>
> > >> dfs = new MiniDFSCluster(conf, 2, true, (String[]) null);
> > >> 41.// set file system to the mini dfs just started up
> > >> 42.FileSystem fs = dfs.getFileSystem();
> > >>
> > >> I think I have a problem accessing running HDFS ..
> > >>
> > >>
> > >> On 24 May 2012 09:12, sakin cali <sakinnncali@gmail.com> wrote:
> > >>
> > >> > Hi,
> > >> >
> > >> > I am trying generate an HFile and load it to hbase programmatically.
> > >> > (importtsv does not fit my requirements)
> > >> >
> > >> > I have modified TestLoadIncrementalHFiles class and try to load an
> > >> example
> > >> > hfile.
> > >> >
> > >> > But I get errors at the last step
> > >> >
> > >> > --------------------------------------
> > >> > java.io.IOException: java.io.IOException: Failed rename of *
> > >> > file:/tmp/hbase/test/myfam/_tmp/mytable,1.top* to *
> > >> >
> > >>
> >
> file:/tmp/hbase-hbase/hbase/mytable/742248ed8cda9fe0dce2f345149fa8d5/myfam/8964877901933124837
> > >> > *
> > >> >  at
> > >> >
> > >>
> > org.apache.hadoop.hbase.regionserver.StoreFile.rename(StoreFile.java:512)
> > >> > ----------------------------------------
> > >> >
> > >> > Although such an error occured, renaming seems succesfully because
> > when
> > >> I
> > >> > checked their md5sum values, I saw that both have the same values...
> > >> >
> > >> > I could not figure out the problem,
> > >> > any ideas?
> > >> >
> > >> > I have attached test code and logs.
> > >> >
> > >> > I'am using cloudera's virtualbox vm image for testing environment
> > >> > ( 64 bit centos, version Cdh3u4 )
> > >> >
> > >> >
> > >> >
> > >>
> > >
> > >
> > >
> > > --
> > >
> > >
> > > ∞
> > > Shashwat Shriparv
> > >
> > >
> > >
> >
> >
> > --
> >
> >
> > ∞
> > Shashwat Shriparv
> >
>



-- 


∞
Shashwat Shriparv

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message