hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gautam Borah <gautam.bo...@gmail.com>
Subject best approach for write and immediate read use case
Date Fri, 23 Aug 2013 07:11:10 GMT
Hello all,

I have an use case where I need to write 1 million to 10 million records
periodically (with intervals of 1 minutes to 10 minutes), into an HBase
table.

Once the insert is completed, these records are queried immediately from
another program - multiple reads.

So, this is one massive write followed by many reads.

I have two approaches to insert these records into the HBase table -

Use HTable or HTableMultiplexer to stream the data to HBase table.

or

Write the data to HDFS store as a sequence file (avro in my case) - run map
reduce job using HFileOutputFormat and then load the output files into
HBase cluster.
Something like,

  LoadIncrementalHFiles loader = new LoadIncrementalHFiles(conf);
  loader.doBulkLoad(new Path(outputDir), hTable);


In my use case which approach would be better?

If I use HTable interface, would the inserted data be in the HBase cache,
before flushing to the files, for immediate read queries?

If I use map reduce job to insert, would the data be loaded into the HBase
cache immediately? or only the output files would be copied to respective
hbase table specific directories?

So, which approach is better for write and then immediate multiple read
operations?

Thanks,
Gautam

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message