hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Leif Wickland <leifwickl...@gmail.com>
Subject Re: HFiles and MapReduce
Date Mon, 01 Aug 2011 21:15:52 GMT
Thanks for the reply, J-D.


> > 1. Is there any case where it's a bad idea to use HFileOutputFormat
> instead
> > of TableOutputFormat when writing to HBase from MapReduce?
>
> Can you think of any?
>

Well, no, but I'm new to this stuff.


> That process isn't atomic, so to be sure you could end up with a
> region failing for some reason (network issues, bug, whatever) and my
> understanding of the code is that it would fail and return immediately
> after any IOException.
>

What would recovery from that scenario look like?  Would the un-adopted
HFiles remain in the directory that they were written to by the
HFileOutputFormat?



> You'd need to write an HFileInputFormat, that's pretty much it.
>

It looks like Tatsuya Kawano offered to write an
HFileInputFormat<http://mail-archives.apache.org/mod_mbox/hbase-dev/201101.mbox/%3CF02D6CF8-189A-4084-AD54-01EAC064C1CF@gmail.com%3E>back
in January. Does anyone know if he ended up sharing that?

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message