hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: M/R scan problem
Date Mon, 04 Jul 2011 14:22:29 GMT
Do you use TableInputFormat ?
To scan large number of rows, it would be better to produce one Split per
region.

What HBase version do you use ?
Do you find any exception in master / region server logs around the moment
of timeout ?

Cheers

On Mon, Jul 4, 2011 at 4:48 AM, Lior Schachter <liors@infolinks.com> wrote:

> Hi all,
> I'm running a scan using the M/R framework.
> My table contains hundreds of millions of rows and I'm scanning using
> start/stop key about 50 million rows.
>
> The problem is that some map tasks get stuck and the task manager kills
> these maps after 600 seconds. When retrying the task everything works fine
> (sometimes).
>
> To verify that the problem is in hbase (and not in the map code) I removed
> all the code from my map function, so it looks like this:
> public void map(ImmutableBytesWritable key, Result value, Context context)
> throws IOException, InterruptedException {
> }
>
> Also, when the map got stuck on a region, I tried to scan this region
> (using
> simple scan from a Java main) and it worked fine.
>
> Any ideas ?
>
> Thanks,
> Lior
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message