hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Balaji Rao <sbalaji...@gmail.com>
Subject Re: Reduce operation stuck
Date Sun, 20 May 2012 16:36:48 GMT
There is only one task attempt that is lurking around and that never
completes. It probably times out or fails after a long time.

-Balaji

On Fri, May 18, 2012 at 11:50 PM, Ranjith <ranjith.raghunath1@gmail.com> wrote:
> This could be a result of the data being skewed. When you look at the job tracker page
are you noticing just a few lurking around?
>
> Thanks,
> Ranjith
>
> On May 18, 2012, at 1:57 PM, Balaji Rao <sbalajirao@gmail.com> wrote:
>
>> I have a simple query running on HIVE using Amazon EMR where the
>> reduce operation seems to be stuck. The table is small with just
>> 300,000+ rows (data_analysis).
>> I intend to work on a 30,000,000 row table once I'm able to fix this.
>> The query (modified to hide some information) is as follows:
>>
>> INSERT OVERWRITE TABLE data_analysis
>> SELECT
>>  a.clientid,  a.isi,  a.os,  a.bro,  a.engaged,  a.iframed,
>>  hover, hower1, hower5, hower60,
>>  Visible,  a.x1,  a.x2,  a.x3,
>>  NoViewMeasure,  a.x4,  a.x5,  a.x6,
>>  a.alerttype,    a.x7,  a.x9,  a.x11,  a.x13,  a.x17,  a.x21,  a.x27,  a.x28,
>>  COUNT(a.auction_id_64),  SUM(CASE WHEN b.auction_id_64 IS NULL THEN
>> 1 ELSE 0 END)
>> FROM
>>  data_subset_2 a
>> LEFT OUTER JOIN
>>  OrderFields b on a.auction_id_64=b.auction_id_64
>> GROUP BY
>>  a.clientid,  a.isi,  a.os,  a.bro,  a.engaged,  a.iframed,
>>  hover, hower1, hower5, hower60,
>>  Visible,  a.x1,  a.x2,  a.x3,
>>  NoViewMeasure,  a.x4,  a.x5,  a.x6,
>>  a.alerttype,    a.x7,  a.x9,  a.x11,  a.x13,  a.x17,  a.x21,  a.x27,  a.x28;
>>
>>
>>
>>
>> 2012-05-18 17:49:32,440 Stage-1 map = 100%,  reduce = 97%
>> 2012-05-18 17:50:32,813 Stage-1 map = 100%,  reduce = 97%
>> 2012-05-18 17:51:33,176 Stage-1 map = 100%,  reduce = 97%
>> 2012-05-18 17:52:33,538 Stage-1 map = 100%,  reduce = 97%
>> 2012-05-18 17:53:33,906 Stage-1 map = 100%,  reduce = 97%
>> 2012-05-18 17:54:34,270 Stage-1 map = 100%,  reduce = 97%
>> 2012-05-18 17:55:34,644 Stage-1 map = 100%,  reduce = 97%
>> 2012-05-18 17:56:35,017 Stage-1 map = 100%,  reduce = 97%
>> 2012-05-18 17:57:35,387 Stage-1 map = 100%,  reduce = 97%
>> 2012-05-18 17:58:35,754 Stage-1 map = 100%,  reduce = 97%
>> 2012-05-18 17:59:36,125 Stage-1 map = 100%,  reduce = 97%
>> 2012-05-18 18:00:36,503 Stage-1 map = 100%,  reduce = 97%
>> 2012-05-18 18:01:36,873 Stage-1 map = 100%,  reduce = 97%
>> 2012-05-18 18:02:36,990 Stage-1 map = 100%,  reduce = 97%
>> 2012-05-18 18:03:37,355 Stage-1 map = 100%,  reduce = 97%
>> 2012-05-18 18:04:37,719 Stage-1 map = 100%,  reduce = 97%
>> 2012-05-18 18:05:38,081 Stage-1 map = 100%,  reduce = 97%
>> 2012-05-18 18:06:38,432 Stage-1 map = 100%,  reduce = 97%
>> 2012-05-18 18:07:38,777 Stage-1 map = 100%,  reduce = 97%
>> 2012-05-18 18:08:39,135 Stage-1 map = 100%,  reduce = 97%
>> 2012-05-18 18:09:39,489 Stage-1 map = 100%,  reduce = 97%
>> 2012-05-18 18:10:39,852 Stage-1 map = 100%,  reduce = 97%
>> 2012-05-18 18:11:40,221 Stage-1 map = 100%,  reduce = 97%
>> 2012-05-18 18:12:40,591 Stage-1 map = 100%,  reduce = 97%
>> 2012-05-18 18:13:40,962 Stage-1 map = 100%,  reduce = 97%
>> 2012-05-18 18:14:41,336 Stage-1 map = 100%,  reduce = 97%
>> 2012-05-18 18:15:41,700 Stage-1 map = 100%,  reduce = 97%
>> 2012-05-18 18:16:42,070 Stage-1 map = 100%,  reduce = 97%
>> 2012-05-18 18:17:42,440 Stage-1 map = 100%,  reduce = 97%
>> 2012-05-18 18:18:42,801 Stage-1 map = 100%,  reduce = 97%
>> 2012-05-18 18:19:43,156 Stage-1 map = 100%,  reduce = 97%
>> 2012-05-18 18:20:43,531 Stage-1 map = 100%,  reduce = 97%
>> 2012-05-18 18:21:43,899 Stage-1 map = 100%,  reduce = 97%
>> 2012-05-18 18:22:44,255 Stage-1 map = 100%,  reduce = 97%
>> 2012-05-18 18:23:44,619 Stage-1 map = 100%,  reduce = 97%
>> 2012-05-18 18:24:44,978 Stage-1 map = 100%,  reduce = 97%
>> 2012-05-18 18:25:45,347 Stage-1 map = 100%,  reduce = 97%
>> 2012-05-18 18:26:45,720 Stage-1 map = 100%,  reduce = 97%
>> 2012-05-18 18:27:46,088 Stage-1 map = 100%,  reduce = 97%
>> 2012-05-18 18:28:46,446 Stage-1 map = 100%,  reduce = 97%
>> 2012-05-18 18:29:46,814 Stage-1 map = 100%,  reduce = 97%
>> 2012-05-18 18:30:47,183 Stage-1 map = 100%,  reduce = 97%
>> 2012-05-18 18:31:47,546 Stage-1 map = 100%,  reduce = 97%
>> 2012-05-18 18:32:47,918 Stage-1 map = 100%,  reduce = 97%
>> 2012-05-18 18:33:48,282 Stage-1 map = 100%,  reduce = 97%
>> 2012-05-18 18:34:48,408 Stage-1 map = 100%,  reduce = 97%
>> 2012-05-18 18:35:48,766 Stage-1 map = 100%,  reduce = 97%
>> 2012-05-18 18:36:49,118 Stage-1 map = 100%,  reduce = 97%
>>
>> Thanks,
>> Balaji

Mime
View raw message