hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Grover <grover.markgro...@gmail.com>
Subject Re: Reduce phase hangs
Date Fri, 09 Nov 2012 07:19:30 GMT
It could be the order by since all the output records have to go through a
single reducer to get ordered.

Can you remove the order by and re-try the query?

You can read up more about what I am saying at
https://cwiki.apache.org/Hive/languagemanual-sortby.html

Mark

On Thu, Nov 8, 2012 at 5:23 AM, Robin Verlangen <robin@us2.nl> wrote:

> Hmm, after looking in the job&task-tracker web interfaces it seemed that
> one of the new nodes was unable to connect to two of the others. This
> caused the copying of data to "hang" (in fact: timeout, on timeout, on
> timeout, ...).
>
> Best regards,
>
> Robin Verlangen
> *Software engineer*
> *
> *
> W http://www.robinverlangen.nl
> E robin@us2.nl
>
> <http://goo.gl/Lt7BC>
>
> Disclaimer: The information contained in this message and attachments is
> intended solely for the attention and use of the named addressee and may be
> confidential. If you are not the intended recipient, you are reminded that
> the information remains the property of the sender. You must not use,
> disclose, distribute, copy, print or rely on this e-mail. If you have
> received this message in error, please contact the sender immediately and
> irrevocably delete this message and any copies.
>
>
>
> On Thu, Nov 8, 2012 at 2:15 PM, Mohammad Tariq <dontariq@gmail.com> wrote:
>
>> Maybe you need more heap or that nodes's health is not fine. Or maybe the
>> data block on that node is causing the code to go in a very long loop.
>> There can be several reasons. It's tough to say anything confidently
>> without knowing the real issue.
>>
>> Regards,
>>     Mohammad Tariq
>>
>>
>>
>> On Thu, Nov 8, 2012 at 5:39 PM, Robin Verlangen <robin@us2.nl> wrote:
>>
>>> It actually seems that only 1 of the 24 reducers hangs at the "copy"
>>> phase. Any solutions for this?
>>>
>>> Best regards,
>>>
>>> Robin Verlangen
>>> *Software engineer*
>>> *
>>> *
>>> W http://www.robinverlangen.nl
>>> E robin@us2.nl
>>>
>>> <http://goo.gl/Lt7BC>
>>>
>>> Disclaimer: The information contained in this message and attachments is
>>> intended solely for the attention and use of the named addressee and may be
>>> confidential. If you are not the intended recipient, you are reminded that
>>> the information remains the property of the sender. You must not use,
>>> disclose, distribute, copy, print or rely on this e-mail. If you have
>>> received this message in error, please contact the sender immediately and
>>> irrevocably delete this message and any copies.
>>>
>>>
>>>
>>> On Thu, Nov 8, 2012 at 11:47 AM, Robin Verlangen <robin@us2.nl> wrote:
>>>
>>>> Nothing special over there. Most of the jobs complete after quite some
>>>> time. However it makes no sense to me that it takes that long. Its probably
>>>> just a couple of hundreds of megabytes.
>>>>
>>>> Best regards,
>>>>
>>>> Robin Verlangen
>>>> *Software engineer*
>>>> *
>>>> *
>>>> W http://www.robinverlangen.nl
>>>> E robin@us2.nl
>>>>
>>>> <http://goo.gl/Lt7BC>
>>>>
>>>> Disclaimer: The information contained in this message and attachments
>>>> is intended solely for the attention and use of the named addressee and may
>>>> be confidential. If you are not the intended recipient, you are reminded
>>>> that the information remains the property of the sender. You must not use,
>>>> disclose, distribute, copy, print or rely on this e-mail. If you have
>>>> received this message in error, please contact the sender immediately and
>>>> irrevocably delete this message and any copies.
>>>>
>>>>
>>>>
>>>> On Thu, Nov 8, 2012 at 11:43 AM, Mohammad Tariq <dontariq@gmail.com>wrote:
>>>>
>>>>> Hello Robin,
>>>>>
>>>>>         Look at the TaskTracker logs and see if you find anything
>>>>> interesting there.
>>>>>
>>>>> Regards,
>>>>>     Mohammad Tariq
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Nov 8, 2012 at 4:03 PM, Robin Verlangen <robin@us2.nl>
wrote:
>>>>>
>>>>>> Hi there,
>>>>>>
>>>>>> It seems that some of my jobs hang in the reduce phase for a very
>>>>>> long time (for example, days). Is there anything I could tweak on?
The
>>>>>> query is pretty simple, like:
>>>>>>
>>>>>> SELECT SUM(colA), to_date(colB) AS dt FROM table GROUP BY
>>>>>> to_date(colB) ORDER BY dt ASC;
>>>>>>
>>>>>> Best regards,
>>>>>>
>>>>>> Robin Verlangen
>>>>>> *Software engineer*
>>>>>> *
>>>>>> *
>>>>>> W http://www.robinverlangen.nl
>>>>>> E robin@us2.nl
>>>>>>
>>>>>> <http://goo.gl/Lt7BC>
>>>>>>
>>>>>> Disclaimer: The information contained in this message and attachments
>>>>>> is intended solely for the attention and use of the named addressee
and may
>>>>>> be confidential. If you are not the intended recipient, you are reminded
>>>>>> that the information remains the property of the sender. You must
not use,
>>>>>> disclose, distribute, copy, print or rely on this e-mail. If you
have
>>>>>> received this message in error, please contact the sender immediately
and
>>>>>> irrevocably delete this message and any copies.
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
View raw message