hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Todd Lipcon <t...@cloudera.com>
Subject Re: Throttling hive queries
Date Fri, 18 Dec 2009 19:03:24 GMT
Hi Sagi,

Sounds like you need CombineFileInputFormat. See:

https://issues.apache.org/jira/browse/HIVE-74

-Todd

On Fri, Dec 18, 2009 at 10:24 AM, Sagi, Lee <lsagi@shopping.com> wrote:

>  Yes that's true, I have a process that runs and pulls 3 weblogs files
> every hour from 10 servers...10*3*24=720 (not all hours have all the files)
>
>
> Lee Sagi | Data Warehouse Tech Lead & Architect | Work: 650-616-6575 |
> Cell: 718-930-7947
>
>
>  ------------------------------
> *From:* Todd Lipcon [mailto:todd@cloudera.com]
> *Sent:* Thursday, December 17, 2009 4:24 PM
>
> *To:* hive-user@hadoop.apache.org
> *Subject:* Re: Throttling hive queries
>
> Hi Sagi,
>
> Any chance you're running on a directory that has 614 small files?
>
> -Todd
>
> On Thu, Dec 17, 2009 at 2:30 PM, Sagi, Lee <lsagi@shopping.com> wrote:
>
>>    Todd, Here is the job info.
>>
>>
>>
>> Counter Map Reduce Total File Systems HDFS bytes read 199,115,508 0
>> 199,115,508 HDFS bytes written 0 9,665,472 9,665,472 Local bytes read 0
>> 321,210,205 321,210,205 Local bytes written 204,404,812 321,210,205
>> 525,615,017 Job Counters Launched reduce tasks 0 0 1 Rack-local map tasks
>> 0 0 614 Launched map tasks 0 0 37,130 Data-local map tasks 0 0 36,516
>> org.apache.hadoop.hive.ql.exec.FilterOperator$Counter PASSED 0 10,572
>> 10,572 FILTERED 0 217,305 217,305
>> org.apache.hadoop.hive.ql.exec.MapOperator$Counter DESERIALIZE_ERRORS 0 0
>> 0 Map-Reduce Framework Reduce input groups 0 429,557 429,557 Combine
>> output records 0 0 0 Map input records 429,557 0 429,557 Reduce output
>> records 0 0 0 Map output bytes 201,425,848 0 201,425,848 Map input bytes
>> 199,115,508 0 199,115,508 Map output records 429,557 0 429,557 Combine
>> input records 0 0 0 Reduce input records 0 429,557 429,557
>>
>>
>> Lee Sagi | Data Warehouse Tech Lead & Architect | Work: 650-616-6575 |
>> Cell: 718-930-7947
>>
>>
>>  ------------------------------
>> *From:* Todd Lipcon [mailto:todd@cloudera.com]
>> *Sent:* Thursday, December 17, 2009 12:18 PM
>>
>> *To:* hive-user@hadoop.apache.org
>> *Subject:* Re: Throttling hive queries
>>
>>   Hi Lee,
>>
>> The MapReduce framework in general makes it hard for you assign fewer
>> mappers than there are blocks in the input data, when using FileInputFormat.
>> Is your input set about 42GB with a 64M block size, or 84G with a 128M block
>> size?
>>
>> -Todd
>>
>> On Thu, Dec 17, 2009 at 11:32 AM, Sagi, Lee <lsagi@shopping.com> wrote:
>>
>>> Here is the query that I am running, just in case someone has an idea of
>>> how to improve it.
>>>
>>> SELECT
>>>      CONCAT(CONCAT('"', PRSS.DATE_KEY), '"'),
>>>      CONCAT(CONCAT('"', PRSC.DATE_KEY), '"'),
>>>      CONCAT(CONCAT('"', PRSS.VOTF_REQUEST_ID), '"'),
>>>      CONCAT(CONCAT('"', PRSC.VOTF_REQUEST_ID), '"'),
>>>      CONCAT(CONCAT('"', PRSS.PRS_REQUEST_ID), '"'),
>>>      CONCAT(CONCAT('"', PRSC.PRS_REQUEST_ID), '"'),
>>>      ...
>>>      ...
>>>      ...
>>>  FROM
>>>      FCT_PRSS PRSS FULL OUTER JOIN FCT_PRSC PRSC ON
>>> (PRSS.PRS_REQUEST_ID = PRSC.PRS_REQUEST_ID)
>>>  WHERE (PRSS.date_key >= '2009121600' AND
>>>        PRSS.date_key < '2009121700') OR
>>>       (PRSC.date_key >= '2009121600' AND
>>>        PRSC.date_key < '2009121700')
>>>
>>>
>>> Lee Sagi | Data Warehouse Tech Lead & Architect | Work: 650-616-6575 |
>>> Cell: 718-930-7947
>>>
>>> -----Original Message-----
>>> From: Edward Capriolo [mailto:edlinuxguru@gmail.com]
>>> Sent: Thursday, December 17, 2009 11:03 AM
>>> To: hive-user@hadoop.apache.org
>>> Subject: Re: Throttling hive queries
>>>
>>>  You should be able
>>>
>>> hive > set mapred.map.tasks=1000
>>> hive > set mapred.reduce.tasks=5
>>>
>>> In some cases mappers is controlled by input files (pre hadoop 20)
>>>
>>>
>>> On Thu, Dec 17, 2009 at 1:58 PM, Sagi, Lee <lsagi@shopping.com> wrote:
>>> > Is there a way to throttle hive queries?
>>> >
>>> > For example, I want to tell hive to not use more then 1000 mappers and
>>>
>>> > 5 reducers for a particular query (or session).
>>> >
>>>
>>
>>
>

Mime
View raw message