hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rajesh Balamohan (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HIVE-15388) HiveParser spends lots of time in parsing queries with lots "("
Date Thu, 08 Dec 2016 16:24:58 GMT

     [ https://issues.apache.org/jira/browse/HIVE-15388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Rajesh Balamohan updated HIVE-15388:
------------------------------------
    Description: 
Branch: apache-master (applicable with previous releases as well)

Queries generated via tools can have lots of "(" for "AND/OR" conditions. This causes huge
delays in parsing phase when the number of expressions are high.

e.g
{noformat}
SELECT `iata`,
       `airport`,
       `city`,
       `state`,
       `country`,
       `lat`,
       `lon`
FROM airports
WHERE ((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((`airports`.`airport`
= "Thigpen"
                                                                                OR `airports`.`airport`
= "Astoria Regional")
                                                                               OR `airports`.`airport`
= "Warsaw Municipal")
                                                                              OR `airports`.`airport`
= "John F Kennedy Memorial")
                                                                             OR `airports`.`airport`
= "Hall-Miller Municipal")
                                                                            OR `airports`.`airport`
= "Atqasuk")
                                                                           OR `airports`.`airport`
= "William B Hartsfield-Atlanta Intl")
                                                                          OR `airports`.`airport`
= "Artesia Municipal")
                                                                         OR `airports`.`airport`
= "Outagamie County Regional")
                                                                        OR `airports`.`airport`
= "Watertown Municipal")
                                                                       OR `airports`.`airport`
= "Augusta State")
                                                                      OR `airports`.`airport`
= "Aurora Municipal")
                                                                     OR `airports`.`airport`
= "Alakanuk")
                                                                    OR `airports`.`airport`
= "Austin Municipal")
                                                                   OR `airports`.`airport`
= "Auburn Municipal")
                                                                  OR `airports`.`airport`
= "Auburn-Opelik")
                                                                 OR `airports`.`airport` =
"Austin-Bergstrom International")
                                                                OR `airports`.`airport` =
"Wausau Municipal")
                                                               OR `airports`.`airport` = "Mecklenburg-Brunswick
Regional")
                                                              OR `airports`.`airport` = "Alva
Regional")
                                                             OR `airports`.`airport` = "Asheville
Regional")
                                                            OR `airports`.`airport` = "Avon
Park Municipal")
                                                           OR `airports`.`airport` = "Wilkes-Barre/Scranton
Intl")
                                                          OR `airports`.`airport` = "Marana
Northwest Regional")
                                                         OR `airports`.`airport` = "Catalina")
                                                        OR `airports`.`airport` = "Washington
Municipal")
                                                       OR `airports`.`airport` = "Wainwright")
                                                      OR `airports`.`airport` = "West Memphis
Municipal")
                                                     OR `airports`.`airport` = "Arlington
Municipal")
                                                    OR `airports`.`airport` = "Algona Municipal")
                                                   OR `airports`.`airport` = "Chandler")
                                                  OR `airports`.`airport` = "Altus Municipal")
                                                 OR `airports`.`airport` = "Neil Armstrong")
                                                OR `airports`.`airport` = "Angel Fire")
                                               OR `airports`.`airport` = "Waycross-Ware County")
                                              OR `airports`.`airport` = "Colorado City Municipal")
                                             OR `airports`.`airport` = "Hazelhurst")
                                            OR `airports`.`airport` = "Kalamazoo County")
                                           OR `airports`.`airport` = "Granville")
                                          OR `airports`.`airport` = "Silver Springs")
                                         OR `airports`.`airport` = "Whitford")
                                        OR `airports`.`airport` = "Biddeford Municipal")
                                       OR `airports`.`airport` = "Sugarloaf Regional")
                                      OR `airports`.`airport` = "Barnes Municipal")
                                     OR `airports`.`airport` = "Columbus Municipal")
                                    OR `airports`.`airport` = "Battle Mountain")
                                   OR `airports`.`airport` = "Huron County Memorial")
                                  OR `airports`.`airport` = "New Braunfels Municipal")
                                 OR `airports`.`airport` = "Benson Municipal")
                                OR `airports`.`airport` = "Curtis")
                               OR `airports`.`airport` = "Marlboro County")
                              OR `airports`.`airport` = "Broken Bow Municipal")
                             OR `airports`.`airport` = "Virginia Tech")
                            OR `airports`.`airport` = "Bryce Canyon")
                           OR `airports`.`airport` = "Black River Falls Area")
                          OR `airports`.`airport` = "Boca Raton")
                         OR `airports`.`airport` = "Birchwood")
                        OR `airports`.`airport` = "Baudette International")
                       OR `airports`.`airport` = "Blanding Muni")
                      OR `airports`.`airport` = "Bradley International")
                     OR `airports`.`airport` = "Morrilton Municipal")
                    OR `airports`.`airport` = "Igor I Sikorsky Memorial")
                   OR `airports`.`airport` = "Broadus")
                  OR `airports`.`airport` = "Laurence G Hanscom")
                 OR `airports`.`airport` = "Southwest Michigan Regional")
                OR `airports`.`airport` = "Bethel")
               OR `airports`.`airport` = "Bradford Reg")
              OR `airports`.`airport` = "Scotts Bluff County")
             OR `airports`.`airport` = "Boeing Field/King County Intl")
            OR `airports`.`airport` = "Buffalo Municipal")
           OR `airports`.`airport` = "Meadows")
          OR `airports`.`airport` = "Mobile Downtown")
         OR `airports`.`airport` = "Virgil I Grissom Municipal")
        OR `airports`.`airport` = "Silver Bay Municipal");
{noformat}

This query would take more than minutes to parse and compile.  

Temp Workaround: If "("s are removed manually, it gets completed in 2-3 seconds.

Problem exists in older versions as well (but in older versions this can make HS2 unresponsive
for large amount of time, as Driver ends up taking global compiler lock in this phase). 

  was:
Branch: apache-master (applicable with previous releases as well)

Queries generated via tools can have lots "(" for "AND/OR" conditions. This causes huge delays
in parsing phase when the number of expressions are high.

e.g
{noformat}
SELECT `iata`,
       `airport`,
       `city`,
       `state`,
       `country`,
       `lat`,
       `lon`
FROM airports
WHERE ((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((`airports`.`airport`
= "Thigpen"
                                                                                OR `airports`.`airport`
= "Astoria Regional")
                                                                               OR `airports`.`airport`
= "Warsaw Municipal")
                                                                              OR `airports`.`airport`
= "John F Kennedy Memorial")
                                                                             OR `airports`.`airport`
= "Hall-Miller Municipal")
                                                                            OR `airports`.`airport`
= "Atqasuk")
                                                                           OR `airports`.`airport`
= "William B Hartsfield-Atlanta Intl")
                                                                          OR `airports`.`airport`
= "Artesia Municipal")
                                                                         OR `airports`.`airport`
= "Outagamie County Regional")
                                                                        OR `airports`.`airport`
= "Watertown Municipal")
                                                                       OR `airports`.`airport`
= "Augusta State")
                                                                      OR `airports`.`airport`
= "Aurora Municipal")
                                                                     OR `airports`.`airport`
= "Alakanuk")
                                                                    OR `airports`.`airport`
= "Austin Municipal")
                                                                   OR `airports`.`airport`
= "Auburn Municipal")
                                                                  OR `airports`.`airport`
= "Auburn-Opelik")
                                                                 OR `airports`.`airport` =
"Austin-Bergstrom International")
                                                                OR `airports`.`airport` =
"Wausau Municipal")
                                                               OR `airports`.`airport` = "Mecklenburg-Brunswick
Regional")
                                                              OR `airports`.`airport` = "Alva
Regional")
                                                             OR `airports`.`airport` = "Asheville
Regional")
                                                            OR `airports`.`airport` = "Avon
Park Municipal")
                                                           OR `airports`.`airport` = "Wilkes-Barre/Scranton
Intl")
                                                          OR `airports`.`airport` = "Marana
Northwest Regional")
                                                         OR `airports`.`airport` = "Catalina")
                                                        OR `airports`.`airport` = "Washington
Municipal")
                                                       OR `airports`.`airport` = "Wainwright")
                                                      OR `airports`.`airport` = "West Memphis
Municipal")
                                                     OR `airports`.`airport` = "Arlington
Municipal")
                                                    OR `airports`.`airport` = "Algona Municipal")
                                                   OR `airports`.`airport` = "Chandler")
                                                  OR `airports`.`airport` = "Altus Municipal")
                                                 OR `airports`.`airport` = "Neil Armstrong")
                                                OR `airports`.`airport` = "Angel Fire")
                                               OR `airports`.`airport` = "Waycross-Ware County")
                                              OR `airports`.`airport` = "Colorado City Municipal")
                                             OR `airports`.`airport` = "Hazelhurst")
                                            OR `airports`.`airport` = "Kalamazoo County")
                                           OR `airports`.`airport` = "Granville")
                                          OR `airports`.`airport` = "Silver Springs")
                                         OR `airports`.`airport` = "Whitford")
                                        OR `airports`.`airport` = "Biddeford Municipal")
                                       OR `airports`.`airport` = "Sugarloaf Regional")
                                      OR `airports`.`airport` = "Barnes Municipal")
                                     OR `airports`.`airport` = "Columbus Municipal")
                                    OR `airports`.`airport` = "Battle Mountain")
                                   OR `airports`.`airport` = "Huron County Memorial")
                                  OR `airports`.`airport` = "New Braunfels Municipal")
                                 OR `airports`.`airport` = "Benson Municipal")
                                OR `airports`.`airport` = "Curtis")
                               OR `airports`.`airport` = "Marlboro County")
                              OR `airports`.`airport` = "Broken Bow Municipal")
                             OR `airports`.`airport` = "Virginia Tech")
                            OR `airports`.`airport` = "Bryce Canyon")
                           OR `airports`.`airport` = "Black River Falls Area")
                          OR `airports`.`airport` = "Boca Raton")
                         OR `airports`.`airport` = "Birchwood")
                        OR `airports`.`airport` = "Baudette International")
                       OR `airports`.`airport` = "Blanding Muni")
                      OR `airports`.`airport` = "Bradley International")
                     OR `airports`.`airport` = "Morrilton Municipal")
                    OR `airports`.`airport` = "Igor I Sikorsky Memorial")
                   OR `airports`.`airport` = "Broadus")
                  OR `airports`.`airport` = "Laurence G Hanscom")
                 OR `airports`.`airport` = "Southwest Michigan Regional")
                OR `airports`.`airport` = "Bethel")
               OR `airports`.`airport` = "Bradford Reg")
              OR `airports`.`airport` = "Scotts Bluff County")
             OR `airports`.`airport` = "Boeing Field/King County Intl")
            OR `airports`.`airport` = "Buffalo Municipal")
           OR `airports`.`airport` = "Meadows")
          OR `airports`.`airport` = "Mobile Downtown")
         OR `airports`.`airport` = "Virgil I Grissom Municipal")
        OR `airports`.`airport` = "Silver Bay Municipal");
{noformat}

This query would take more than minutes to parse and compile.  

Temp Workaround: If "(" are removed manually, it gets completed in 2-3 seconds.


> HiveParser spends lots of time in parsing queries with lots "("
> ---------------------------------------------------------------
>
>                 Key: HIVE-15388
>                 URL: https://issues.apache.org/jira/browse/HIVE-15388
>             Project: Hive
>          Issue Type: Improvement
>    Affects Versions: 2.2.0
>            Reporter: Rajesh Balamohan
>         Attachments: hive-15388.stacktrace.txt
>
>
> Branch: apache-master (applicable with previous releases as well)
> Queries generated via tools can have lots of "(" for "AND/OR" conditions. This causes
huge delays in parsing phase when the number of expressions are high.
> e.g
> {noformat}
> SELECT `iata`,
>        `airport`,
>        `city`,
>        `state`,
>        `country`,
>        `lat`,
>        `lon`
> FROM airports
> WHERE ((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((`airports`.`airport`
= "Thigpen"
>                                                                                 OR `airports`.`airport`
= "Astoria Regional")
>                                                                                OR `airports`.`airport`
= "Warsaw Municipal")
>                                                                               OR `airports`.`airport`
= "John F Kennedy Memorial")
>                                                                              OR `airports`.`airport`
= "Hall-Miller Municipal")
>                                                                             OR `airports`.`airport`
= "Atqasuk")
>                                                                            OR `airports`.`airport`
= "William B Hartsfield-Atlanta Intl")
>                                                                           OR `airports`.`airport`
= "Artesia Municipal")
>                                                                          OR `airports`.`airport`
= "Outagamie County Regional")
>                                                                         OR `airports`.`airport`
= "Watertown Municipal")
>                                                                        OR `airports`.`airport`
= "Augusta State")
>                                                                       OR `airports`.`airport`
= "Aurora Municipal")
>                                                                      OR `airports`.`airport`
= "Alakanuk")
>                                                                     OR `airports`.`airport`
= "Austin Municipal")
>                                                                    OR `airports`.`airport`
= "Auburn Municipal")
>                                                                   OR `airports`.`airport`
= "Auburn-Opelik")
>                                                                  OR `airports`.`airport`
= "Austin-Bergstrom International")
>                                                                 OR `airports`.`airport`
= "Wausau Municipal")
>                                                                OR `airports`.`airport`
= "Mecklenburg-Brunswick Regional")
>                                                               OR `airports`.`airport`
= "Alva Regional")
>                                                              OR `airports`.`airport`
= "Asheville Regional")
>                                                             OR `airports`.`airport` =
"Avon Park Municipal")
>                                                            OR `airports`.`airport` =
"Wilkes-Barre/Scranton Intl")
>                                                           OR `airports`.`airport` = "Marana
Northwest Regional")
>                                                          OR `airports`.`airport` = "Catalina")
>                                                         OR `airports`.`airport` = "Washington
Municipal")
>                                                        OR `airports`.`airport` = "Wainwright")
>                                                       OR `airports`.`airport` = "West
Memphis Municipal")
>                                                      OR `airports`.`airport` = "Arlington
Municipal")
>                                                     OR `airports`.`airport` = "Algona
Municipal")
>                                                    OR `airports`.`airport` = "Chandler")
>                                                   OR `airports`.`airport` = "Altus Municipal")
>                                                  OR `airports`.`airport` = "Neil Armstrong")
>                                                 OR `airports`.`airport` = "Angel Fire")
>                                                OR `airports`.`airport` = "Waycross-Ware
County")
>                                               OR `airports`.`airport` = "Colorado City
Municipal")
>                                              OR `airports`.`airport` = "Hazelhurst")
>                                             OR `airports`.`airport` = "Kalamazoo County")
>                                            OR `airports`.`airport` = "Granville")
>                                           OR `airports`.`airport` = "Silver Springs")
>                                          OR `airports`.`airport` = "Whitford")
>                                         OR `airports`.`airport` = "Biddeford Municipal")
>                                        OR `airports`.`airport` = "Sugarloaf Regional")
>                                       OR `airports`.`airport` = "Barnes Municipal")
>                                      OR `airports`.`airport` = "Columbus Municipal")
>                                     OR `airports`.`airport` = "Battle Mountain")
>                                    OR `airports`.`airport` = "Huron County Memorial")
>                                   OR `airports`.`airport` = "New Braunfels Municipal")
>                                  OR `airports`.`airport` = "Benson Municipal")
>                                 OR `airports`.`airport` = "Curtis")
>                                OR `airports`.`airport` = "Marlboro County")
>                               OR `airports`.`airport` = "Broken Bow Municipal")
>                              OR `airports`.`airport` = "Virginia Tech")
>                             OR `airports`.`airport` = "Bryce Canyon")
>                            OR `airports`.`airport` = "Black River Falls Area")
>                           OR `airports`.`airport` = "Boca Raton")
>                          OR `airports`.`airport` = "Birchwood")
>                         OR `airports`.`airport` = "Baudette International")
>                        OR `airports`.`airport` = "Blanding Muni")
>                       OR `airports`.`airport` = "Bradley International")
>                      OR `airports`.`airport` = "Morrilton Municipal")
>                     OR `airports`.`airport` = "Igor I Sikorsky Memorial")
>                    OR `airports`.`airport` = "Broadus")
>                   OR `airports`.`airport` = "Laurence G Hanscom")
>                  OR `airports`.`airport` = "Southwest Michigan Regional")
>                 OR `airports`.`airport` = "Bethel")
>                OR `airports`.`airport` = "Bradford Reg")
>               OR `airports`.`airport` = "Scotts Bluff County")
>              OR `airports`.`airport` = "Boeing Field/King County Intl")
>             OR `airports`.`airport` = "Buffalo Municipal")
>            OR `airports`.`airport` = "Meadows")
>           OR `airports`.`airport` = "Mobile Downtown")
>          OR `airports`.`airport` = "Virgil I Grissom Municipal")
>         OR `airports`.`airport` = "Silver Bay Municipal");
> {noformat}
> This query would take more than minutes to parse and compile.  
> Temp Workaround: If "("s are removed manually, it gets completed in 2-3 seconds.
> Problem exists in older versions as well (but in older versions this can make HS2 unresponsive
for large amount of time, as Driver ends up taking global compiler lock in this phase). 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message