hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Navis (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-7218) java.io.IOException: error=7, Argument list too long
Date Thu, 19 Jun 2014 00:53:25 GMT

    [ https://issues.apache.org/jira/browse/HIVE-7218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14036758#comment-14036758
] 

Navis commented on HIVE-7218:
-----------------------------

If we cannot change the limit of total size of environment variables, we can walk-around this
problem by specifying only needed ones for the script (I don't think 'mapred.input.dir' could
be useful information in the script). For example,
{noformat}
hive>set X=someX;
hive>set Y=someY;
hive>SELECT TRANSFORM('echo ${X:-x} ${Y:-y} ${Z:-z}') USING 'sh' AS x,y,z WITH ('Z'='someZ',
'Y')
ROW FORMAT DELIMITED FIELDS TERMINATED BY ' ' FROM src tablesample (1 rows);
x	someY	someZ
{noformat}
In the above query, by specifying WITH ('Z'='someZ', 'Y'), script accepts only two environment
variables 'Z=someZ' (which is specified explicitly) and 'Y=someY' (which is from HiveConf).

> java.io.IOException: error=7, Argument list too long
> ----------------------------------------------------
>
>                 Key: HIVE-7218
>                 URL: https://issues.apache.org/jira/browse/HIVE-7218
>             Project: Hive
>          Issue Type: Bug
>          Components: Query Processor
>    Affects Versions: 0.7.0, 0.7.1, 0.8.0, 0.8.1, 0.9.0, 0.10.0, 0.11.0, 0.12.0, 0.13.0,
0.13.1
>            Reporter: Ryan Harris
>            Assignee: Navis
>         Attachments: HIVE-7218.1.patch.txt
>
>
> HIVE-2372 was originally created in response to this error message, however that patch
was merely a work-around to handle the condition where mapred.input.dir is too long.
> Any other environment variable that is "too long" for the host OS will still cause a
job failure.
> In my case:
> While creating a table with a large number of columns, a large hive variable is temporarily
created using SET, the variable contains the columns and column descriptions.
> A CREATE TABLE statement then successfully uses that large variable.
> After successfully creating the table the hive script attempts to load data into the
table using a TRANSFORM script, triggering the error:
> java.io.IOException: error=7, Argument list too long
> Since the variable is no longer used after the table is created, the hive script was
updated to SET the large variable to empty.
> After setting the variable empty the second statement in the hive script ran fine.
> Hive should more gracefully notify the user as to the cause of the problem and offer
a configurable approach for automatically handling the condition.
> In this case, originally identifying the cause of the issue was somewhat confusing since
the portion of the hive script that referenced the long variable ran successfully, and the
portion of the script that failed didn't even use/reference the variable that was causing
that portion to fail.
> Since HIVE-2372 has already been "Fixed" this JIRA re-opens the issue since the original
issue was worked around, not resolved...



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message