pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dmitriy V. Ryaboy (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (PIG-2925) Extremely long JobConf values should not be added to Streaming environment
Date Fri, 28 Sep 2012 03:51:07 GMT

     [ https://issues.apache.org/jira/browse/PIG-2925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Dmitriy V. Ryaboy updated PIG-2925:
-----------------------------------

    Attachment: PIG-2925.patch

Decided it's easier to set the max length, and allow users to bump it up or down as needed,
than to mess with explicit blacklists. Too hard to control what everyone might decide to shove
into their config.
                
> Extremely long JobConf values should not be added to Streaming environment
> --------------------------------------------------------------------------
>
>                 Key: PIG-2925
>                 URL: https://issues.apache.org/jira/browse/PIG-2925
>             Project: Pig
>          Issue Type: Bug
>            Reporter: Dmitriy V. Ryaboy
>            Assignee: Dmitriy V. Ryaboy
>         Attachments: PIG-2925.patch
>
>
> PIG-2900 made Pig match Hadoop's practice of making all JobConf properties available
as environment variables for pig streaming. 
> As it happens, ProcessBuilder creates a long string with all environment variables and
their values on it, then runs it; having some of the weightier properties like serialized
plans in the environment can make the total command length exceed OS limits. 
> We propose adding a blacklist of properties which should not be added to streaming environment.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message