hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sergey Shelukhin (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HIVE-17827) refactor TezTask file management
Date Wed, 18 Oct 2017 01:40:00 GMT

     [ https://issues.apache.org/jira/browse/HIVE-17827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Sergey Shelukhin updated HIVE-17827:
------------------------------------
    Description: 
There are about 5 different duplicate and intersecting structures used in TezTask to manage
the file list (additionalLr, inputOutputX xN (array and list and/or map?), resourceMap; plus
a list in the actual session object); multiple methods named addExtraResourcesBlah, localizeResourcesBlah;
at least 2 places where TezClient is invoked to add resources to AM, on the same path.
All this mess needs to be changed to have exactly one place that figures out what needs to
be localized, and exactly one place (in TezSessionState so we don't duplicate state and TezClient
usage inside and outside of it) where we look at what's already there and localize things
that are not.
Meanwhile signatures can also be cleaned up, e.g. since branch-1 jobConf is passed to updateSession
and ignored, etc.

  was:
There are about 5 different duplicate and intersecting structures used in TezTask to manage
the file list (additionalLr, inputOutputX xN (array and list and/or map?), resourceMap; plus
a list in the actual session object); multiple methods named addExtraResourcesBlah, localizeResourcesBlah;
at least 2 places where TezClient is invoked to add resources to AM, on the same path.
All this mess needs to be changed to have exactly one place that figures out what needs to
be localized, and exactly one place (in TezSessionState so we don't duplicate state and TezClient
usage inside and outside of it) where we look at what's already there and localize things
that are not.


> refactor TezTask file management
> --------------------------------
>
>                 Key: HIVE-17827
>                 URL: https://issues.apache.org/jira/browse/HIVE-17827
>             Project: Hive
>          Issue Type: Bug
>            Reporter: Sergey Shelukhin
>
> There are about 5 different duplicate and intersecting structures used in TezTask to
manage the file list (additionalLr, inputOutputX xN (array and list and/or map?), resourceMap;
plus a list in the actual session object); multiple methods named addExtraResourcesBlah, localizeResourcesBlah;
at least 2 places where TezClient is invoked to add resources to AM, on the same path.
> All this mess needs to be changed to have exactly one place that figures out what needs
to be localized, and exactly one place (in TezSessionState so we don't duplicate state and
TezClient usage inside and outside of it) where we look at what's already there and localize
things that are not.
> Meanwhile signatures can also be cleaned up, e.g. since branch-1 jobConf is passed to
updateSession and ignored, etc.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message