hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Joydeep Sen Sarma (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HIVE-1570) referencing an added file by it's name in a transform script does not work in hive local mode
Date Mon, 04 Oct 2010 22:20:34 GMT

     [ https://issues.apache.org/jira/browse/HIVE-1570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Joydeep Sen Sarma updated HIVE-1570:
------------------------------------

    Attachment: 1570.1.patch

before running a map-reduce job in local mode we:
1. set a new working directory
2. symlink all added files from that working directory

this is pretty much identical to how hadoop sets up task execution environment. all references
to scripts and add files using their names only now resolve correctly in local mode.

there was some hacky code in SemanticAnalyzer.java to deal with this that doesn't work in
all cases (when referenced file is not the first item in command line or in automatic local
mode). i have deleted it.

duplicated one of the tests so that we get coverage against a real cluster (scriptfile1.q
executed against minimr) and local mode (scriptfile2.q).

still running tests.

> referencing an added file by it's name in a transform script does not work in hive local
mode
> ---------------------------------------------------------------------------------------------
>
>                 Key: HIVE-1570
>                 URL: https://issues.apache.org/jira/browse/HIVE-1570
>             Project: Hadoop Hive
>          Issue Type: Bug
>          Components: Query Processor
>            Reporter: Joydeep Sen Sarma
>            Assignee: Joydeep Sen Sarma
>         Attachments: 1570.1.patch
>
>
> Yongqiang tried this and it fails in local mode:
> add file ../data/scripts/dumpdata_script.py;
> select count(distinct subq.key) from
> (FROM src MAP src.key USING 'python dumpdata_script.py' AS key WHERE src.key = 10) subq;
> this needs to be fixed because it means we cannot choose local mode automatically in
case of transform scripts (since different paths need to be used for cluster vs. local mode
execution)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message