hadoop-hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zheng Shao (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HIVE-1272) Add SymlinkTextInputFormat to Hive
Date Wed, 31 Mar 2010 18:47:28 GMT

    [ https://issues.apache.org/jira/browse/HIVE-1272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12852040#action_12852040
] 

Zheng Shao commented on HIVE-1272:
----------------------------------

We can add a file with name data/symlink.txt which contains the text "../src/kv.txt"
then in ql/src/test/clientpositive/mysymlink.q we can do this:

{code}
CREATE TABLE mysymlink (key STRING, value STRING) STORE AS INPUTFORMAT ...
dfs -cp "../data/symlink.txt" "../build/ql/test/data/warehouse/mysymlink/symlink1.txt";
dfs -cp "../data/symlink.txt" "../build/ql/test/data/warehouse/mysymlink/symlink2.txt";

SELECT * FROM mysymlink;
SELECT count(1) FROM mysymlink;
{code}

In order to test, run:
ant test -Doffline=true -Dtestcase=TestCliDriver -Dqfile=mysymlink.q -Doverwrite=true
And do "svn add ql/.../mysymlink.q.out"

Run without -Doverwrite=true to verify the result.





> Add SymlinkTextInputFormat to Hive
> ----------------------------------
>
>                 Key: HIVE-1272
>                 URL: https://issues.apache.org/jira/browse/HIVE-1272
>             Project: Hadoop Hive
>          Issue Type: New Feature
>    Affects Versions: 0.5.0
>            Reporter: Zheng Shao
>            Assignee: Guanghao Shen
>         Attachments: HIVE-1272.1.patch
>
>
> We'd like to add a symlink text input format so that we can specify the list of files
for a table/partition based on the content of a text file.
> For example, the location of the table is "/user/hive/mytable".
> There is a file called "/user/hive/mytable/myfile.txt".
> Inside the file, there are 2 lines, "/user/myname/textfile1.txt" and "/user/myname/textfile2.txt"
> We can do:
> {code}
> CREATE TABLE mytable (...) STORED AS INPUTFORMAT 'org.apache.hadoop.hive.io.SymlinkTextInputFormat'
LOCATION '/user/hive/mytable';
> SELECT * FROM mytable;
> {code}
> which will return the content of the 2 files: "/user/myname/textfile1.txt" and "/user/myname/textfile2.txt"

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message