nifi-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andre (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (NIFI-1924) Create set of WebHDFS processors
Date Mon, 30 May 2016 00:57:12 GMT

    [ https://issues.apache.org/jira/browse/NIFI-1924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15306132#comment-15306132
] 

Andre commented on NIFI-1924:
-----------------------------

Correction. It does look like it will work!!

I accidentally set the ListHDFS processor "Hadoop Configuration Resources" field to path/hdfs-site.xml
twice... Correctly setting core-site.xml seems to make ListHDFS make the proper WebHDFS requests.

This is what tcpdump is displaying

GET./webhdfs/v1/user/test?op=LISTSTATUS&user.name=nifi.HTTP/1.1

Doing some extra testing now.

Cheers

> Create set of WebHDFS processors
> --------------------------------
>
>                 Key: NIFI-1924
>                 URL: https://issues.apache.org/jira/browse/NIFI-1924
>             Project: Apache NiFi
>          Issue Type: Improvement
>            Reporter: Andre
>
> Create processors to handle, 
> Supported commands for WebHDFS V1
> * GET -> GetWebHDFS
>    * OPEN
>    * GETFILESTATUS
>    * LISTSTATUS
>    * GETCONTENTSUMMARy
>    * GETFILECHECKSUM
>    * GETHOMEDIRECTORY
>    * GETDELEGATIONTOKEN
>    * GETDELEGATIONTOKENS
>    * GETXATTRS
>    * LISTXATTRS
>    * CHECKACCESS
> * PUT -> PutWebHDFS
>    * CREATE
>    * MKDIRS
>    * CREATESYMLINK
>    * RENAME
>    * SETREPLICATION
>    * SETOWNER
>    * SETPERMISSION
>    * SETTIMES
>    * RENEWDELEGATIONTOKEN
>    * CANCELDELEGATIONTOKEN
>    * CREATESNAPSHOT
>    * RENAMESNAPSHOT
>    * SETXATTR
>    * REMOVEXATTR
> * POST -> PostWebHDFS
>    * APPEND
>    * CONCAT
>    * TRUNCATE
> * DELETE -> DeleteWebHDFS
>    * DELETE
>    * DELETESNAPSHOT
> operations against WebHDFS as discussed here:
> http://mail-archives.apache.org/mod_mbox/nifi-users/201604.mbox/%3CCABtjSZk2KLJJVk%2Byk-ySioPxVUV5SLHcN%2BaWMZs1ARpkfy%2B0Ow%40mail.gmail.com%3E
> Required Processor Properties:
> * Namenode Host
> * WebHDFS port
> * WebHDFS API version. Default to v1. Dropdown list of available options. Currently there
is only v1 but would be good to have that in place to show our intentions.
> * Hadoop configuration files. core-site.xml and hdfs-site.xml. These configurations files
should not be required to keep things lite. However they would be required to access a kerberized
Hadoop cluster. If these configuration files are present they should take precedence over
the properties set in NameNode host and WebHDFS port.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message