Mailing-List: contact issues-help@nifi.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@nifi.apache.org
Date: Tue, 14 Feb 2017 22:23:42 +0000 (UTC)
From: "ASF GitHub Bot (JIRA)" <jira@apache.org>
To: issues@nifi.apache.org
Message-ID: <JIRA.12965317.1462465441000.80993.1487111022598@Atlassian.JIRA>
In-Reply-To: <JIRA.12965317.1462465441000@Atlassian.JIRA>
References: <JIRA.12965317.1462465441000@Atlassian.JIRA> <JIRA.12965317.1462465441926@jira-lw-us.apache.org>
Subject: [jira] [Commented] (NIFI-1856) ExecuteStreamCommand Needs to
 Consume Standard Error
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Tue, 14 Feb 2017 22:23:49 -0000


    [ https://issues.apache.org/jira/browse/NIFI-1856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15866827#comment-15866827 ] 

ASF GitHub Bot commented on NIFI-1856:
--------------------------------------

Github user brosander commented on the issue:

    https://github.com/apache/nifi/pull/1364
  
    Reviewing


> ExecuteStreamCommand Needs to Consume Standard Error
> ----------------------------------------------------
>
>                 Key: NIFI-1856
>                 URL: https://issues.apache.org/jira/browse/NIFI-1856
>             Project: Apache NiFi
>          Issue Type: Bug
>            Reporter: Alan Jackoway
>            Assignee: Karthik Narayanan
>
> I was using ExecuteStreamProcess to run certain hdfs commands that are tricky to write in nifi but easy in bash (e.g. {{hadoop fs -rm -r /data/*/2014/05/05}})
> However, my larger commands kept hanging even though when I run them from the command line they finish quickly.
> Based on http://www.javaworld.com/article/2071275/core-java/when-runtime-exec---won-t.html I believe that ExecuteStreamCommand and possibly other processors need to consume the standard error stream to prevent the processes from blocking when standard error gets filled.
> To reproduce. Create this as ~/write.py
> {code:python}
> import sys
> count = int(sys.argv[1])
> for x in range(count):
>     sys.stderr.write("ERROR %d\n" % x)
>     sys.stdout.write("OUTPUT %d\n" % x)
> {code}
> Create a flow that goes 
> # GenerateFlowFile - 5 minutes schedule 0 bytes size 
> # ExecuteStreamCommand - Command arguments /Users/alanj/write.py;100 Command Path python
> # PutFile - /tmp/write/
> routing output stream of ExecuteStreamCommand to PutFile
> When you turn everything on, you get 100 lines (not 200) of just the standard output in /tmp/write.
> Next, change the command arguments to /Users/alanj/write.py;100000 and turn everything on again. The command will hang.
> I believe that whenever you execute a process the way ExecuteStreamCommand is doing, you need to consume the standard error stream to keep it from blocking. This may also affect things like ExecuteProcess and ExecuteScript as well.


--
This message was sent by Atlassian JIRA
(v6.3.15#6346)