flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-377) Create a general purpose framework for language bindings
Date Thu, 20 Nov 2014 21:24:34 GMT

    [ https://issues.apache.org/jira/browse/FLINK-377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14220023#comment-14220023
] 

ASF GitHub Bot commented on FLINK-377:
--------------------------------------

Github user rmetzger commented on the pull request:

    https://github.com/apache/incubator-flink/pull/202#issuecomment-63882316
  
    With the fixed python call, I'm getting from the example.
    ```
    StreamPrinter: Traceback (most recent call last):
    StreamPrinter:   File "/tmp/flink_plan/plan.py", line 26, in <module>
    StreamPrinter:     data = env.read_text(textfile)
    StreamPrinter: NameError: name 'textfile' is not defined
    ```
    
    Would be cool to have a) an included data set b) let the user pass the path's from the
command line.
    
    Once I've fixed that, I'm getting another error:
    ```
    Error: The main method caused an error.
    org.apache.flink.client.program.ProgramInvocationException: The main method caused an
error.
    	at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:445)
    	at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:346)
    	at org.apache.flink.client.program.Client.run(Client.java:244)
    	at org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:347)
    	at org.apache.flink.client.CliFrontend.run(CliFrontend.java:334)
    	at org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1001)
    	at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1025)
    Caused by: java.io.IOException: The given HDFS file URI (hdfs:/tmp/flink) did not describe
the HDFS Namenode. The attempt to use a default HDFS configuration, as specified in the 'fs.hdfs.hdfsdefault'
or 'fs.hdfs.hdfssite' config parameter failed due to the following problem: Either no default
hdfs configuration was registered, or the provided configuration contains no valid hdfs namenode
address (fs.default.name or fs.defaultFS) describing the hdfs namenode host and port.
    	at org.apache.flink.runtime.fs.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:281)
    	at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:255)
    	at org.apache.flink.languagebinding.api.java.python.PythonPlanBinder.clearPath(PythonPlanBinder.java:149)
    	at org.apache.flink.languagebinding.api.java.python.PythonPlanBinder.distributeFiles(PythonPlanBinder.java:142)
    	at org.apache.flink.languagebinding.api.java.python.PythonPlanBinder.go(PythonPlanBinder.java:78)
    	at org.apache.flink.languagebinding.api.java.python.PythonPlanBinder.main(PythonPlanBinder.java:62)
    	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    	at java.lang.reflect.Method.invoke(Method.java:483)
    	at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:430)
    	... 6 more
    ```
    
    It seems that HDFS is hardcoded. It would be better to just use the "fs.defaultFS" parameter
to let the user decide on the file system.


> Create a general purpose framework for language bindings
> --------------------------------------------------------
>
>                 Key: FLINK-377
>                 URL: https://issues.apache.org/jira/browse/FLINK-377
>             Project: Flink
>          Issue Type: Improvement
>            Reporter: GitHub Import
>              Labels: github-import
>             Fix For: pre-apache
>
>
> A general purpose API to run operators with arbitrary binaries. 
> This will allow to run Stratosphere programs written in Python, JavaScript, Ruby, Go
or whatever you like. 
> We suggest using Google Protocol Buffers for data serialization. This is the list of
languages that currently support ProtoBuf: https://code.google.com/p/protobuf/wiki/ThirdPartyAddOns

> Very early prototype with python: https://github.com/rmetzger/scratch/tree/learn-protobuf
(basically testing protobuf)
> For Ruby: https://github.com/infochimps-labs/wukong
> Two new students working at Stratosphere (@skunert and @filiphaase) are working on this.
> The reference binding language will be for Python, but other bindings are very welcome.
> The best name for this so far is "stratosphere-lang-bindings".
> I created this issue to track the progress (and give everybody a chance to comment on
this)
> ---------------- Imported from GitHub ----------------
> Url: https://github.com/stratosphere/stratosphere/issues/377
> Created by: [rmetzger|https://github.com/rmetzger]
> Labels: enhancement, 
> Assignee: [filiphaase|https://github.com/filiphaase]
> Created at: Tue Jan 07 19:47:20 CET 2014
> State: open



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message