flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Metzger <rmetz...@apache.org>
Subject Re: flink k-means on hadoop cluster
Date Thu, 04 Jun 2015 14:25:37 GMT
It should be certainly possible to run Flink on a cloudera live VM

I think these are the commands you need to execute:

wget
http://stratosphere-bin.s3-website-us-east-1.amazonaws.com/flink-0.9-SNAPSHOT-bin-hadoop2.tgz
tar xvzf flink-0.9-SNAPSHOT-bin-hadoop2.tgz
cd flink-0.9-SNAPSHOT/
*export HADOOP_CONF_DIR=/usr/lib/hadoop/etc/hadoop/*
./bin/yarn-session.sh -n 1 -jm 1024 -tm 1024

If that is not working for you, please post the exact error message you are
getting and I can help you to get it to run.


On Thu, Jun 4, 2015 at 4:18 PM, Pa Rö <paul.roewer1990@googlemail.com>
wrote:

> hi robert,
>
> i think the problem is the hue api,
> i had the same problem with spark submit script,
> but on the new hue release, they have a spark submit api.
>
> i asked the group for the same problem with spark, no reply.
>
> i want test my app on local cluster, before i run it on the big cluster,
> for that i use cloudera live. maybe it give an other way to test flink on
> a local cluster vm?
>
> 2015-06-04 16:12 GMT+02:00 Robert Metzger <rmetzger@apache.org>:
>
>> Hi Paul,
>>
>> why did running Flink from the regular scripts not work for you?
>>
>> I'm not an expert on Hue, I would recommend asking in the Hue user forum
>> / mailing list:
>> https://groups.google.com/a/cloudera.org/forum/#!forum/hue-user.
>>
>> On Thu, Jun 4, 2015 at 4:09 PM, Pa Rö <paul.roewer1990@googlemail.com>
>> wrote:
>>
>>> thanks,
>>> now i want run my app on cloudera live vm single node,
>>> how i can define my flink job with hue?
>>> i try to run the flink script in the hdfs, it's not work.
>>>
>>> best regards,
>>> paul
>>>
>>> 2015-06-02 14:50 GMT+02:00 Robert Metzger <rmetzger@apache.org>:
>>>
>>>> I would recommend using HDFS.
>>>> For that, you need to specify the paths like this: hdfs:///path/to/data.
>>>>
>>>> On Tue, Jun 2, 2015 at 2:48 PM, Pa Rö <paul.roewer1990@googlemail.com>
>>>> wrote:
>>>>
>>>>> nice,
>>>>>
>>>>> which file system i must use for the cluster? java.io or hadoop.fs or
>>>>> flink?
>>>>>
>>>>> 2015-06-02 14:29 GMT+02:00 Robert Metzger <rmetzger@apache.org>:
>>>>>
>>>>>> Hi,
>>>>>> you can start Flink on YARN on the Cloudera distribution.
>>>>>>
>>>>>> See here for more:
>>>>>> http://ci.apache.org/projects/flink/flink-docs-master/setup/yarn_setup.html
>>>>>>
>>>>>> These are the commands you need to execute
>>>>>>
>>>>>> wget http://stratosphere-bin.s3-website-us-east-1.amazonaws.com/flink-0.9-SNAPSHOT-bin-hadoop2.tgz
>>>>>> tar xvzf flink-0.9-SNAPSHOT-bin-hadoop2.tgzcd flink-0.9-SNAPSHOT/
>>>>>> ./bin/yarn-session.sh -n 4 -jm 1024 -tm 4096
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Tue, Jun 2, 2015 at 2:03 PM, Pa Rö <paul.roewer1990@googlemail.com
>>>>>> > wrote:
>>>>>>
>>>>>>> hi community,
>>>>>>>
>>>>>>> i want test my flink k-means on a hadoop cluster. i use the cloudera
>>>>>>> live distribution. how i can run flink on this cluster? maybe
only the java
>>>>>>> dependencies are engouth?
>>>>>>>
>>>>>>> best regards,
>>>>>>> paul
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
View raw message