flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tzu-Li (Gordon) Tai" <tzuli...@apache.org>
Subject Re: 回复:Re: flink one transformation end,the next transformation start
Date Fri, 31 Mar 2017 12:12:17 GMT
Sorry, I just realized our previous conversation on this question was done via private email
and not to user@flink.apache.org

Forwarding the previous content of the conversation back to the mailing list:

On March 30, 2017 at 4:15:46 PM, rimin515@sina.cn (rimin515@sina.cn) wrote:

the job can run success,but the result is error.
the record 1 and the record 14 is same,so the vector compute cos value is 1,but on the yarn
the value is not 1,and others are different from the result which run on local.

so,i guess,the step:
1 val data = env.readTextFile("hdfs:///....")//DataSet[(String,Array[String])]
2 val dataVec = computeDataVect(data)//DataSet[(String,Int,Array[(Int,Double)])]
3 val rescomm = computeCosSims (dataVec)//DataSet[(String,Array[(String,Double)])]
the record is from 1,2,3;but the step 3 must start when step 2 is end,because step 3 compute
all record cos sim value must use all data.so is there some operate can set the step 3 start
when step 2 is end.
----- 原始邮件 -----
发件人:"Tzu-Li (Gordon) Tai" <tzulitai@apache.org>
收件人:rimin515@sina.cn
主题:Re: flink one transformation end,the next transformation start
日期:2017年03月30日 15点54分

Hi,

What exactly is the error you’re running into on YARN? You should be able to find them in
the TM logs.
It’ll be helpful to understand the problem if you can provide them (just the relevant parts
of the error will do).
Otherwise, I currently can not tell much from the information here.

Cheers,
Gordon


On March 30, 2017 at 3:33:53 PM, rimin515@sina.cn (rimin515@sina.cn) wrote:

hi,all,
i run a job,it is :
---------------------------------------------------------
val data = env.readTextFile("hdfs:///....")//DataSet[(String,Array[String])]
val dataVec = computeDataVect(data)//DataSet[(String,Int,Array[(Int,Double)])]
val rescomm = computeCosSims (dataVec)//DataSet[(String,Array[(String,Double)])]

but when run on the yarn cluster,the result was error,the job can success;and run on the local,in
eclipse on my computer,the result is correct.

so,i run twice,
first:
val data = env.readTextFile("hdfs:///....")//DataSet[(String,Array[String])]
val dataVec = computeDataVect(data)//DataSet[(String,Int,Array[(Int,Double)])]
dataVec.writeAsText("hdfs///vec")//the vector is correct,

second:
val readVec = env.readTextFile("hdfs:///vec").map(...)//DataSet[(String,Int,Array[(Int,Double)])]
val rescomm = computeCosSims (dataVec)//DataSet[(String,Array[(String,Double)])]
and the result is correct,is the same as on local,in eclispe.
----------------------------------
someone can solve the problem?


Mime
View raw message