flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bohinski, Kevin" <Kevin_Bohin...@comcast.com>
Subject Re: Native K8S not creating TMs
Date Thu, 25 Jun 2020 19:05:12 GMT
Hi Yang,

Thanks for your help, that command worked, so we connected a remote debugger and found the
root exception was initially a timeout exception from okhttp. The increases you mentioned
worked.

Thanks again for all the help!
Best,
kevin


On 2020/06/19 03:46:36, Yang Wang <d...@gmail.com<mailto:d...@gmail.com>> wrote:
> Thanks for sharing the DEBUG level log.>
>
> I carefully check the logs and find that the kubernetes-client discovered>
> the>
> api server address and token successfully.  However, it could not contact>
> with>
> api server(10.100.0.1:443). Could you check whether you api server is>
> configured>
> to allow accessing within cluster.>
>
> I think you could start any pod and tunnel in to run the following command.>
>
> KUBE_TOKEN=$(</var/run/secrets/kubernetes.io/serviceaccount/token)>
> wget -vO- --ca-certificate>
> /var/run/secrets/kubernetes.io/serviceaccount/ca.crt  --header>
> "Authorization: Bearer $KUBE_TOKEN">
> https://$KUBERNETES_SERVICE_HOST:$KUBERNETES_PORT_443_TCP_PORT/api>
>
> BTW, what's your kubernetes version? And i am not sure whether increasing>
> the timeout>
> could help.>
>
> -Dcontainerized.master.env.KUBERNETES_REQUEST_TIMEOUT=60000>
> -Dcontainerized.master.env.KUBERNETES_CONNECTION_TIMEOUT=60000>
>
>
> Best,>
> Yang>
>
>
> Yang Wang <da...@gmail.com<mailto:da...@gmail.com>> 于2020年6月16日周二
下午12:00写道:>
>
> > Hi Kevin,>
> >>
> > Sorry for not notice your last response.>
> > Could you share you full DEBUG level jobmanager logs? I will try to figure>
> > out>
> > whether it is a issue of Flink or K8s. Because i could not reproduce your>
> > situation>
> > with my local K8s cluster.>
> >>
> >>
> > Best,>
> > Yang>
> >>
> > Yang Wang <da...@gmail.com<mailto:da...@gmail.com>> 于2020年6月8日周一
上午11:02写道:>
> >>
> >> Hi Kevin,>
> >>>
> >> It may because the characters length limitation of K8s(no more than>
> >> 63)[1]. So the pod>
> >> name could not be too long. I notice that you are using the client>
> >> automatic generated>
> >> cluster-id. It may cause problem and could you set a meaningful>
> >> cluster-id for your Flink>
> >> session? For example,>
> >>>
> >> kubernetes-session.sh ... -Dkubernetes.cluster-id=my-flink-k8s-session>
> >>>
> >> This behavior has been improved in Flink 1.11 to check the length in>
> >> client side before submission.>
> >>>
> >> If it still could not work, could you share your full command and>
> >> jobmanager logs? It will help a lot>
> >> to find the root cause.>
> >>>
> >>>
> >> [1].>
> >> https://stackoverflow.com/questions/50412837/kubernetes-label-name-63-character-limit>
> >>>
> >>>
> >> Best,>
> >> Yang>
> >>>
> >> kb <Ke...@comcast.com<mailto:Ke...@comcast.com>> 于2020年6月6日周六
上午1:00写道:>
> >>>
> >>> Thanks Yang for the suggestion, I have tried it and I'm still getting the>
> >>> same exception. Is it possible its due to the null pod name? Operation:>
> >>> [create]  for kind: [Pod]  with name: [null]  in namespace: [default]>
> >>> failed.>
> >>>>
> >>> Best,>
> >>> kevin>
> >>>>
> >>>>
> >>>>
> >>> -->
> >>> Sent from:>
> >>> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/>
> >>>>
> >>>
>

Best,
kevin

Mime
View raw message