Mailing-List: contact user-help@flink.apache.org; run by ezmlm
Precedence: bulk
MIME-Version: 1.0
In-Reply-To: <CAKhqdDzadmx634wgaVxFqg7ge+6AjV1yph8SSQ+4iNmvyG+W=A@mail.gmail.com>
References: <CAKhqdDy4v23Ct_ASDZ++ihZUcGNh6j6=pmt_qLfAiYvxAhz2fg@mail.gmail.com>
 <c70bb000-054f-597d-8eff-6317a9f809c9@apache.org> <CAKhqdDzadmx634wgaVxFqg7ge+6AjV1yph8SSQ+4iNmvyG+W=A@mail.gmail.com>
From: Tony Wei <tony19920430@gmail.com>
Date: Sat, 23 Sep 2017 19:11:54 +0800
Message-ID: <CAKhqdDwe+dE=JAUbbs_kLp=Gabyn2OJ+Y+uN4uJDRQ6rGpN1pA@mail.gmail.com>
Subject: Re: Get EOF from PrometheusReporter in JM
To: Chesnay Schepler <chesnay@apache.org>
Cc: user <user@flink.apache.org>
Content-Type: multipart/alternative; boundary="001a114a8496a1d0100559d964c6"
archived-at: Sat, 23 Sep 2017 11:12:04 -0000

--001a114a8496a1d0100559d964c6
Content-Type: text/plain; charset="UTF-8"

Hi Chesnay,

I built another flink cluster using version 1.4, set the log level to
DEBUG, and I found that the root cause might be this exception:
*java.lang.NullPointerException:
Value returned by gauge lastCheckpointExternalPath was null*.

I updated `CheckpointStatsTracker` to ignore external path when it is null,
and this exception didn't happen again. The prometheus reporter works as
well.

I have created a Jira issue for it:
https://issues.apache.org/jira/browse/FLINK-7675
<https://issues.apache.org/jira/browse/FLINK-7675.>, and I will submit the
PR after I passed Travis CI for my repository.

Best Regards,
Tony Wei


2017-09-22 22:20 GMT+08:00 Tony Wei <tony19920430@gmail.com>:

> Hi Chesnay,
>
> I didn't try it in 1.4, so I have no idea if this also occurs in 1.4.
> For my setting for logging, It have already set to INFO level, but there
> wasn't any error or warning in log file as well.
>
> Best Regards,
> Tony Wei
>
> 2017-09-22 22:07 GMT+08:00 Chesnay Schepler <chesnay@apache.org>:
>
>> The Prometheus reporter should work with 1.3.2.
>>
>> Does this also occur with the reporter that currently exists in 1.4? (to
>> rule out new bugs from the PR).
>>
>> To investigate this further, please set the logging level to WARN and try
>> again, as all errors in the metric system are logged on that level.
>>
>>
>> On 22.09.2017 10:33, Tony Wei wrote:
>>
>> Hi,
>>
>> I have built the Prometheus reporter package from this PR
>> https://github.com/apache/flink/pull/4586, and used it on Flink 1.3.2 to
>> record every default metrics and those from `FlinkKafkaConsumer`.
>>
>> Originally, everything was fine. I could get those metrics in TM from
>> Prometheus just like I saw on Flink Web UI.
>> However, when I turned to JM, I found Prometheus gives this error to me: Get
>> http://localhost:9249/metrics: EOF.
>> I checked the log on JM and saw nothing in it. There was no error message
>> and 9249 port was still alive.
>>
>> To figure out what happened, I created another cluster and I found
>> Prometheus could connect to Flink cluster if there is no running job. After
>> JM triggered or completed the first checkpoint, Prometheus started getting
>> ERR_EMPTY_RESPONSE from JM, but not for TM. There was still no error in
>> log file and 9249 port was still alive.
>>
>> I was wondering where did the error occur. Flink or Prometheus reporter?
>> Or It is incorrect to use Prometheus reporter on Flink 1.3.2 ? Thank you.
>>
>> Best Regards,
>> Tony Wei
>>
>>
>>
>

--001a114a8496a1d0100559d964c6
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Hi Chesnay,<div><br></div><div>I built another flink clust=
er using version 1.4, set the log level to DEBUG, and I found that the root=
 cause might be this exception:=C2=A0<b>java.lang.<wbr>NullPointerException=
: Value returned by gauge lastCheckpointExternalPath was null</b>.</div><di=
v><br></div><div>I updated `CheckpointStatsTracker` to ignore external path=
 when it is null, and this exception didn&#39;t happen again. The prometheu=
s reporter works as well.</div><div><br></div><div>I have created a Jira is=
sue for it:=C2=A0<a href=3D"https://issues.apache.org/jira/browse/FLINK-767=
5.">https://issues.apache.org/jira/browse/FLINK-7675</a>, and=C2=A0I will s=
ubmit the PR after I passed Travis CI for my repository.</div><div><br></di=
v><div>Best Regards,</div><div>Tony Wei</div><div><br></div><div>=C2=A0</di=
v></div><div class=3D"gmail_extra"><br><div class=3D"gmail_quote">2017-09-2=
2 22:20 GMT+08:00 Tony Wei <span dir=3D"ltr">&lt;<a href=3D"mailto:tony1992=
0430@gmail.com" target=3D"_blank">tony19920430@gmail.com</a>&gt;</span>:<br=
><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1=
px #ccc solid;padding-left:1ex"><div dir=3D"ltr">Hi Chesnay,<div><br></div>=
<div>I didn&#39;t try it in 1.4, so I have no idea if this also occurs in 1=
.4.</div><div>For my setting for logging, It have already set to INFO level=
, but there wasn&#39;t any error or warning in log file as well.</div><div>=
<br></div><div>Best Regards,</div><div>Tony Wei</div></div><div class=3D"HO=
EnZb"><div class=3D"h5"><div class=3D"gmail_extra"><br><div class=3D"gmail_=
quote">2017-09-22 22:07 GMT+08:00 Chesnay Schepler <span dir=3D"ltr">&lt;<a=
 href=3D"mailto:chesnay@apache.org" target=3D"_blank">chesnay@apache.org</a=
>&gt;</span>:<br><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8=
ex;border-left:1px #ccc solid;padding-left:1ex">
 =20
   =20
 =20
  <div text=3D"#000000" bgcolor=3D"#FFFFFF">
    <div class=3D"m_4105823822822225131m_714822832086554093moz-cite-prefix"=
>The Prometheus reporter should work
      with 1.3.2.<br>
      <br>
      Does this also occur with the reporter that currently exists in
      1.4? (to rule out new bugs from the PR).<br>
      <br>
      To investigate this further, please set the logging level to WARN
      and try again, as all errors in the metric system are logged on
      that level.<div><div class=3D"m_4105823822822225131h5"><br>
      <br>
      On 22.09.2017 10:33, Tony Wei wrote:<br>
    </div></div></div><div><div class=3D"m_4105823822822225131h5">
    <blockquote type=3D"cite">
      <div dir=3D"ltr">Hi,=C2=A0
        <div><br>
        </div>
        <div>I have built the Prometheus reporter package from this PR=C2=
=A0<a href=3D"https://github.com/apache/flink/pull/4586" target=3D"_blank">=
https://github.com/apache/f<wbr>link/pull/4586</a>,
          and used it on Flink 1.3.2 to record every default metrics and
          those from `FlinkKafkaConsumer`.</div>
        <div><br>
        </div>
        <div>Originally, everything was fine. I could get those metrics
          in TM from Prometheus just like I saw on Flink Web UI.</div>
        <div>However, when I turned to JM, I found Prometheus gives this
          error to me:=C2=A0<span>Get <a href=3D"http://localhost:9249/metr=
ics" target=3D"_blank">http://localhost:9249/metrics</a>:
            EOF.</span></div>
        <div>I checked the log on JM and saw nothing in it. There was no
          error message and 9249 port was still alive.</div>
        <div><br>
        </div>
        <div>To figure out what happened, I created another cluster and
          I found Prometheus could connect to Flink cluster if there is
          no running job. After JM triggered or completed the first
          checkpoint, Prometheus started getting=C2=A0<span>ERR_EMPTY_RESPO=
NSE</span>=C2=A0fro<wbr>m
          JM, but not for TM. There was still no error in log file and
          9249 port was still alive.</div>
        <div><br>
        </div>
        <div>I was wondering where did the error occur. Flink or
          Prometheus reporter?</div>
        <div>Or It is incorrect to use Prometheus reporter on Flink
          1.3.2 ? Thank you.</div>
        <div><br>
        </div>
        <div>Best Regards,</div>
        <div>Tony Wei</div>
      </div>
    </blockquote>
    <p><br>
    </p>
  </div></div></div>

</blockquote></div><br></div>
</div></div></blockquote></div><br></div>

--001a114a8496a1d0100559d964c6--