flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hailu, Andreas" <Andreas.Ha...@gs.com>
Subject RE: History Server Not Showing Any Jobs - File Not Found?
Date Fri, 24 Apr 2020 16:24:20 GMT
I'm having a further look at the code in HistoryServerStaticFileServerHandler - is there an
assumption about where overview.json is supposed to be located?

// ah

From: Hailu, Andreas [Engineering]
Sent: Wednesday, April 22, 2020 1:32 PM
To: 'Chesnay Schepler' <chesnay@apache.org>; Hailu, Andreas [Engineering] <Andreas.Hailu@ny.email.gs.com>;
user@flink.apache.org
Subject: RE: History Server Not Showing Any Jobs - File Not Found?

Hi Chesnay, thanks for responding. We're using Flink 1.9.1. I enabled DEBUG level logging
and this is something relevant I see:

2020-04-22 13:25:52,566 [Flink-HistoryServer-ArchiveFetcher-thread-1] DEBUG DFSInputStream
- Connecting to datanode 10.79.252.101:1019
2020-04-22 13:25:52,567 [Flink-HistoryServer-ArchiveFetcher-thread-1] DEBUG SaslDataTransferClient
- SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2020-04-22 13:25:52,567 [Flink-HistoryServer-ArchiveFetcher-thread-1] DEBUG SaslDataTransferClient
- SASL client skipping handshake in secured configuration with privileged port for addr =
/10.79.252.101, datanodeId = DatanodeI
nfoWithStorage[10.79.252.101:1019,DS-7f4ec55d-7c5f-4a0e-b817-d9e635480b21,DISK]
2020-04-22 13:25:52,571 [Flink-HistoryServer-ArchiveFetcher-thread-1] DEBUG DFSInputStream
- DFSInputStream has been closed already
2020-04-22 13:25:52,573 [nioEventLoopGroup-3-6] DEBUG HistoryServerStaticFileServerHandler
- Unable to load requested file /jobs/overview.json from classloader
2020-04-22 13:25:52,576 [IPC Parameter Sending Thread #0] DEBUG Client$Connection$3 - IPC
Client (1578587450) connection to d279536-002.dc.gs.com/10.59.61.87:8020 from delp@GS.COM<mailto:delp@GS.COM>
sending #1391

Aside from that, it looks like a lot of logging around datanodes and block location metadata.
Did I miss something in my classpath, perhaps? If so, do you have a suggestion on what I could
try?

// ah

From: Chesnay Schepler <chesnay@apache.org<mailto:chesnay@apache.org>>
Sent: Wednesday, April 22, 2020 2:16 AM
To: Hailu, Andreas [Engineering] <Andreas.Hailu@ny.email.gs.com<mailto:Andreas.Hailu@ny.email.gs.com>>;
user@flink.apache.org<mailto:user@flink.apache.org>
Subject: Re: History Server Not Showing Any Jobs - File Not Found?

Which Flink version are you using?
Have you checked the history server logs after enabling debug logging?

On 21/04/2020 17:16, Hailu, Andreas [Engineering] wrote:
Hi,

I'm trying to set up the History Server, but none of my applications are showing up in the
Web UI. Looking at the console, I see that all of the calls to /overview return the following
404 response: {"errors":["File not found."]}.

I've set up my configuration as follows:

JobManager Archive directory:
jobmanager.archive.fs.dir: hdfs:///user/p2epda/lake/delp_qa/flink_hs/
-bash-4.1$ hdfs dfs -ls /user/p2epda/lake/delp_qa/flink_hs
Found 44282 items
-rw-r-----   3 delp datalake_admin_dev      50569 2020-03-21 23:17 /user/p2epda/lake/delp_qa/flink_hs/000144dba9dc0f235768a46b2f26e936
-rw-r-----   3 delp datalake_admin_dev      49578 2020-03-03 08:45 /user/p2epda/lake/delp_qa/flink_hs/000347625d8128ee3fd0b672018e38a5
-rw-r-----   3 delp datalake_admin_dev      50842 2020-03-24 15:19 /user/p2epda/lake/delp_qa/flink_hs/0004be6ce01ba9677d1eb619ad0fa757
...
...

History Server will fetch the archived jobs from the same location:
historyserver.archive.fs.dir: hdfs:///user/p2epda/lake/delp_qa/flink_hs/

So I'm able to confirm that there are indeed archived applications that I should be able to
view in the histserver. I'm not able to find out what file the overview service is looking
for from the repository - any suggestions as to what I could look into next?

Best,
Andreas

________________________________

Your Personal Data: We may collect and process information about you that may be subject to
data protection laws. For more information about how we use and disclose your personal data,
how we protect your information, our legal basis to use your information, your rights and
who you can contact, please refer to: www.gs.com/privacy-notices<http://www.gs.com/privacy-notices>



________________________________

Your Personal Data: We may collect and process information about you that may be subject to
data protection laws. For more information about how we use and disclose your personal data,
how we protect your information, our legal basis to use your information, your rights and
who you can contact, please refer to: www.gs.com/privacy-notices<http://www.gs.com/privacy-notices>

Mime
View raw message