flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jash, Shaswata (Nokia - IN/Bangalore)" <shaswata.j...@nokia.com>
Subject RE: Checkpointing in Flink 1.5.0
Date Tue, 03 Jul 2018 14:35:31 GMT
Hello Chesnay,

Cluster (in kubernetes)-wide checkpointing directory using glusterfs volume mount (thus file
access protocol file:///) was working fine till 1.4.2 for us. So we like to understand where
the breakage happened in 1.5.0.
Can you please mention me the relevant source code files related to rocksdb “custom file
path” parsing logic? We would be interested to investigate this.

I also observed below in the log –


Config uses deprecated configuration key 'state.backend.rocksdb.checkpointdir' instead of
proper key 'state.backend.rocksdb.localdir'
Regards,
Shaswata

From: Chesnay Schepler [mailto:chesnay@apache.org]
Sent: Tuesday, July 03, 2018 5:52 PM
To: Data Engineer <dataengineer21@gmail.com>
Cc: user@flink.apache.org
Subject: Re: Checkpointing in Flink 1.5.0

The code appears to be working fine.

This may happen because you're using a GlusterFS volume.
The RocksDBStateBackend uses java Files internally (NOT nio Paths), which AFAIK only work
properly against the plain local file-system.

The GlusterFS nio FIleSystem implementation also explicitly does not support conversions to
File<https://github.com/gluster/glusterfs-java-filesystem/blob/master/glusterfs-java-filesystem/src/main/java/com/peircean/glusterfs/GlusterPath.java#L271>.

On 03.07.2018 13:53, Chesnay Schepler wrote:
Thanks. Looks like RocksDBStateBackend.setDbStoragePaths has some custom file path parsing
logic, will probe it a bit to see what the issue is.

On 03.07.2018 13:45, Data Engineer wrote:

2018-07-03 11:30:35,703 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint      
  - --------------------------------------------------------------------------------

2018-07-03 11:30:35,705 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint      
  -  Starting StandaloneSessionClusterEntrypoint (Version: <unknown>, Rev:c61b108, Date:24.05.2018
@ 16:54:44 CEST)

2018-07-03 11:30:35,705 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint      
  -  OS current user: flink

2018-07-03 11:30:35,705 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint      
  -  Current Hadoop/Kerberos user: <no hadoop dependency found>

2018-07-03 11:30:35,706 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint      
  -  JVM: OpenJDK 64-Bit Server VM - Oracle Corporation - 1.8/25.171-b10

2018-07-03 11:30:35,706 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint      
  -  Maximum heap size: 981 MiBytes

2018-07-03 11:30:35,706 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint      
  -  JAVA_HOME: /etc/alternatives/jre_openjdk/

2018-07-03 11:30:35,707 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint      
  -  No Hadoop Dependency available

2018-07-03 11:30:35,707 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint      
  -  JVM Options:

2018-07-03 11:30:35,707 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint      
  -     -Xms1024m

2018-07-03 11:30:35,707 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint      
  -     -Xmx1024m

2018-07-03 11:30:35,708 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint      
  -     -Dlog.file=/opt/flink-1.5.0/log/flink--standalonesession-0-myfl-flink-jobmanager-7b4d8c4dd4-bv6zf.log

2018-07-03 11:30:35,708 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint      
  -     -Dlog4j.configuration=file:/opt/flink-1.5.0/conf/log4j.properties<file://opt/flink-1.5.0/conf/log4j.properties>

2018-07-03 11:30:35,708 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint      
  -     -Dlogback.configurationFile=file:/opt/flink-1.5.0/conf/logback.xml<file://opt/flink-1.5.0/conf/logback.xml>

2018-07-03 11:30:35,708 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint      
  -  Program Arguments:

2018-07-03 11:30:35,709 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint      
  -     --configDir

2018-07-03 11:30:35,709 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint      
  -     /opt/flink-1.5.0/conf

2018-07-03 11:30:35,709 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint      
  -     --executionMode

2018-07-03 11:30:35,709 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint      
  -     cluster

2018-07-03 11:30:35,710 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint      
  -     --host

2018-07-03 11:30:35,710 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint      
  -     cluster

2018-07-03 11:30:35,710 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint      
  -  Classpath: /opt/flink-1.5.0/lib/flink-cep_2.11-1.5.0.jar:/opt/flink-1.5.0/lib/flink-connectors-1.5.0.jar:/opt/flink-1.5.0/lib/flink-gelly_2.11-1.5.0.jar:/opt/flink-1.5.0/lib/flink-ml_2.11-1.5.0.jar:/opt/flink-1.5.0/lib/flink-python_2.11-1.5.0.jar:/opt/flink-1.5.0/lib/flink-table_2.11-1.5.0.jar:/opt/flink-1.5.0/lib/log4j-1.2.17.jar:/opt/flink-1.5.0/lib/slf4j-log4j12-1.7.7.jar:/opt/flink-1.5.0/lib/flink-dist_2.11-1.5.0.jar:::

2018-07-03 11:30:35,710 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint      
  - --------------------------------------------------------------------------------

2018-07-03 11:30:35,712 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint      
  - Registered UNIX signal handlers for [TERM, HUP, INT]

2018-07-03 11:30:35,720 INFO  org.apache.flink.configuration.GlobalConfiguration         
  - Loading configuration property: blob.server.port, 4124

2018-07-03 11:30:35,720 INFO  org.apache.flink.configuration.GlobalConfiguration         
  - Loading configuration property: jobmanager.rpc.address, myfl-flink-jobmanager

2018-07-03 11:30:35,720 INFO  org.apache.flink.configuration.GlobalConfiguration         
  - Loading configuration property: jobmanager.rpc.port, 4123

2018-07-03 11:30:35,721 INFO  org.apache.flink.configuration.GlobalConfiguration         
  - Loading configuration property: jobmanager.heap.mb, 1024

2018-07-03 11:30:35,721 INFO  org.apache.flink.configuration.GlobalConfiguration         
  - Loading configuration property: taskmanager.heap.mb, 1024

2018-07-03 11:30:35,721 INFO  org.apache.flink.configuration.GlobalConfiguration         
  - Loading configuration property: taskmanager.rpc.port, 4122

2018-07-03 11:30:35,721 INFO  org.apache.flink.configuration.GlobalConfiguration         
  - Loading configuration property: taskmanager.data.port, 4121

2018-07-03 11:30:35,721 INFO  org.apache.flink.configuration.GlobalConfiguration         
  - Loading configuration property: taskmanager.query.port, 4125

2018-07-03 11:30:35,722 INFO  org.apache.flink.configuration.GlobalConfiguration         
  - Loading configuration property: rest.port, 8081

2018-07-03 11:30:35,762 INFO  org.apache.flink.configuration.GlobalConfiguration         
  - Loading configuration property: rest.address, myfl-flink-jobmanager-ui

2018-07-03 11:30:35,762 INFO  org.apache.flink.configuration.GlobalConfiguration         
  - Loading configuration property: state.backend, rocksdb

2018-07-03 11:30:35,762 INFO  org.apache.flink.configuration.GlobalConfiguration         
  - Loading configuration property: state.checkpoints.dir, file:///opt/flink/share/myfl-flink/checkpoints/ext_checkpoints

2018-07-03 11:30:35,763 INFO  org.apache.flink.configuration.GlobalConfiguration         
  - Loading configuration property: state.backend.fs.checkpointdir, file:///opt/flink/share/myfl-flink/checkpoints/fs_state

2018-07-03 11:30:35,763 INFO  org.apache.flink.configuration.GlobalConfiguration         
  - Loading configuration property: state.backend.rocksdb.checkpointdir, file:///opt/flink/share/myfl-flink/checkpoints/rocksdb_state

2018-07-03 11:30:35,763 INFO  org.apache.flink.configuration.GlobalConfiguration         
  - Loading configuration property: taskmanager.numberOfTaskSlots, 4

2018-07-03 11:30:35,763 INFO  org.apache.flink.configuration.GlobalConfiguration         
  - Loading configuration property: restart-strategy, fixed-delay

2018-07-03 11:30:35,764 INFO  org.apache.flink.configuration.GlobalConfiguration         
  - Loading configuration property: restart-strategy.fixed-delay.attempts, 100

2018-07-03 11:30:35,764 INFO  org.apache.flink.configuration.GlobalConfiguration         
  - Loading configuration property: restart-strategy.fixed-delay.delay, 1 s

2018-07-03 11:30:35,885 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint      
  - Starting StandaloneSessionClusterEntrypoint.

2018-07-03 11:30:35,885 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint      
  - Install default filesystem.

2018-07-03 11:30:35,892 INFO  org.apache.flink.core.fs.FileSystem                        
  - Hadoop is not in the classpath/dependencies. The extended set of supported File Systems
via Hadoop is not available.

2018-07-03 11:30:35,963 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint      
  - Install security context.

2018-07-03 11:30:35,970 INFO  org.apache.flink.runtime.security.modules.HadoopModuleFactory
 - Cannot create Hadoop Security Module because Hadoop cannot be found in the Classpath.

2018-07-03 11:30:35,988 INFO  org.apache.flink.runtime.security.SecurityUtils            
  - Cannot install HadoopSecurityContext because Hadoop cannot be found in the Classpath.

2018-07-03 11:30:35,989 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint      
  - Initializing cluster services.

2018-07-03 11:30:36,003 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint      
  - Trying to start actor system at myfl-flink-jobmanager:4123

2018-07-03 11:30:37,288 INFO  akka.event.slf4j.Slf4jLogger                               
  - Slf4jLogger started

2018-07-03 11:30:37,396 INFO  akka.remote.Remoting                                       
  - Starting remoting

2018-07-03 11:30:37,583 INFO  akka.remote.Remoting                                       
  - Remoting started; listening on addresses :[akka.tcp://flink@myfl-flink-jobmanager:4123]

2018-07-03 11:30:37,591 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint      
  - Actor system started at akka.tcp://flink@myfl-flink-jobmanager:4123

2018-07-03 11:30:37,611 INFO  org.apache.flink.runtime.blob.BlobServer                   
  - Created BLOB server storage directory /tmp/blobStore-e445bc66-cee3-4a3d-b810-74df02627eca

2018-07-03 11:30:37,613 INFO  org.apache.flink.runtime.blob.BlobServer                   
  - Started BLOB server at 0.0.0.0:4124<http://0.0.0.0:4124> - max concurrent requests:
50 - max backlog: 1000

2018-07-03 11:30:37,629 INFO  org.apache.flink.runtime.metrics.MetricRegistryImpl        
  - No metrics reporter configured, no metrics will be exposed/reported.

2018-07-03 11:30:37,664 INFO  org.apache.flink.runtime.dispatcher.FileArchivedExecutionGraphStore
 - Initializing FileArchivedExecutionGraphStore: Storage directory /tmp/executionGraphStore-4ff546b1-4bfb-4911-9314-89c61d7e7149,
expiration time 3600000, maximum cache size 52428800 bytes.

2018-07-03 11:30:37,694 INFO  org.apache.flink.runtime.blob.TransientBlobCache           
  - Created BLOB cache storage directory /tmp/blobStore-7e0efdb8-f70b-42ed-9387-c0e1b8090b36

2018-07-03 11:30:37,702 WARN  org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint 
  - Upload directory /tmp/flink-web-e68a12b9-b9cc-4508-be00-4bf9f113afcd/flink-web-upload
does not exist, or has been deleted externally. Previously uploaded files are no longer available.

2018-07-03 11:30:37,703 INFO  org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint 
  - Created directory /tmp/flink-web-e68a12b9-b9cc-4508-be00-4bf9f113afcd/flink-web-upload
for file uploads.

2018-07-03 11:30:37,706 INFO  org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint 
  - Starting rest endpoint.

2018-07-03 11:30:38,369 INFO  org.apache.flink.runtime.webmonitor.WebMonitorUtils        
  - Determined location of main cluster component log file: /opt/flink-1.5.0/log/flink--standalonesession-0-myfl-flink-jobmanager-7b4d8c4dd4-bv6zf.log

2018-07-03 11:30:38,369 INFO  org.apache.flink.runtime.webmonitor.WebMonitorUtils        
  - Determined location of main cluster component stdout file: /opt/flink-1.5.0/log/flink--standalonesession-0-myfl-flink-jobmanager-7b4d8c4dd4-bv6zf.out

2018-07-03 11:30:38,567 INFO  org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint 
  - Rest endpoint listening at myfl-flink-jobmanager-ui:8081

2018-07-03 11:30:38,568 INFO  org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint 
  - http://myfl-flink-jobmanager-ui:8081 was granted leadership with leaderSessionID=00000000-0000-0000-0000-000000000000

2018-07-03 11:30:38,568 INFO  org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint 
  - Web frontend listening at http://myfl-flink-jobmanager-ui:8081.

2018-07-03 11:30:38,578 INFO  org.apache.flink.runtime.rpc.akka.AkkaRpcService           
  - Starting RPC endpoint for org.apache.flink.runtime.resourcemanager.StandaloneResourceManager
at akka://flink/user/resourcemanager .

2018-07-03 11:30:38,966 INFO  org.apache.flink.runtime.rpc.akka.AkkaRpcService           
  - Starting RPC endpoint for org.apache.flink.runtime.dispatcher.StandaloneDispatcher at
akka://flink/user/dispatcher .

2018-07-03 11:30:39,068 INFO  org.apache.flink.runtime.resourcemanager.StandaloneResourceManager
 - ResourceManager akka.tcp://flink@myfl-flink-jobmanager:4123/user/resourcemanager was granted
leadership with fencing token 00000000000000000000000000000000

2018-07-03 11:30:39,069 INFO  org.apache.flink.runtime.resourcemanager.slotmanager.SlotManager
 - Starting the SlotManager.

2018-07-03 11:30:39,164 INFO  org.apache.flink.runtime.dispatcher.StandaloneDispatcher   
  - Dispatcher akka.tcp://flink@myfl-flink-jobmanager:4123/user/dispatcher was granted leadership
with fencing token 00000000000000000000000000000000

2018-07-03 11:30:39,165 INFO  org.apache.flink.runtime.dispatcher.StandaloneDispatcher   
  - Recovering all persisted jobs.

2018-07-03 11:30:39,682 INFO  org.apache.flink.runtime.resourcemanager.StandaloneResourceManager
 - Replacing old registration of TaskExecutor 068c693b9585900f68c53b00507ee889.

2018-07-03 11:30:39,683 INFO  org.apache.flink.runtime.resourcemanager.slotmanager.SlotManager
 - Unregister TaskManager 8a5cee3aa38081030dc8558ac477d3b3 from the SlotManager.

2018-07-03 11:30:39,683 INFO  org.apache.flink.runtime.resourcemanager.StandaloneResourceManager
 - The target with resource ID 068c693b9585900f68c53b00507ee889 is already been monitored.

2018-07-03 11:30:39,770 INFO  org.apache.flink.runtime.resourcemanager.slotmanager.SlotManager
 - Registering TaskManager 068c693b9585900f68c53b00507ee889 under 03d409e5166fad4f4082b6165eb0de2e
at the SlotManager.

2018-07-03 11:34:20,257 INFO  org.apache.flink.runtime.dispatcher.StandaloneDispatcher   
  - Submitting job b684656d9afd75cc384a7bcd071bf55e (CSV Files Read -> CSV to Avro encode
-> Kafka publish).

2018-07-03 11:34:20,269 INFO  org.apache.flink.runtime.rpc.akka.AkkaRpcService           
  - Starting RPC endpoint for org.apache.flink.runtime.jobmaster.JobMaster at akka://flink/user/jobmanager_0
.

2018-07-03 11:34:20,278 INFO  org.apache.flink.runtime.jobmaster.JobMaster               
  - Initializing job CSV Files Read -> CSV to Avro encode -> Kafka publish (b684656d9afd75cc384a7bcd071bf55e).

2018-07-03 11:34:20,285 INFO  org.apache.flink.runtime.jobmaster.JobMaster               
  - Using restart strategy FixedDelayRestartStrategy(maxNumberRestartAttempts=2147483647,
delayBetweenRestartAttempts=0) for CSV Files Read -> CSV to Avro encode -> Kafka publish
(b684656d9afd75cc384a7bcd071bf55e).

2018-07-03 11:34:20,289 INFO  org.apache.flink.runtime.rpc.akka.AkkaRpcService           
  - Starting RPC endpoint for org.apache.flink.runtime.jobmaster.slotpool.SlotPool at akka://flink/user/67ceb2ae-1cb1-44be-a09e-601032e23fb5
.

2018-07-03 11:34:20,481 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph     
  - Job recovers via failover strategy: full graph restart

2018-07-03 11:34:20,562 INFO  org.apache.flink.runtime.jobmaster.JobMaster               
  - Running initialization on master for job CSV Files Read -> CSV to Avro encode ->
Kafka publish (b684656d9afd75cc384a7bcd071bf55e).

2018-07-03 11:34:20,563 INFO  org.apache.flink.runtime.jobmaster.JobMaster               
  - Successfully ran initialization on master in 0 ms.

2018-07-03 11:34:20,580 INFO  org.apache.flink.runtime.jobmaster.JobMaster               
  - Loading state backend via factory org.apache.flink.contrib.streaming.state.RocksDBStateBackendFactory

2018-07-03 11:34:20,590 WARN  org.apache.flink.configuration.Configuration               
  - Config uses deprecated configuration key 'state.backend.rocksdb.checkpointdir' instead
of proper key 'state.backend.rocksdb.localdir'

2018-07-03 11:34:20,592 ERROR org.apache.flink.runtime.rest.handler.job.JobSubmitHandler 
  - Implementation error: Unhandled exception.

org.apache.flink.util.FlinkException: Failed to submit job b684656d9afd75cc384a7bcd071bf55e.

  at org.apache.flink.runtime.dispatcher.Dispatcher.submitJob(Dispatcher.java:254)

  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

  at java.lang.reflect.Method.invoke(Method.java:498)

  at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcInvocation(AkkaRpcActor.java:247)

  at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:162)

  at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:70)

  at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.onReceive(AkkaRpcActor.java:142)

  at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.onReceive(FencedAkkaRpcActor.java:40)

  at akka.actor.UntypedActor$$anonfun$receive$1.applyOrElse(UntypedActor.scala:165)

  at akka.actor.Actor$class.aroundReceive(Actor.scala:502)

  at akka.actor.UntypedActor.aroundReceive(UntypedActor.scala:95)

  at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)

  at akka.actor.ActorCell.invoke(ActorCell.scala:495)

  at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)

  at akka.dispatch.Mailbox.run(Mailbox.scala:224)

  at akka.dispatch.Mailbox.exec(Mailbox.scala:234)

  at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)

  at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)

  at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)

  at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

Caused by: org.apache.flink.runtime.client.JobExecutionException: Could not set up JobManager

  at org.apache.flink.runtime.jobmaster.JobManagerRunner.<init>(JobManagerRunner.java:169)

  at org.apache.flink.runtime.dispatcher.Dispatcher$DefaultJobManagerRunnerFactory.createJobManagerRunner(Dispatcher.java:885)

  at org.apache.flink.runtime.dispatcher.Dispatcher.createJobManagerRunner(Dispatcher.java:287)

  at org.apache.flink.runtime.dispatcher.Dispatcher.runJob(Dispatcher.java:277)

  at org.apache.flink.runtime.dispatcher.Dispatcher.persistAndRunJob(Dispatcher.java:262)

  at org.apache.flink.runtime.dispatcher.Dispatcher.submitJob(Dispatcher.java:249)

  ... 21 more

Caused by: org.apache.flink.runtime.client.JobExecutionException: Could not instantiate configured
state backend

  at org.apache.flink.runtime.executiongraph.ExecutionGraphBuilder.buildGraph(ExecutionGraphBuilder.java:308)

  at org.apache.flink.runtime.executiongraph.ExecutionGraphBuilder.buildGraph(ExecutionGraphBuilder.java:100)

  at org.apache.flink.runtime.jobmaster.JobMaster.createExecutionGraph(JobMaster.java:1150)

  at org.apache.flink.runtime.jobmaster.JobMaster.createAndRestoreExecutionGraph(JobMaster.java:1130)

  at org.apache.flink.runtime.jobmaster.JobMaster.<init>(JobMaster.java:298)

  at org.apache.flink.runtime.jobmaster.JobManagerRunner.<init>(JobManagerRunner.java:151)

  ... 26 more

Caused by: org.apache.flink.configuration.IllegalConfigurationException: Invalid configuration
for RocksDB state backend's local storage directories: Relative paths are not supported

  at org.apache.flink.contrib.streaming.state.RocksDBStateBackend.<init>(RocksDBStateBackend.java:273)

  at org.apache.flink.contrib.streaming.state.RocksDBStateBackend.configure(RocksDBStateBackend.java:296)

  at org.apache.flink.contrib.streaming.state.RocksDBStateBackendFactory.createFromConfig(RocksDBStateBackendFactory.java:47)

  at org.apache.flink.contrib.streaming.state.RocksDBStateBackendFactory.createFromConfig(RocksDBStateBackendFactory.java:32)

  at org.apache.flink.runtime.state.StateBackendLoader.loadStateBackendFromConfig(StateBackendLoader.java:157)

  at org.apache.flink.runtime.state.StateBackendLoader.fromApplicationOrConfigOrDefault(StateBackendLoader.java:222)

  at org.apache.flink.runtime.executiongraph.ExecutionGraphBuilder.buildGraph(ExecutionGraphBuilder.java:304)

  ... 31 more

Caused by: java.lang.IllegalArgumentException: Relative paths are not supported

  at org.apache.flink.contrib.streaming.state.RocksDBStateBackend.setDbStoragePaths(RocksDBStateBackend.java:518)

  at org.apache.flink.contrib.streaming.state.RocksDBStateBackend.<init>(RocksDBStateBackend.java:269)

  ... 37 more


On Tue, Jul 3, 2018 at 5:11 PM, Chesnay Schepler <chesnay@apache.org<mailto:chesnay@apache.org>>
wrote:
Doesn't sound like intended behavior, can you give us the stacktrace?


On 03.07.2018 13:17, Data Engineer wrote:
 The Flink documentation says that we need to specify the filesystem type (file://<file:///>,
hdfs://) when configuring the rocksdb backend dir.
https://ci.apache.org/projects/flink/flink-docs-release-1.5/ops/state/state_backends.html#the-rocksdbstatebackend

But when I do this, I get an error on job submission saying that relative paths are not permitted
in the rocksdb stand backend.
I am submitting the job via flink cli (bin/flink run).

Also, even though I give a local file system path "file:///home/abc/share"<file:///home/abc/share>,
it is a shared GlusterFS volume mount, so it will be accessible by the JobManager and all
TaskManagers.

I removed the filesystem type from the rocksdb backend dir configuration, and though the job
got submitted, the rocksdb checkpoint directory was not created.
I have enabled checkpointing in my Flink application.

I am using Flink 1.5.0.

Any help or pointers would be appreciated.






Mime
View raw message