gobblin-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Birger Kamp <birger.k...@proum.de>
Subject Set JobConf Dir in Gobblin CLI
Date Thu, 25 Jan 2018 09:01:40 GMT
Hi friends of Gobblin,

I’m trying to build a Gobblin CLI application which runs multiple jobs. For this I copied
the WikipediaExampleApp and modified it for myself. Let’s name it “MyApp”.

MyApp has just a single important method and that’s the constructor:

public IngestionApp(String pathToJobs, String pathToJar) {
   super("IngestAllJobs");
   this.setConfiguration(ConfigurationKeys.JOB_CONFIG_FILE_GENERAL_PATH_KEY, pathToJobs);
   this.distributeJar(pathToJar);
}
There you can see how I’m configuring the FQDN path to the jobs. This works already if I’m
using the Standalone-Cluster, so it should work also for MyApp.

If running "bin/gobblin run ingestAllJobs hdfs://localhost:9000/jobs path/to/distribute_dependencies.jar”
the QuickApp is starting and stops with a NPE:

2018-01-25 09:48:12 WARN  JobContext - Property task.data.root.dir is missing.
2018-01-25 09:48:12 ERROR EmbeddedGobblin - Job launch failed: java.lang.RuntimeException:
JobLauncher creation failed: java.lang.RuntimeException: Failed to create job launcher: java.lang.NullPointerException
java.lang.RuntimeException: JobLauncher creation failed: java.lang.RuntimeException: Failed
to create job launcher: java.lang.NullPointerException
	at org.apache.gobblin.runtime.job_exec.JobLauncherExecutionDriver.createLauncher(JobLauncherExecutionDriver.java:177)
	at org.apache.gobblin.runtime.job_exec.JobLauncherExecutionDriver.create(JobLauncherExecutionDriver.java:121)
	at org.apache.gobblin.runtime.job_exec.JobLauncherExecutionDriver$Launcher.launchJob(JobLauncherExecutionDriver.java:453)
	at org.apache.gobblin.runtime.instance.DefaultGobblinInstanceDriverImpl$JobSpecRunnable.run(DefaultGobblinInstanceDriverImpl.java:209)
	at org.apache.gobblin.runtime.scheduler.AbstractJobSpecScheduler$TriggerRunnable.run(AbstractJobSpecScheduler.java:177)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: Failed to create job launcher: java.lang.NullPointerException
	at org.apache.gobblin.runtime.JobLauncherFactory.newJobLauncher(JobLauncherFactory.java:120)
	at org.apache.gobblin.runtime.JobLauncherFactory.newJobLauncher(JobLauncherFactory.java:85)
	at org.apache.gobblin.runtime.job_exec.JobLauncherExecutionDriver.createLauncher(JobLauncherExecutionDriver.java:174)
	... 5 more
Caused by: java.lang.NullPointerException
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:264)
	at org.apache.gobblin.runtime.JobContext.createSource(JobContext.java:242)
	at org.apache.gobblin.runtime.JobContext.<init>(JobContext.java:172)
	at org.apache.gobblin.runtime.AbstractJobLauncher.<init>(AbstractJobLauncher.java:183)
	at org.apache.gobblin.runtime.local.LocalJobLauncher.<init>(LocalJobLauncher.java:77)
	at org.apache.gobblin.runtime.JobLauncherFactory.newJobLauncher(JobLauncherFactory.java:106)
	... 7 more
2018-01-25 09:48:16 WARN  EmbeddedGobblin - Timeout waiting for job to start. Aborting.
2018-01-25 09:48:16 INFO  EmbeddedGobblin - Shutting down driver …

After some investigating I’m not sure if the QuickApp is even build to work this way. Seems
like you can only configure a single job into a QuickApp by defining concrete Source, Extractor,
Converter and stuff. Or use a template.
But it’s not working for configuring a whole job directory. Is that correct? Is there a
workaround? Or do I have to touch Gobblin source code for getting this?

Thank you for all your help! (In Mails and Gitter!)
Birger
Mime
View raw message