From user-return-157-archive-asf-public=cust-asf.ponee.io@gobblin.incubator.apache.org Thu Jan 25 10:02:02 2018 Return-Path: X-Original-To: archive-asf-public@eu.ponee.io Delivered-To: archive-asf-public@eu.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by mx-eu-01.ponee.io (Postfix) with ESMTP id 8F826180651 for ; Thu, 25 Jan 2018 10:02:02 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 7F288160C3D; Thu, 25 Jan 2018 09:02:02 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 77925160C13 for ; Thu, 25 Jan 2018 10:02:01 +0100 (CET) Received: (qmail 13773 invoked by uid 500); 25 Jan 2018 09:02:00 -0000 Mailing-List: contact user-help@gobblin.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@gobblin.incubator.apache.org Delivered-To: mailing list user@gobblin.incubator.apache.org Received: (qmail 13762 invoked by uid 99); 25 Jan 2018 09:02:00 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 25 Jan 2018 09:02:00 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 1F7C5C15CD for ; Thu, 25 Jan 2018 09:02:00 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.1 X-Spam-Level: ** X-Spam-Status: No, score=2.1 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, HTML_MESSAGE=2, T_DKIM_INVALID=0.01, T_RP_MATCHES_RCVD=-0.01] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=neutral reason="invalid (public key: not available)" header.d=proum.de Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id yFF16cX27D43 for ; Thu, 25 Jan 2018 09:01:54 +0000 (UTC) Received: from mail.proum.de (proum.de [176.9.114.245]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 2F8985F1B3 for ; Thu, 25 Jan 2018 09:01:53 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by mail.proum.de (Postfix) with ESMTP id F378918E394F for ; Thu, 25 Jan 2018 09:01:46 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at proum.de Received: from mail.proum.de ([127.0.0.1]) by localhost (mail.proum.de [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id bixw8ZnhrtfQ for ; Thu, 25 Jan 2018 09:01:41 +0000 (UTC) Received: from [192.168.188.44] (unknown [80.83.103.98]) (Authenticated sender: bka@mail.proum.de) by mail.proum.de (Postfix) with ESMTPSA id CD42D18E3905 for ; Thu, 25 Jan 2018 09:01:40 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=proum.de; s=mail; t=1516870901; bh=GAf55FiqDFFuZSxISTlq2aF6rk8TE8rLUxd3PcKPO0k=; h=From:Subject:To; b=H2Hc/O3QeDVUMCFIqWIjyN+rM3blb88n3dvT7F1NXY5Rth1YJr/KPhISlmJmZEUFV kQ7cxrOG2HqG18mxmLo68uHiM/XDTHEyBK3r6nTqZ6j5i5uLksJSZWfCxOPOlYrVQa ahv+jZrjQrFwasg/Jbdwv8MeF+EELPIgDR8X69RI= From: Birger Kamp Content-Type: multipart/alternative; boundary="Apple-Mail=_75DD5642-CE07-46E4-BC2D-8FB16D00C2B6" Mime-Version: 1.0 (Mac OS X Mail 11.2 \(3445.5.20\)) Subject: Set JobConf Dir in Gobblin CLI Message-Id: <0C985CB8-C555-465D-8074-2DA72655AA36@proum.de> Date: Thu, 25 Jan 2018 10:01:40 +0100 To: user@gobblin.incubator.apache.org X-Mailer: Apple Mail (2.3445.5.20) --Apple-Mail=_75DD5642-CE07-46E4-BC2D-8FB16D00C2B6 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Hi friends of Gobblin, I=E2=80=99m trying to build a Gobblin CLI application which runs = multiple jobs. For this I copied the WikipediaExampleApp and modified it = for myself. Let=E2=80=99s name it =E2=80=9CMyApp=E2=80=9D. MyApp has just a single important method and that=E2=80=99s the = constructor: public IngestionApp(String pathToJobs, String pathToJar) { super("IngestAllJobs"); = this.setConfiguration(ConfigurationKeys.JOB_CONFIG_FILE_GENERAL_PATH_KEY, = pathToJobs); this.distributeJar(pathToJar); } There you can see how I=E2=80=99m configuring the FQDN path to the jobs. = This works already if I=E2=80=99m using the Standalone-Cluster, so it = should work also for MyApp. If running "bin/gobblin run ingestAllJobs hdfs://localhost:9000/jobs = path/to/distribute_dependencies.jar=E2=80=9D the QuickApp is starting = and stops with a NPE: 2018-01-25 09:48:12 WARN JobContext - Property task.data.root.dir is = missing. 2018-01-25 09:48:12 ERROR EmbeddedGobblin - Job launch failed: = java.lang.RuntimeException: JobLauncher creation failed: = java.lang.RuntimeException: Failed to create job launcher: = java.lang.NullPointerException java.lang.RuntimeException: JobLauncher creation failed: = java.lang.RuntimeException: Failed to create job launcher: = java.lang.NullPointerException at = org.apache.gobblin.runtime.job_exec.JobLauncherExecutionDriver.createLaunc= her(JobLauncherExecutionDriver.java:177) at = org.apache.gobblin.runtime.job_exec.JobLauncherExecutionDriver.create(JobL= auncherExecutionDriver.java:121) at = org.apache.gobblin.runtime.job_exec.JobLauncherExecutionDriver$Launcher.la= unchJob(JobLauncherExecutionDriver.java:453) at = org.apache.gobblin.runtime.instance.DefaultGobblinInstanceDriverImpl$JobSp= ecRunnable.run(DefaultGobblinInstanceDriverImpl.java:209) at = org.apache.gobblin.runtime.scheduler.AbstractJobSpecScheduler$TriggerRunna= ble.run(AbstractJobSpecScheduler.java:177) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.RuntimeException: Failed to create job launcher: = java.lang.NullPointerException at = org.apache.gobblin.runtime.JobLauncherFactory.newJobLauncher(JobLauncherFa= ctory.java:120) at = org.apache.gobblin.runtime.JobLauncherFactory.newJobLauncher(JobLauncherFa= ctory.java:85) at = org.apache.gobblin.runtime.job_exec.JobLauncherExecutionDriver.createLaunc= her(JobLauncherExecutionDriver.java:174) ... 5 more Caused by: java.lang.NullPointerException at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at = org.apache.gobblin.runtime.JobContext.createSource(JobContext.java:242) at = org.apache.gobblin.runtime.JobContext.(JobContext.java:172) at = org.apache.gobblin.runtime.AbstractJobLauncher.(AbstractJobLauncher.= java:183) at = org.apache.gobblin.runtime.local.LocalJobLauncher.(LocalJobLauncher.= java:77) at = org.apache.gobblin.runtime.JobLauncherFactory.newJobLauncher(JobLauncherFa= ctory.java:106) ... 7 more 2018-01-25 09:48:16 WARN EmbeddedGobblin - Timeout waiting for job to = start. Aborting. 2018-01-25 09:48:16 INFO EmbeddedGobblin - Shutting down driver =E2=80=A6= After some investigating I=E2=80=99m not sure if the QuickApp is even = build to work this way. Seems like you can only configure a single job = into a QuickApp by defining concrete Source, Extractor, Converter and = stuff. Or use a template. But it=E2=80=99s not working for configuring a whole job directory. Is = that correct? Is there a workaround? Or do I have to touch Gobblin = source code for getting this? Thank you for all your help! (In Mails and Gitter!) Birger= --Apple-Mail=_75DD5642-CE07-46E4-BC2D-8FB16D00C2B6 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 Hi = friends of Gobblin,

I=E2=80=99m trying to build a Gobblin CLI application which = runs multiple jobs. For this I copied the WikipediaExampleApp and = modified it for myself. Let=E2=80=99s name it =E2=80=9CMyApp=E2=80=9D.

MyApp has just a = single important method and that=E2=80=99s the constructor:

public =
IngestionApp(String pathToJobs, String pathToJar) {
 =
  super("IngestAllJobs");
   this.setConfiguration(ConfigurationKeys.JOB_CONFIG_FILE_GENERAL_PATH_KEY, pathToJobs);
   this.distributeJar(pathToJar);
}
There you can see how I=E2=80=99m configuring the FQDN path = to the jobs. This works already if I=E2=80=99m using the = Standalone-Cluster, so it should work also for MyApp.

If running "bin/gobblin = run ingestAllJobs hdfs://localhost:9000/jobs = path/to/distribute_dependencies.jar=E2=80=9D the QuickApp is starting = and stops with a NPE:

2018-01-25 09:48:12 WARN  JobContext - = Property task.data.root.dir is missing.
2018-01-25 = 09:48:12 ERROR EmbeddedGobblin - Job launch failed: = java.lang.RuntimeException: JobLauncher creation failed: = java.lang.RuntimeException: Failed to create job launcher: = java.lang.NullPointerException
java.lang.RuntimeException: JobLauncher creation failed: = java.lang.RuntimeException: Failed to create job launcher: = java.lang.NullPointerException
at = org.apache.gobblin.runtime.job_exec.JobLauncherExecutionDriver.createLaunc= her(JobLauncherExecutionDriver.java:177)
at = org.apache.gobblin.runtime.job_exec.JobLauncherExecutionDriver.create(JobL= auncherExecutionDriver.java:121)
at = org.apache.gobblin.runtime.job_exec.JobLauncherExecutionDriver$Launcher.la= unchJob(JobLauncherExecutionDriver.java:453)
at = org.apache.gobblin.runtime.instance.DefaultGobblinInstanceDriverImpl$JobSp= ecRunnable.run(DefaultGobblinInstanceDriverImpl.java:209)
= at = org.apache.gobblin.runtime.scheduler.AbstractJobSpecScheduler$TriggerRunna= ble.run(AbstractJobSpecScheduler.java:177)
at = java.lang.Thread.run(Thread.java:745)
Caused by: = java.lang.RuntimeException: Failed to create job launcher: = java.lang.NullPointerException
at = org.apache.gobblin.runtime.JobLauncherFactory.newJobLauncher(JobLauncherFa= ctory.java:120)
at = org.apache.gobblin.runtime.JobLauncherFactory.newJobLauncher(JobLauncherFa= ctory.java:85)
at = org.apache.gobblin.runtime.job_exec.JobLauncherExecutionDriver.createLaunc= her(JobLauncherExecutionDriver.java:174)
... 5 = more
Caused by: = java.lang.NullPointerException
at = java.lang.Class.forName0(Native Method)
at = java.lang.Class.forName(Class.java:264)
at = org.apache.gobblin.runtime.JobContext.createSource(JobContext.java:242)
at = org.apache.gobblin.runtime.JobContext.<init>(JobContext.java:172)
at = org.apache.gobblin.runtime.AbstractJobLauncher.<init>(AbstractJobLau= ncher.java:183)
at = org.apache.gobblin.runtime.local.LocalJobLauncher.<init>(LocalJobLau= ncher.java:77)
at = org.apache.gobblin.runtime.JobLauncherFactory.newJobLauncher(JobLauncherFa= ctory.java:106)
... 7 more
2018-01-25 09:48:16 WARN  EmbeddedGobblin - Timeout = waiting for job to start. Aborting.
2018-01-25 = 09:48:16 INFO  EmbeddedGobblin - Shutting down driver = =E2=80=A6

After some investigating I=E2=80=99m not sure if the QuickApp = is even build to work this way. Seems like you can only configure a = single job into a QuickApp by defining concrete Source, Extractor, = Converter and stuff. Or use a template.
But it=E2=80=99= s not working for configuring a whole job directory. Is that correct? Is = there a workaround? Or do I have to touch Gobblin source code for = getting this?

Thank you for all your help! (In Mails and Gitter!)
Birger
= --Apple-Mail=_75DD5642-CE07-46E4-BC2D-8FB16D00C2B6--