Return-Path: X-Original-To: apmail-hadoop-common-user-archive@www.apache.org Delivered-To: apmail-hadoop-common-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 559BA17D94 for ; Fri, 8 May 2015 17:05:26 +0000 (UTC) Received: (qmail 61838 invoked by uid 500); 8 May 2015 17:05:18 -0000 Delivered-To: apmail-hadoop-common-user-archive@hadoop.apache.org Received: (qmail 61739 invoked by uid 500); 8 May 2015 17:05:18 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 61729 invoked by uid 99); 8 May 2015 17:05:18 -0000 Received: from Unknown (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 08 May 2015 17:05:18 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 0BD101827D4 for ; Fri, 8 May 2015 17:05:18 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.981 X-Spam-Level: ** X-Spam-Status: No, score=2.981 tagged_above=-999 required=6.31 tests=[HTML_MESSAGE=3, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, URIBL_BLOCKED=0.001] autolearn=disabled Received: from mx1-us-east.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id B7h_kz2c0mgd for ; Fri, 8 May 2015 17:05:04 +0000 (UTC) Received: from mail-wi0-f182.google.com (mail-wi0-f182.google.com [209.85.212.182]) by mx1-us-east.apache.org (ASF Mail Server at mx1-us-east.apache.org) with ESMTPS id F1DDF40E1C for ; Fri, 8 May 2015 17:05:03 +0000 (UTC) Received: by widdi4 with SMTP id di4so37136704wid.0 for ; Fri, 08 May 2015 10:05:03 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:mime-version:subject:message-id:date :references:in-reply-to:to:content-type:content-transfer-encoding; bh=63c84HCmqGDANhi7tR89sXQNaECs5d2o3x8VQjZYLRY=; b=lrjTEqwyAOux1Bm29EfV+K3tvmb1aREPxfwfemRSrrmHzBl7eVEQtLNaSXg6jhC1Gg bvLb0EC2F1J0SGOs/wI2IgdfqCin1HffZDey20ZZKunqPHZQrTaR4lSW5pQvTCu9JyQ2 XRRmDiWq1lGMgLdfIJpv0wo7gUEsRDUO0gExq3Dy0TuwHCyML6mPzR3BRUgsDaIDBtgY zEMfZsoB028GcZC4GDYm+UctZnW9rjZL36IcihqJfD+RWnCwBvHHZqupMM1KTJXRf1o/ huc/LgXG+FUjbKV7GlUEPAPGa+RnO3DgbKGiyD9oHwOmtHcQQU2vJVRDdQ6Qb07FXFGc kkvg== X-Gm-Message-State: ALoCoQliE5qt6kjRsD7tHxBvX4LLtpbE5KJ+f9LCPIWhYuOsH48Ld90Zcp1XsOpTJo3dkivlMvvF7Lq1HF6j8l5nacZJmPHFkbbex2bVz10NYjV6C8Gi528= X-Received: by 10.180.97.7 with SMTP id dw7mr7998733wib.74.1431104702910; Fri, 08 May 2015 10:05:02 -0700 (PDT) Received: from ?IPv6:2003:84:ae0e:e500:e0d0:2edb:6924:2c98? (p20030084AE0EE500E0D02EDB69242C98.dip0.t-ipconnect.de. [2003:84:ae0e:e500:e0d0:2edb:6924:2c98]) by mx.google.com with ESMTPSA id q4sm9186078wja.24.2015.05.08.10.05.01 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Fri, 08 May 2015 10:05:02 -0700 (PDT) From: Silvan Kaiser Mime-Version: 1.0 (1.0) Subject: Re: URI missing scheme and authority in job start with new FileSystem implementation Message-Id: <99A73733-25A0-4F4B-BBD1-CEADC37F6196@quobyte.com> Date: Fri, 8 May 2015 19:04:59 +0200 References: In-Reply-To: To: "user@hadoop.apache.org" X-Mailer: iPhone Mail (12F70) Content-Type: multipart/alternative; boundary=Apple-Mail-67B37C1D-ADD6-4D9C-B86D-EAC05C992470 Content-Transfer-Encoding: 7bit --Apple-Mail-67B37C1D-ADD6-4D9C-B86D-EAC05C992470 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Dear Varun, thanks for the hints! Yep it's Mapreduce.=20 Can't look at the code right now but I cannot remember overriding resolvePa= th. Will check it out and give feedback later on! Best Silvan Von meinem iPhone gesendet > Am 08.05.2015 um 18:10 schrieb Varun Vasudev : >=20 > Hi Silvan, >=20 > Are you trying to run MapReduce? You could look at createApplicationSubmi= ssionContext and createApplicationResource in YarnRunner.java. Have you ove= rridden the resolvePath function in your filesystem implementation? >=20 > -Varun >=20 > From: Silvan Kaiser > Reply-To: "user@hadoop.apache.org" > Date: Friday, May 8, 2015 at 3:52 PM > To: "user@hadoop.apache.org" > Subject: URI missing scheme and authority in job start with new FileSyst= em implementation >=20 >=20 > Hello! > I am running a new FileSystem implementation, testing it with a simple st= andalone operation from the Hadoop "Single Node Setup" page. Staging exampl= e jobs works alright but upon scheduling the container management fails bec= ause of an URI without scheme > (and authority). >=20 > NodeManager log snippet: > ... >=20 > 2015-05-08 15:18:15,331 INFO org.apache.hadoop.yarn.server.nodemanager.co= ntainermanager.container.ContainerImpl: Got exception parsing jobSubmitDir/= job.splitmetainfo and value resource { port: -1 file: "/tmp/hadoop-yarn/sta= ging/kaisers/.staging/job_1430315033077_0086/job.splitmetainfo" > } size: 771 timestamp: 1431091092000 type: FILE visibility: APPLICATION > 2015-05-08 15:18:15,331 WARN org.apache.hadoop.yarn.server.nodemanager.co= ntainermanager.container.ContainerImpl: Failed to parse resource-request > java.net.URISyntaxException: Expected scheme name at index 0: :///tmp/had= oop-yarn/staging/kaisers/.staging/job_1430315033077_0086/job.splitmetainfo >=20 > ... >=20 >=20 > The path element of the URI is correct, it simply is missing it's scheme = and authority but i cannot find out why so far. > Please note that the URIs used during staging are correct and contain sch= eme & authority correctly. This means the incomplete URI is created during = job creation. >=20 > My guess so far: this is either because I am missing some configuration f= or the Nodemanager (therefore he does not know about my FileSystem implemen= tation and throws the unknown scheme & authority out the window) or that th= e FileSystem implementation has > an error / did not override a method that requires adoption. >=20 > Can someone please give me a hint where the URIs used by the NodeManager = are constructed prior to scheduling? This would allow me to trace back wher= e the incomplete URI is coming from... >=20 > Best regards > Silvan >=20 >=20 > -- > Quobyte GmbH > Boyenstr. 41 - 10115 Berlin-Mitte - Germany > +49-30-814 591 800 - www.quobyte.com > Amtsgericht Berlin-Charlottenburg, HRB 149012B > Management board: Dr. Felix Hupfeld, Dr. Bj=C3=B6rn Kolbeck, Dr. Jan Sten= der >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 > -- > Quobyte GmbH > Hardenbergplatz 2 - 10623 Berlin - Germany > +49-30-814 591 800 - www.quobyte.com Amtsgericht= Berlin-Charlottenburg, HRB 149012B > management board: Dr. Felix Hupfeld, Dr. Bj=C3=B6rn Kolbeck, Dr. Jan Sten= der --=20 -- *Quobyte* GmbH Hardenbergplatz 2 - 10623 Berlin - Germany +49-30-814 591 800 - www.quobyte.com Amtsgericht Berlin-Charlottenburg, HRB 149012B management board: Dr. Felix Hupfeld, Dr. Bj=C3=B6rn Kolbeck, Dr. Jan Stende= r --Apple-Mail-67B37C1D-ADD6-4D9C-B86D-EAC05C992470 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Dear Varun,
thanks for t= he hints! Yep it's Mapreduce. 
Can't look at the code right = now but I cannot remember overriding resolvePath. Will check it out and giv= e feedback later on!
Best
Silvan


Von = meinem iPhone gesendet

Am 08.05.2015 um 18:10 schrieb Varun V= asudev <vvasudev@hortonworks= .com>:

Hi Silvan,

Are you trying to run MapReduce? You could look at createApplicationSu= bmissionContext and createApplicationResource in YarnRunner.java. Have you = overridden the resolvePath function in your filesystem implementation?

-Varun

From:  Silvan Kaiser
Date:  Friday, May 8, 2015 at 3:52 PM
Subject:  URI missing scheme and authority in job start with= new FileSystem implementation


Hello!
I am running a new FileSystem implementation, testing it with a simple= standalone operation from the Hadoop "Single Node Setup" page. Staging exa= mple jobs works alright but upon scheduling the container management fails = because of an URI without scheme
(and authority).

NodeManager log snippet:
...

2015-05-08 15:18:15,331 INFO org.apache.hadoop.yarn.server.nodemanager= .containermanager.container.ContainerImpl: Got exception parsing jobSubmitD= ir/job.splitmetainfo and value resource { port: -1 file: "/tmp/hadoop-yarn/= staging/kaisers/.staging/job_1430315033077_0086/job.splitmetainfo"
} size: 771 timestamp: 1431091092000 type: FILE visibility: APPLICATIO= N
2015-05-08 15:18:15,331 WARN org.apache.hadoop.yarn.server.nodemanager= .containermanager.container.ContainerImpl: Failed to parse resource-request=
java.net.URISyntaxException: Expected scheme name at index 0: :///tmp/= hadoop-yarn/staging/kaisers/.staging/job_1430315033077_0086/job.splitmetain= fo

...


The path element of the URI is correct, it simply is missing it's sche= me and authority but i cannot find out why so far.
Please note that the URIs used during staging are correct and contain = scheme & authority correctly. This means the incomplete URI is created = during job creation.

My guess so far: this is either because I am missing some configuratio= n for the Nodemanager (therefore he does not know about my FileSystem imple= mentation and throws the unknown scheme & authority out the window) or = that the FileSystem implementation has
an error / did not override a method that requires adoption.

Can someone please give me a hint where the URIs used by the NodeManag= er are constructed prior to scheduling? This would allow me to trace back w= here the incomplete URI is coming from...

Best regards
Silvan


--
Quobyte GmbH
Boyenstr. 41 - 10115 Berlin-Mitte - Germany
Amtsgericht Berlin-Charlottenburg, HRB 149012B
Management board: Dr. Felix Hupfeld, Dr. Bj=C3=B6rn Kolbeck, Dr. Jan S= tender









--
Quobyte GmbH
Hardenbergplatz 2 - 10623 Berlin - Germany
+49-30-814 591 800 - www.quobyte.co= m <http://www.quo= byte.com/>Amtsgericht Berlin-Charlottenburg, HRB 149012B
management board: Dr. Felix Hupfeld, Dr. Bj=C3=B6rn Kolbeck, Dr. Jan S= tender

=
--
Quobyte=C2=A0GmbH
Hardenbergplatz 2 - 10623 Berlin - Germany
= +49-30-814 591 800=C2=A0-=C2=A0www.quobyte.com
Amtsge= richt Berlin-Charlottenburg, HRB 149012B
management board: Dr. Felix Hup= feld, Dr. Bj=C3=B6rn Kolbeck, Dr. Jan Stender
--Apple-Mail-67B37C1D-ADD6-4D9C-B86D-EAC05C992470--