Return-Path: X-Original-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 8378417A0B for ; Mon, 20 Apr 2015 13:19:17 +0000 (UTC) Received: (qmail 26746 invoked by uid 500); 20 Apr 2015 13:19:12 -0000 Delivered-To: apmail-hadoop-mapreduce-user-archive@hadoop.apache.org Received: (qmail 26650 invoked by uid 500); 20 Apr 2015 13:19:12 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 26640 invoked by uid 99); 20 Apr 2015 13:19:11 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Apr 2015 13:19:11 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: message received from 54.76.25.247 which is an MX secondary for user@hadoop.apache.org) Received: from [54.76.25.247] (HELO mx1-eu-west.apache.org) (54.76.25.247) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Apr 2015 13:18:45 +0000 Received: from mail-oi0-f54.google.com (mail-oi0-f54.google.com [209.85.218.54]) by mx1-eu-west.apache.org (ASF Mail Server at mx1-eu-west.apache.org) with ESMTPS id 0A5AC2532C for ; Mon, 20 Apr 2015 13:18:44 +0000 (UTC) Received: by oign205 with SMTP id n205so124336391oig.2 for ; Mon, 20 Apr 2015 06:17:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=IkOGdB0PEdizv5JN8xVXliRYi2NFgFYOz4M+uDHvzqM=; b=WgLi5qh8BexZUMJ8MfiUt7QCqJjPHUaHbvIShNWohh0wFF2e4s/XQUZUhI2/D8m1VP LFNmeIIOpLmYqC0aoYH8TFa4cuIz/Lq/1XDiU8BWIObMAp9g61kfsrQ5dPx5DC/8NBXL JNp/WR4MEym3kWVlz7hXd21HgbNNpLgERwIOE/0jepScUk67tDKnqxl5edY1f5tXZfBH YkE0IRcIBSKS8qgV6+g1JvUYaDOypf7jyOO+y4902Rn8ZlN5OMcSiY3iUDKM6GxSbLjO h2SIvEZlKkiV9eRE9Lbjksj/5M5ApxOGSwkEU/2KpMMIPMv29PVDoSW6GKknHEqUV/3m m3/A== MIME-Version: 1.0 X-Received: by 10.182.92.165 with SMTP id cn5mr14296221obb.10.1429535827008; Mon, 20 Apr 2015 06:17:07 -0700 (PDT) Received: by 10.202.189.70 with HTTP; Mon, 20 Apr 2015 06:17:06 -0700 (PDT) Date: Mon, 20 Apr 2015 09:17:06 -0400 Message-ID: Subject: Unable to Find S3N Filesystem Hadoop 2.6 From: Billy Watson To: "user@hadoop.apache.org" Content-Type: multipart/alternative; boundary=001a11c330d029a282051427bf06 X-Virus-Checked: Checked by ClamAV on apache.org --001a11c330d029a282051427bf06 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Hi, I am able to run a `hadoop fs -ls s3n://my-s3-bucket` from the command line without issue. I have set some options in hadoop-env.sh to make sure all the S3 stuff for hadoop 2.6 is set up correctly. (This was very confusing, BTW and not enough searchable documentation on changes to the s3 stuff in hadoop 2.6 IMHO). Anyways, when I run a pig job which accesses s3, it gets to 16%, does not fail in pig, but rather fails in mapreduce with "Error: java.io.IOException: No FileSystem for scheme: s3n.=E2=80=9D I have added [hadoop-install-loc]/lib and [hadoop-install-loc]/share/hadoop/tools/lib/ to the HADOOP_CLASSPATH env variable in hadoop-env.sh.erb. When I do not do this, the pig job will fail at 0% (before it ever gets to mapreduce) with a very similar =E2=80=9CNo fi= leystem for scheme s3n=E2=80=9D error. I feel like at this point I just have to add the share/hadoop/tools/lib directory (and maybe lib) to the right environment variable, but I can=E2= =80=99t figure out which environment variable that should be. I appreciate any help, thanks!! Stack trace: org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2584) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2591) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2630) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2612) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370) at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296) at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.setInputPaths(FileInp= utFormat.java:498) at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.setInputPaths(FileInp= utFormat.java:467) at org.apache.pig.piggybank.storage.CSVExcelStorage.setLocation(CSVExcelStorag= e.java:609) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat= .mergeSplitSpecificConf(PigInputFormat.java:129) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat= .createRecordReader(PigInputFormat.java:103) at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.(MapTask.jav= a:512) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:755) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.j= ava:1628) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) =E2=80=94 Billy Watson --=20 William Watson Software Engineer (904) 705-7056 PCS --001a11c330d029a282051427bf06 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Hi,

I am able to run a `hadoop fs -ls s3n://my-s3-bucket` from the c= ommand line without issue. I have set some options in hadoop-env.sh to make= sure all the S3 stuff for hadoop 2.6 is set up correctly. (This was very c= onfusing, BTW and not enough searchable documentation on changes to the s3 = stuff in hadoop 2.6 IMHO).

Anyways, when I run a pig job which acces= ses s3, it gets to 16%, does not fail in pig, but rather fails in mapreduce= with "Error: java.io.IOException: No FileSystem for scheme: s3n.=E2= =80=9D

I have added [hadoop-install-loc]/lib and [hadoop-install-lo= c]/share/hadoop/tools/lib/ to the HADOOP_CLASSPATH env variable in hadoop-e= nv.sh.erb. When I do not do this, the pig job will fail at 0% (before it ev= er gets to mapreduce) with a very similar =E2=80=9CNo fileystem for scheme = s3n=E2=80=9D error.

I feel like at this point I just have to add the= share/hadoop/tools/lib directory (and maybe lib) to the right environment = variable, but I can=E2=80=99t figure out which environment variable that sh= ould be.

I appreciate any help, thanks!!


Stack trace:org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2584) a= t org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2591) at= org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91) at org.apac= he.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2630) at org.apac= he.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2612) at org.apache.hadoo= p.fs.FileSystem.get(FileSystem.java:370) at org.apache.hadoop.fs.Path.getFi= leSystem(Path.java:296) at org.apache.hadoop.mapreduce.lib.input.FileInputF= ormat.setInputPaths(FileInputFormat.java:498) at org.apache.hadoop.mapreduc= e.lib.input.FileInputFormat.setInputPaths(FileInputFormat.java:467) at org.= apache.pig.piggybank.storage.CSVExcelStorage.setLocation(CSVExcelStorage.ja= va:609) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Pig= InputFormat.mergeSplitSpecificConf(PigInputFormat.java:129) at org.apache.p= ig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.createRecor= dReader(PigInputFormat.java:103) at org.apache.hadoop.mapred.MapTask$NewTra= ckingRecordReader.<init>(MapTask.java:512) at org.apache.hadoop.mapre= d.MapTask.runNewMapper(MapTask.java:755) at org.apache.hadoop.mapred.MapTas= k.run(MapTask.java:341) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChi= ld.java:163) at java.security.AccessController.doPrivileged(Native Method) = at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.= security.UserGroupInformation.doAs(UserGroupInformation.java:1628) at org.a= pache.hadoop.mapred.YarnChild.main(YarnChild.java:158)


=E2=80=94= Billy Watson

--
William = Watson
Software Engineer
(904) 705-7056 PCS
<= br> --001a11c330d029a282051427bf06--