hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: NoSuchElementException while executing MapReduce-job
Date Thu, 28 Aug 2014 15:57:59 GMT
Maybe mapred-site.xml was not in classpath ?
It should have something like this:

  <property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
  </property>

Cheers


On Thu, Aug 28, 2014 at 5:51 AM, Malte Maltesmann <
malte.maltesmann@gmail.com> wrote:

> I did not use maven earlier for this, but if I try it with maven and the
> following pom.xml:
>
> <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="
> http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="
> http://maven.apache.org/POM/4.0.0
> http://maven.apache.org/xsd/maven-4.0.0.xsd">
>   <modelVersion>4.0.0</modelVersion>
>   <groupId>de.test.malte</groupId>
>   <artifactId>MapRedMaven</artifactId>
>   <version>0.0.1-SNAPSHOT</version>
>   <dependencies>
>       <dependency>
>           <groupId>org.apache.hadoop</groupId>
>           <artifactId>hadoop-common</artifactId>
>           <version>2.4.1</version>
>       </dependency>
>       <dependency>
>           <groupId>org.apache.hadoop</groupId>
>           <artifactId>hadoop-hdfs</artifactId>
>           <version>2.4.1</version>
>       </dependency>
>       <dependency>
>           <groupId>org.apache.hadoop</groupId>
>           <artifactId>hadoop-mapreduce-client-core</artifactId>
>           <version>2.4.1</version>
>       </dependency>
>   </dependencies>
> </project>
>
> I get the following exception on both, Linux and FreeBSD:
>
> 14/08/28 14:45:56 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> Exception in thread "main" java.io.IOException: Cannot initialize Cluster.
> Please check your configuration for mapreduce.framework.name and the
> correspond server addresses.
>     at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:120)
>     at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:82)
>     at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:75)
>     at org.apache.hadoop.mapreduce.Job$9.run(Job.java:1255)
>     at org.apache.hadoop.mapreduce.Job$9.run(Job.java:1251)
>     at java.security.AccessController.doPrivileged(Native Method)
>     at javax.security.auth.Subject.doAs(Subject.java:415)
>     at
>
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1556)
>     at org.apache.hadoop.mapreduce.Job.connect(Job.java:1250)
>     at org.apache.hadoop.mapreduce.Job.submit(Job.java:1279)
>     at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1303)
>     at
> NewMaxTemperature$NewMaxTemperatureMapper.main(NewMaxTemperature.java:39)
>
> By the way, I am now trying to run this
> <
> http://gerrymcnicol.com/index.php/2014/01/02/hadoop-and-cassandra-part-4-writing-your-first-mapreduce-job/
> >
> example on only one node. It runs fine on Linux without using maven but
> crashes on Linux and FreeBSD with maven with the above Exception and
> without maven on FreeBSD with the NoSuchElementException.
>
> Regards
> Malte
>
>
> 2014-08-27 16:44 GMT+02:00 Artem Ervits <are9004@nyp.org>:
>
> > Check your pom.xml, some artifacts changed from earlier releases.
> >
> >
> > Artem Ervits
> > Data Analyst
> > New York Presbyterian Hospital
> >
> > ----- Original Message -----
> > From: Malte Otten [mailto:malte.maltesmann@gmail.com]
> > Sent: Monday, August 25, 2014 02:12 PM
> > To: user@hbase.apache.org <user@hbase.apache.org>
> > Subject: Re: NoSuchElementException while executing MapReduce-job
> >
> > Hi Ted,
> >
> > thanks for your quick answer.
> > I don't think I'm running in a secure setup, because I didn't do any of
> > the things specified in the reference guide chapter from your link and
> > as far as I can see, the secure setup is not the default. My
> > hbase-site.xml looks like the following:
> >
> > <configuration>
> > <property>
> >    <name>hbase.cluster.distributed</name>
> >    <value>true</value>
> > </property>
> > <property>
> > <name>hbase.master.wait.on.regionservers.mintostart</name>
> >    <value>1</value>
> > </property>
> >    <property>
> >      <name>hbase.rootdir</name>
> > <value>hdfs://my.hbase.server.de:8020/hbase</value>
> >    </property>
> >    <property>
> >      <name>hbase.zookeeper.property.dataDir</name>
> >      <value>/my/path/hbase-data/zookeeper</value>
> >    </property>
> > <property>
> >    <name>hbase.zookeeper.quorum</name>
> >    <value>my.hbase.server.de</value>
> > </property>
> > </configuration>
> >
> > Do you have any other suggestions?
> >
> > Regards
> > Malte
> >
> >
> > Am 25.08.2014 um 18:01 schrieb Ted Yu:
> > > Looks like you were running in a secure setup.
> > >
> > > If that's the case, you should read:
> > > http://hbase.apache.org/book.html#hbase.secure.configuration
> > >
> > > Cheers
> > >
> > >
> > > On Mon, Aug 25, 2014 at 5:48 AM, Malte Maltesmann <
> > > malte.maltesmann@gmail.com> wrote:
> > >
> > >> Hi all,
> > >>
> > >> when I'm trying to execute a MapReduce-Job over the HBase-Java-API I
> > get an
> > >> exception with the follwoing Stacktrace:
> > >>
> > >> java.util.NoSuchElementException
> > >>      at java.util.StringTokenizer.nextToken(StringTokenizer.java:349)
> > >>      at
> > >>
> > >>
> >
> org.apache.hadoop.fs.RawLocalFileSystem$DeprecatedRawLocalFileStatus.loadPermissionInfo(RawLocalFileSystem.java:565)
> > >>      at
> > >>
> > >>
> >
> org.apache.hadoop.fs.RawLocalFileSystem$DeprecatedRawLocalFileStatus.getPermission(RawLocalFileSystem.java:534)
> > >>      at
> > >>
> > >>
> >
> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.checkPermissionOfOther(ClientDistributedCacheManager.java:276)
> > >>      at
> > >>
> > >>
> >
> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.isPublic(ClientDistributedCacheManager.java:240)
> > >>      at
> > >>
> > >>
> >
> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineCacheVisibilities(ClientDistributedCacheManager.java:162)
> > >>      at
> > >>
> > >>
> >
> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineTimestampsAndCacheVisibilities(ClientDistributedCacheManager.java:58)
> > >>      at
> > >>
> > >>
> >
> org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:265)
> > >>      at
> > >>
> > >>
> >
> org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:301)
> > >>      at
> > >>
> > >>
> >
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:389)
> > >>      at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1285)
> > >>      at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1282)
> > >>      at java.security.AccessController.doPrivileged(Native Method)
> > >>      at javax.security.auth.Subject.doAs(Subject.java:415)
> > >>      at
> > >>
> > >>
> >
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1556)
> > >>      at org.apache.hadoop.mapreduce.Job.submit(Job.java:1282)
> > >>      at
> org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1303)
> > >> ...
> > >>
> > >> In the RawLocalFileSystem.java I can see, that its trying to get some
> > >> Permissions here, but the StringTokenizer gets an empty String, so the
> > >> first t.nextToken() fails with the Exception above:
> > >>
> > >> /// loads permissions, owner, and group from `ls -ld`
> > >>      private void loadPermissionInfo() {
> > >>        IOException e = null;
> > >>        try {
> > >>          String output = FileUtil.execCommand(new
> > File(getPath().toUri()),
> > >>              Shell.getGetPermissionCommand());
> > >>          StringTokenizer t =
> > >>              new StringTokenizer(output, Shell.TOKEN_SEPARATOR_REGEX);
> > >>          //expected format
> > >>          //-rw-------    1 username groupname ...
> > >>          String permission = t.nextToken();
> > >>          if (permission.length() >
> FsPermission.MAX_PERMISSION_LENGTH) {
> > >>            //files with ACLs might have a '+'
> > >>            permission = permission.substring(0,
> > >>              FsPermission.MAX_PERMISSION_LENGTH);
> > >>          }
> > >>          setPermission(FsPermission.valueOf(permission));
> > >>          t.nextToken();
> > >>
> > >>          String owner = t.nextToken();
> > >>          // If on windows domain, token format is DOMAIN\\user and we
> > want
> > >> to
> > >>          // extract only the user name
> > >>          if (Shell.WINDOWS) {
> > >>            int i = owner.indexOf('\\');
> > >>            if (i != -1)
> > >>              owner = owner.substring(i + 1);
> > >>          }
> > >>          setOwner(owner);
> > >>
> > >>          setGroup(t.nextToken());
> > >>        } catch (Shell.ExitCodeException ioe) {
> > >>          if (ioe.getExitCode() != 1) {
> > >>            e = ioe;
> > >>          } else {
> > >>            setPermission(null);
> > >>            setOwner(null);
> > >>            setGroup(null);
> > >>          }
> > >>        } catch (IOException ioe) {
> > >>          e = ioe;
> > >>        } finally {
> > >>          if (e != null) {
> > >>            throw new RuntimeException("Error while running command to
> > get "
> > >> +
> > >>                                       "file permissions : " +
> > >>
> >  StringUtils.stringifyException(e));
> > >>          }
> > >>        }
> > >>      }
> > >>
> > >> So my question is, what could have caused this Problem. My setup is
> the
> > >> following:
> > >>
> > >> Two nodes running on FreeBSD 9.0
> > >> Hadoop 2.4.1
> > >> HBase 0.98.4
> > >>
> > >> The shell-command "ls -ld" delivers the same output on FreeBSD as on
> any
> > >> other Linux-distribution, so this shouldn't be the problem.
> > >> Also I can use simple operations from the HBase-API like get or put
> > without
> > >> any problems.
> > >>
> > >> I also posted this on stackoverflow
> > >> <
> > >>
> >
> http://stackoverflow.com/questions/25364802/hbase-mapreduce-nosuchelementexception
> > >> a week ago, but didn't get any answers.
> > >>
> > >> Thanks in advance,
> > >> Malte
> > >>
> >
> >
> > ________________________________
> >
> > This electronic message is intended to be for the use only of the named
> > recipient, and may contain information that is confidential or
> privileged.
> > If you are not the intended recipient, you are hereby notified that any
> > disclosure, copying, distribution or use of the contents of this message
> is
> > strictly prohibited. If you have received this message in error or are
> not
> > the named recipient, please notify us immediately by contacting the
> sender
> > at the electronic mail address noted above, and delete and destroy all
> > copies of this message. Thank you.
> >
> > This electronic message is intended to be for the use only of the named
> > recipient, and may contain information that is confidential or
> privileged.
> > If you are not the intended recipient, you are hereby notified that any
> > disclosure, copying, distribution or use of the contents of this message
> is
> > strictly prohibited.  If you have received this message in error or are
> not
> > the named recipient, please notify us immediately by contacting the
> sender
> > at the electronic mail address noted above, and delete and destroy all
> > copies of this message.  Thank you.
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message