crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Micah Whitacre <mkw...@gmail.com>
Subject Re: hbase error I don't understand
Date Mon, 31 Mar 2014 15:44:44 GMT
We've had issues with IPv6 (though I haven't had the problem with Crunch
specifically).  When we last had the issue it was actually ZK that had
trouble connecting to IPv6.

There are two fixes we've found:

The first is to set surefire to use IPv4

           <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <configuration>
                    <argLine>-Xmx2048m -Djava.net.preferIPv4Stack=true</argLine>
                </configuration>
            </plugin>

The second is to change the devices /etc/hosts to be more explicit on
the IPv4 entries so it chose that over the the IPv6.

Obviously the first would probably be preferred for repeatable builds.



On Mon, Mar 31, 2014 at 10:24 AM, Josh Wills <josh.wills@gmail.com> wrote:

> Got it, thanks Gabriel. Will try it again when I'm @ my hotel.
>
> J
>
>
> On Mon, Mar 31, 2014 at 8:23 AM, Gabriel Reid <gabriel.reid@gmail.com
> >wrote:
>
> > Hi Josh,
> >
> > I've run into things like this before as well when running HBase
> > integration tests. As far as I can understand it, it has something to
> > do with IPv6 addresses being returned when doing a reverse DNS lookup.
> > I think that I've only gotten this intermittently when running things
> > from home, which has made me assume that my ISP was doing something
> > with their DNS setup at the time.
> >
> > Unfortunately, I don't have any suggestions as to a resolution. This
> > problem has always disappeared on its own before I've figured out how
> > to resolve it properly.
> >
> > - Gabriel
> >
> > On Mon, Mar 31, 2014 at 5:13 PM, Josh Wills <jwills@cloudera.com> wrote:
> > > Hey devs,
> > >
> > > I'm back from a long break and I'm getting an odd exception on my hbase
> > > integration tests running on my mac--
> > >
> > > INFO
> >  org.apache.crunch.hadoop.mapreduce.lib.jobcontrol.CrunchControlledJob
> > >  - java.lang.NumberFormatException: For input string:
> > ":22e5:2aff:fe15:2686"
> > >  at
> > >
> >
> java.lang.NumberFormatException.forInputString(NumberFormatException.java:48)
> > > at java.lang.Integer.parseInt(Integer.java:449)
> > >  at java.lang.Integer.parseInt(Integer.java:499)
> > > at com.sun.jndi.dns.DnsClient.<init>(DnsClient.java:103)
> > >  at com.sun.jndi.dns.Resolver.<init>(Resolver.java:44)
> > > at com.sun.jndi.dns.DnsContext.getResolver(DnsContext.java:553)
> > >  at com.sun.jndi.dns.DnsContext.c_getAttributes(DnsContext.java:413)
> > > at
> > >
> >
> com.sun.jndi.toolkit.ctx.ComponentDirContext.p_getAttributes(ComponentDirContext.java:213)
> > >  at
> > >
> >
> com.sun.jndi.toolkit.ctx.PartialCompositeDirContext.getAttributes(PartialCompositeDirContext.java:121)
> > > at
> > >
> >
> com.sun.jndi.toolkit.url.GenericURLDirContext.getAttributes(GenericURLDirContext.java:85)
> > >  at
> > >
> >
> javax.naming.directory.InitialDirContext.getAttributes(InitialDirContext.java:123)
> > > at org.apache.hadoop.net.DNS.reverseDns(DNS.java:84)
> > >  at
> > >
> >
> org.apache.hadoop.hbase.mapreduce.TableInputFormatBase.reverseDNS(TableInputFormatBase.java:223)
> > > at
> > >
> >
> org.apache.hadoop.hbase.mapreduce.TableInputFormatBase.getSplits(TableInputFormatBase.java:189)
> > >
> > > I get it w/both hadoop-1 and hadoop-2-- any clues as to what's going on
> > > here? To be fair, I'm running this from the Cloudera UK office for the
> > > first time evah, and I'm wondering if there's some thing I need to do
> > > differently here.
> > >
> > > Thanks,
> > > J
> > >
> > > --
> > > Director of Data Science
> > > Cloudera <http://www.cloudera.com>
> > > Twitter: @josh_wills <http://twitter.com/josh_wills>
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message