hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Carmen Manzulli <carmenmanzu...@gmail.com>
Subject Re: How to implement a servlet to submit job to Hadoop 2.6.0 cluster
Date Thu, 28 May 2015 08:58:19 GMT
Thanks you were right!!!! but loading *-site.xml files in hdfs for example
in a folder /configurationFiles/ created by: hdfs dfs -mkdir
/configurationFiles and writing

config.addResource(new
Path("hdfs://master:9000/configurationFiles/core-site.xml"));

i get the same error... this is the complete error:

java.io.IOException: Cannot initialize Cluster. Please check your
configuration for mapreduce.framework.name and the correspond server
addresses.
	org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:120)
	org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:82)
	org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:75)
	org.apache.hadoop.mapreduce.Job$9.run(Job.java:1266)
	org.apache.hadoop.mapreduce.Job$9.run(Job.java:1262)
	java.security.AccessController.doPrivileged(Native Method)
	javax.security.auth.Subject.doAs(Subject.java:415)
	org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
	org.apache.hadoop.mapreduce.Job.connect(Job.java:1261)
	org.apache.hadoop.mapreduce.Job.submit(Job.java:1290)
	org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1314)
	serv.QueryServlet.processRequest(QueryServlet.java:97)
<---In line 97 I've job.waitForCompletion(true);
	serv.QueryServlet.doPost(QueryServlet.java:133)
	javax.servlet.http.HttpServlet.service(HttpServlet.java:648)
	javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
	org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
	org.netbeans.modules.web.monitor.server.MonitorFilter.doFilter(MonitorFilter.java:393)


Any Idea?


2015-05-28 3:10 GMT+02:00 杨浩 <yanghaogn@gmail.com>:

> config.addResource(new Path("/usr/local/hadoop-2.6.0/etc/hadoop/core-site.xml"));
>>
>> the path is org.apache.hadoop.fs.Path, so the resource should be in hdfs,
> do you have the resource in hdfs?
>
> can you try the API
>
>> config.addResource(InputStream in)
>
>
> 2015-05-25 18:36 GMT+08:00 Carmen Manzulli <carmenmanzulli@gmail.com>:
>
>>
>> Hi,
>>
>> I'm trying to run a servlet (QueryServlet) - using Tomcat 8- which
>> submits a job called "ArqOnHadoop2" to hadoop 2.6.0... this last one is
>> configured using single node setting up in /usr/local folder. This job
>> works if i start it from command line but, when i try to execute the follow
>> code from netbeans,I receive "HTTP Status 500 - Cannot initialize Cluster.
>> Please check your configuration for mapreduce.framework.name and the
>> correspond server addresses."
>>
>> Configuration config = new Configuration();
>> config.addResource(new Path("/usr/local/hadoop-2.6.0/etc/hadoop/core-site.xml"));
>> config.addResource(new Path("/usr/local/hadoop-2.6.0/etc/hadoop/hdfs-site.xml"));
>> config.addResource(new Path("/usr/local/hadoop-2.6.0/etc/hadoop/yarn-site.xml"));
>> config.addResource(new Path("/usr/local/hadoop-2.6.0/etc/hadoop/mapred-site.xml"));
>>
>> config.set("fs.hdfs.impl", org.apache.hadoop.hdfs.DistributedFileSystem.class.getName());
>> config.set("yarn.resourcemanager.address","master:8032");
>> config.set("mapreduce.framework.name","yarn");
>> config.set("fs.defaultFS","hdfs://master:9000");
>> //input query parameter as string
>> config.set(ArqReducerConstants.MY_QUERY,args[0]);
>> Job job = Job.getInstance(config);
>> job.setJarByClass(QueryServlet.class);//input number of lines for mapper parameter
as int, used to reduce number of splits and run map() method quicklyString N=args[4];int n=Integer.parseInt(N);
>> job.getConfiguration().setInt(NLineInputFormat.LINES_PER_MAP, n);
>>
>> job.setMapperClass(MapperDoesNothing.class);
>> job.setMapOutputKeyClass(NullWritable.class);
>> job.setMapOutputValueClass(TripleWritable.class);
>>
>> job.setReducerClass(QueryReducer.class);
>>
>> job.setInputFormatClass( BlockedNTriplesInputFormat.class);
>> job.setOutputFormatClass(TextOutputFormat.class);String in="hdfs://master:9000"+args[1];String
out="hdfs://master:9000"+args[2];//input and output paths parametersFileInputFormat.setInputPaths(job,
new Path(in));FileOutputFormat.setOutputPath(job, new Path(out));
>>
>> job.waitForCompletion(true);
>>
>> //where args is just a name of a String array with some input
>> parameters..
>>
>> There are no problems with hadoop or in the .xml files; there are no
>> problems with permissions, because i've disable its with:
>> dfs.permission.enables=false in hdfs-site.xml; there are no problem with
>> permissions also for my hadoop folder because i've used chmod -R 777;
>>
>> So...what does my project miss to obtain the goal? I need help...
>>
>> i think there is something missing when i set the Configuration Object,
>> or problem could be due to something about jar in the classpath but in this
>> case i don't know how to insert all hadoop jar using maven...
>>
>> Thanks in advance, Carmen.
>>
>>
>

Mime
View raw message