hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ranjini Rathinam <ranjinibe...@gmail.com>
Subject Re: Hadoop-MapReduce
Date Tue, 17 Dec 2013 09:24:35 GMT
Hi,

I want to know , when should i use Mapper , Reduce and Combiner.

What all methods are there in them.

Please suggest for study in detail. As I am fresher .

Thanks in advance

Ranjini

On Tue, Dec 17, 2013 at 2:34 PM, unmesha sreeveni <unmeshabiju@gmail.com>wrote:

>  Ranjini can u pls check this. This is not perfect ..I simply did to
> check my xml data.
>
> https://github.com/studhadoop/xmlparsing-hadoop/blob/master/XmlParser11.java
>
>
> On Tue, Dec 17, 2013 at 2:26 PM, Ranjini Rathinam <ranjinibecse@gmail.com>wrote:
>
>> Hi,
>>
>> The driver class and my Mapper class i have used
>> org.apache.hadoop.mapreduce.lib
>>
>> and in the XmlInputFormat.java class also i have used the
>> org.apache.hadoop.mapreduce.lib
>>
>> but still iam getting this error.
>>
>> Please suggest.
>>
>> Thanks in advance
>>
>> Ranjini
>>
>>  On Tue, Dec 17, 2013 at 2:07 PM, Shekhar Sharma <shekhar2581@gmail.com>wrote:
>>
>>> Hello Ranjini,
>>> This error will come when you use mix and match newer and older API..
>>>
>>> You might have written program using newer API and the the XML input
>>> format is using older api..
>>> The older api has package structure of org.apache.hadoop.mapred
>>>
>>> The newer api has package structure package of
>>> org.apache.hadoop.mapreduce.lib
>>>
>>> Check out the XMLINputFormat.java, which package of FileInputFormat
>>> they have used...
>>>
>>>
>>> Regards,
>>> Som Shekhar Sharma
>>> +91-8197243810
>>>
>>>
>>> On Tue, Dec 17, 2013 at 12:55 PM, Ranjini Rathinam
>>>  <ranjinibecse@gmail.com> wrote:
>>> > Hi,
>>> >
>>> > I am using hadoop 0.20 version
>>> >
>>> > In that while exceuting the XmlInformat class
>>> > I am getting the error as
>>> >
>>> > "Error: Found Class  org.apache.hadoop.mapreduce.TaskAttemptContext,
>>> but
>>> > interface was excepted,."
>>> >
>>> > Please suggest to fix the error.
>>> >
>>> > Thanks in advance.
>>> >
>>> > Ranjini
>>> >
>>> > On Wed, Dec 11, 2013 at 12:30 PM, Ranjini Rathinam <
>>> ranjinibecse@gmail.com>
>>> > wrote:
>>> >>
>>> >> hi,
>>> >>
>>> >> I have fixed the error , the code is running fine, but this code just
>>> >> split the part of the tag.
>>> >>
>>> >> i want to convert into text format so that i can load them into
>>> tables of
>>> >> hbase and hive.
>>> >>
>>> >> I have used the DOM Parser but this parser uses File as Object  but
>>> hdfs
>>> >> uses FileSystem.
>>> >>
>>> >> Eg,
>>> >>
>>> >> File fXmlFile = new File("D:/elango/test.xml");
>>> >>
>>> >>  System.out.println(g);
>>> >>  DocumentBuilderFactory dbFactory =
>>> DocumentBuilderFactory.newInstance();
>>> >>  DocumentBuilder dBuilder = dbFactory.newDocumentBuilder();
>>> >>  Document doc = dBuilder.parse(fXmlFile);
>>> >>
>>> >>
>>> >> This cant be used as hdfs, because hdfs path  is accessed through
>>> >> FileSystem.
>>> >>
>>> >> I kindly request u to , Please suggest me to fix the above issue.
>>> >>
>>> >> Thanks in advance
>>> >>
>>> >> Ranjini R
>>> >>
>>> >>
>>> >>
>>> >>
>>> >> On Tue, Dec 10, 2013 at 11:07 AM, Ranjini Rathinam
>>> >> <ranjinibecse@gmail.com> wrote:
>>> >>>
>>> >>>
>>> >>>
>>> >>> ---------- Forwarded message ----------
>>> >>> From: Shekhar Sharma <shekhar2581@gmail.com>
>>> >>> Date: Mon, Dec 9, 2013 at 10:23 PM
>>> >>> Subject: Re: Hadoop-MapReduce
>>> >>> To: user@hadoop.apache.org
>>> >>> Cc: ssanyal@datameer.com
>>> >>>
>>> >>>
>>> >>> It does work i have used it long back..
>>> >>>
>>> >>> BTW if it is not working, write the custom input format and implement
>>> >>> your record reader. That would be far more easy than breaking your
>>> >>> head with others code.
>>> >>>
>>> >>> Break your problem in step:
>>> >>>
>>> >>> (1) First the XML data is multiline...Meaning multiple lines makes
a
>>> >>> single record for you...May be a record for you would be
>>> >>>
>>> >>> <person>
>>> >>>  <fname>x</fname>
>>> >>>   <lname>y</lname>
>>> >>> </person>
>>> >>>
>>> >>> (2) Implement a record reader that looks out for the starting and
>>> >>> ending person tag ( Checkout how RecordReader.java is written)
>>> >>>
>>> >>> (3) Once you got the contents between starting and ending tag, now
>>> you
>>> >>> can use a xml parser to parse the contents into an java object and
>>> >>> form your own key value pairs ( custom key and custom value)
>>> >>>
>>> >>>
>>> >>> Hope you have enough pointers to write the code.
>>> >>>
>>> >>>
>>> >>> Regards,
>>> >>> Som Shekhar Sharma
>>> >>> +91-8197243810
>>> >>>
>>> >>>
>>> >>> On Mon, Dec 9, 2013 at 6:30 PM, Ranjini Rathinam <
>>> ranjinibecse@gmail.com>
>>> >>> wrote:
>>> >>> > Hi Subroto Sanyal,
>>> >>> >
>>> >>> > The link  provided about xml, it does not work . The Class
written
>>> >>> > XmlContent is not allowed in the XmlInputFormat.
>>> >>> >
>>> >>> > I request you to help , whether this scenaio some one has coded,
>>> and
>>> >>> > needed
>>> >>> > working code.
>>> >>> >
>>> >>> > I have written using SAX Parser too, but eventhough the jars
are
>>> added
>>> >>> > in
>>> >>> > classpath THe error is is coming has NoClasFoung Exception.
>>> >>> >
>>> >>> > Please provide sample code for the same.
>>> >>> >
>>> >>> > Thanks in advance,
>>> >>> > Ranjini.R
>>> >>> >
>>> >>> > On Mon, Dec 9, 2013 at 12:34 PM, Ranjini Rathinam
>>> >>> > <ranjinibecse@gmail.com>
>>> >>> > wrote:
>>> >>> >>
>>> >>> >>
>>> >>> >>>> Hi,
>>> >>> >>>>
>>> >>> >>>> As suggest by the link below , i have used for
my program ,
>>> >>> >>>>
>>> >>> >>>> but i am facing the below issues, please help me
to fix these
>>> error.
>>> >>> >>>>
>>> >>> >>>>
>>> >>> >>>> XmlReader.java:8: XmlReader.Map is not abstract
and does not
>>> >>> >>>> override
>>> >>> >>>> abstract method
>>> >>> >>>>
>>> >>> >>>>
>>> map(org.apache.hadoop.io.LongWritable,org.apache.hadoop.io.Text,org.apache.hadoop.mapred.OutputCollector<org.apache.hadoop.io.Text,org.apache.hadoop.io.Text>,org.apache.hadoop.mapred.Reporter)
>>> >>> >>>> in org.apache.hadoop.mapred.Mapper
>>> >>> >>>>  public static class Map extends MapReduceBase
implements Mapper
>>> >>> >>>> <LongWritable, Text, Text, Text> {
>>> >>> >>>>                ^
>>> >>> >>>> ./XmlInputFormat.java:16: XmlInputFormat.XmlRecordReader
is not
>>> >>> >>>> abstract
>>> >>> >>>> and does not override abstract method
>>> >>> >>>> next(java.lang.Object,java.lang.Object) in
>>> >>> >>>> org.apache.hadoop.mapred.RecordReader
>>> >>> >>>> public class XmlRecordReader implements RecordReader
{
>>> >>> >>>>        ^
>>> >>> >>>> Note: XmlReader.java uses unchecked or unsafe operations.
>>> >>> >>>> Note: Recompile with -Xlint:unchecked for details.
>>> >>> >>>> 2 errors
>>> >>> >>>>
>>> >>> >>>>
>>> >>> >>>> i am using hadoop 0.20 version and java 1.6 .
>>> >>> >>>>
>>> >>> >>>> Please suggest.
>>> >>> >>>>
>>> >>> >>>> Thanks in advance.
>>> >>> >>>>
>>> >>> >>>> Regrads,
>>> >>> >>>> Ranjini. R
>>> >>> >>>> On Mon, Dec 9, 2013 at 11:08 AM, Ranjini Rathinam
>>> >>> >>>> <ranjinibecse@gmail.com> wrote:
>>> >>> >>>>>
>>> >>> >>>>>
>>> >>> >>>>>
>>> >>> >>>>> ---------- Forwarded message ----------
>>> >>> >>>>> From: Subroto <ssanyal@datameer.com>
>>> >>> >>>>> Date: Fri, Dec 6, 2013 at 4:42 PM
>>> >>> >>>>> Subject: Re: Hadoop-MapReduce
>>> >>> >>>>> To: user@hadoop.apache.org
>>> >>> >>>>>
>>> >>> >>>>>
>>> >>> >>>>> Hi Ranjini,
>>> >>> >>>>>
>>> >>> >>>>> A good example to look into :
>>> >>> >>>>> http://www.undercloud.org/?p=408
>>> >>> >>>>>
>>> >>> >>>>> Cheers,
>>> >>> >>>>> Subroto Sanyal
>>> >>> >>>>>
>>> >>> >>>>> On Dec 6, 2013, at 12:02 PM, Ranjini Rathinam
wrote:
>>> >>> >>>>>
>>> >>> >>>>> Hi,
>>> >>> >>>>>
>>> >>> >>>>> How to read xml file via mapreduce and load
them in hbase and
>>> hive
>>> >>> >>>>> using java.
>>> >>> >>>>>
>>> >>> >>>>> Please provide sample code.
>>> >>> >>>>>
>>> >>> >>>>> I am using hadoop 0.20 version and java 1.6.
Which parser
>>> version
>>> >>> >>>>> should be used.
>>> >>> >>>>>
>>> >>> >>>>> Thanks in advance.
>>> >>> >>>>>
>>> >>> >>>>> Ranjini
>>> >>> >>>>>
>>> >>> >>>>>
>>> >>> >>>>>
>>> >>> >>>>
>>> >>> >>>
>>> >>> >>
>>> >>> >
>>> >>>
>>> >>
>>> >
>>>
>>
>>
>
>
> --
> *Thanks & Regards*
>
>  Unmesha Sreeveni U.B
>
> *Junior Developer*
>
>
>

Mime
View raw message