hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Peyman Mohajerian <mohaj...@gmail.com>
Subject Re: hadoop+python+text mining
Date Fri, 25 Apr 2014 00:59:31 GMT
At the high level I think you have these choices and more:
1) Hadoop Streaming, leverage some of your python could, but not all b/c
you have to deal with map/reduce.
2) Use Mahout.
3) Use a distro of R that works with Hadoop
..


On Thu, Apr 24, 2014 at 1:58 PM, qiaoresearcher <qiaoresearcher@gmail.com>wrote:

> I have Hadoop and python installed with nltk. Now I have an large input
> file which has three columns:
> column 1  | column 2 | column 3
> positive         id1          some tweet message
> negative       id2          other tweet message
> positive         id3          tweet message
> negative       id4          tweet message
> positive         id5          tweet message
> ....                    ...                ....
>
> I want to use text mining to construct TFIDF vectors from the tweet
> messages (also use stop words, stem, etc) and then use some classifier to
> classify tweet message as positive or negative. I know how to do it just
> using python and nltk. But how to do the same thing on hadoop?
>
> thanks!
>
>
>

Mime
View raw message