I have a corpus of 300,000 raw HTML files that I want to read in and
parse using Hadoop. What is the best input file format to use in this
case? I want to have access to each page's raw HTML in the mapper, so
I can parse from there.
I was thinking of preprocessing all the files, removing the new
lines, and putting them in a big <key, value> file:
url1, html with stripped new lines
url2, ....
url3, ....
...
urlN, ....
I'd rather not do all this preprocessing, just to wrangle the text
into Hadoop. Any other suggestions? What if I just stored the path to
the HTML file in a <key, value> type
url1, path_to_file1
url2, path_to_file2
...
urlN, path_to_fileN
Then in the mapper, I could read each file in from the DFS on the
fly. Anyone have any other good ideas? I feel like there's some key
function that I'm just stupidly overlooking...
Thanks!
David Balatero
|