hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Fengyun RAO <raofeng...@gmail.com>
Subject Re: any suggestions on IIS log storage and analysis?
Date Mon, 30 Dec 2013 09:44:40 GMT
what do you mean by join the data sets?

a fake sample log file:
#Software: Microsoft Internet Information Services 7.5
#Version: 1.0
#Date: 2013-07-04 20:00:00
#Fields: date time s-ip cs-method cs-uri-stem cs-uri-query s-port
cs-username c-ip cs(User-Agent) sc-status sc-substatus sc-win32-status
time-taken
2013-07-04 20:00:00 1.2.2.9 GET /gs.gif xxx 80 - 98.248.105.227
Mozilla/5.0+(Windows+NT+6.1;+WOW64)+AppleWebKit/537.1+(KHTML,+like+Gecko)+Chrome/21.0.1180.89+Safari/537.1
200 0 0 390
2013-07-04 20:00:01 1.2.2.9 GET /gs.gif xxx 80 - 98.248.105.227
Mozilla/5.0+(Windows+NT+6.1;+WOW64)+AppleWebKit/537.1+(KHTML,+like+Gecko)+Chrome/21.0.1180.89+Safari/537.1
200 0 0 390
2013-07-04 20:00:02 1.2.2.9 GET /gs.gif xxx 80 - 98.248.105.227
Mozilla/5.0+(Windows+NT+6.1;+WOW64)+AppleWebKit/537.1+(KHTML,+like+Gecko)+Chrome/21.0.1180.89+Safari/537.1
200 0 0 390
#Software: Microsoft Internet Information Services 7.5
#Version: 1.0
#Date: 2013-07-04 20:00:03
#Fields: date time s-ip cs-method cs-uri-stem cs-uri-query s-port
cs-username c-ip cs(User-Agent) sc-status sc-substatus sc-win32-status
time-taken
2013-07-04 20:00:03 1.2.2.9 GET /gs.gif xxx 80 - 98.248.105.227
Mozilla/5.0+(Windows+NT+6.1;+WOW64)+AppleWebKit/537.1+(KHTML,+like+Gecko)+Chrome/21.0.1180.89+Safari/537.1
200 0 0 390


"#Fileds:" line is needed to parse the following IIS log, however, it may
change, which makes splitting not supported.


2013/12/30 Azuryy Yu <azuryyyu@gmail.com>

> You can run a mapreduce firstly, Join these data sets into one data set.
> then analyze the joined dataset.
>
>
> On Mon, Dec 30, 2013 at 3:58 PM, Fengyun RAO <raofengyun@gmail.com> wrote:
>
>> Hi,
>>
>> HDFS splits files into blocks, and mapreduce runs a map task for each
>> block. However, Fields could be changed in IIS log files, which means
>> fields in one block may depend on another, and thus make it not suitable
>> for mapreduce job. It seems there should be some preprocess before storing
>> and analyzing the IIS log files. We plan to parse each line to the same
>> fields and store in Avro files with compression. Any other alternatives?
>> Hbase?  or any suggestions on analyzing IIS log files?
>>
>> thanks!
>>
>>
>>
>

Mime
View raw message