pig-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Apache Wiki <wikidi...@apache.org>
Subject [Pig Wiki] Trivial Update of "PigTutorial" by CorinneC
Date Mon, 09 Jun 2008 20:07:18 GMT
Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Pig Wiki" for change notification.

The following page has been changed by CorinneC:
http://wiki.apache.org/pig/PigTutorial

------------------------------------------------------------------------------
  
  The tutorial pig script (tutorial.pig or tutorial-local.pig) does the following:
  
- 
   * Register the tutorial JAR file so that the included UDFs can be called in the script.
  {{{ 
  REGISTER ./tutorial.jar; 
  }}}
  
- 
-  * Use the [http://wiki.apache.org/pig/PigBuiltins PigStorage] function to load the excite
log file (excite.log or excite-small.log) into the “raw” bag as an array of records with
the fields '''user''', '''time''', and '''query'''. Input: (user,time,query) 
+  * Use the [http://wiki.apache.org/pig/PigBuiltins PigStorage] function to load the excite
log file (excite.log or excite-small.log) into the “raw” bag as an array of records with
the fields '''user''', '''time''', and '''query'''. 
  {{{
  raw = LOAD 'excite.log' USING PigStorage('\t') AS (user, time, query);
  }}}
@@ -177, +175 @@

  ordered_uniq_frequency = ORDER filtered_uniq_frequency BY (hour, score);
  }}}
  
-  * Use the [http://wiki.apache.org/pig/PigBuiltins PigStorage] function to store the results.
The output file contains a list of n-grams with the following fields: '''hour''', '''ngram''',
'''score''', '''count''', '''mean''' Output: (hour, n-gram, score, count, average_counts_among_all_hours)
+  * Use the [http://wiki.apache.org/pig/PigBuiltins PigStorage] function to store the results.
The output file contains a list of n-grams with the following fields: '''hour''', '''ngram''',
'''score''', '''count''', '''mean'''.
  {{{ 
  STORE ordered_uniq_frequency INTO '/tmp/tutorial-results' USING PigStorage(); 
  }}}
@@ -187, +185 @@

  
  The tutorial-join pig script (tutorial-join.pig or tutorial-join-local.pig) does the following:
  
-  * Registers the tutorial JAR file so that the user-defined functions (UDFs) can be called
in the script. 
+  * Register the tutorial JAR file so that the user-defined functions (UDFs) can be called
in the script.
+ {{{
+ REGISTER ./tutorial.jar;
+ }}}
+  
-  * Uses the [http://wiki.apache.org/pig/PigBuiltins PigStorage] function to load the excite
log file (excite.log or excite-small.log) into the “raw” bag as an array of records with
the fields '''user''', '''time''', and '''query'''. 
+  * Use the [http://wiki.apache.org/pig/PigBuiltins PigStorage] function to load the excite
log file (excite.log or excite-small.log) into the “raw” bag as an array of records with
the fields '''user''', '''time''', and '''query'''.
+ {{{
+ raw = LOAD 'excite.log' USING PigStorage('\t') AS (user, time, query);
+ }}}
+ 
-  * Calls the NonURLDetector UDF to remove records if the query field is empty or a URL.

+  * Call the NonURLDetector UDF to remove records if the query field is empty or a URL.
+ {{{
+ clean1 = FILTER raw BY org.apache.pig.tutorial.NonURLDetector(query);
+ }}}
+ 
+  
-  * Calls the !ToLower UDF to change the query field to lowercase.
+  * Call the !ToLower UDF to change the query field to lowercase.
+ {{{
+ clean2 = FOREACH clean1 GENERATE user, time, org.apache.pig.tutorial.ToLower(query) as query;
+ }}}
+ 
+ 
-  * Calls the Non!PornDetector UDF to remove records if the query field contains porn terms.

+  * Call the Non!PornDetector UDF to remove records if the query field contains porn terms.
-  * Calls the !ExtractHour UDF to extract the hour from the time field.
+ {{{
+ clean3 = FILTER clean2 BY org.apache.pig.tutorial.NonPornDetector(query);
+ }}}
+ 
+  
+  * Because the log file only contains queries for a single day, we are only interested in
the hour. The excite query log timestamp format is YYMMDDHHMMSS. Call the !ExtractHour UDF
to extract the hour from the time field.
+ {{{
+ houred = FOREACH clean3 GENERATE user, org.apache.pig.tutorial.ExtractHour(time) as hour,
query;
+ }}}
+ 
+ 
-  * Calls the N!GramGenerator UDF to compose the n-grams of the query. 
+  * Call the N!GramGenerator UDF to compose the n-grams of the query.
+ {{{
+ ngramed1 = FOREACH houred GENERATE user, hour, flatten(org.apache.pig.tutorial.NGramGenerator(query))
as ngram;
+ }}}
+ 
+  
-  * Uses the [http://wiki.apache.org/pig/PigLatin#DISTINCT:_Eliminating_duplicates_in_data
DISTINCT] command to get the unique n-grams for all records. 
+  * Use the [http://wiki.apache.org/pig/PigLatin#DISTINCT:_Eliminating_duplicates_in_data
DISTINCT] command to get the unique n-grams for all records. 
+ {{{
+ ngramed2 = DISTINCT ngramed1;
+ }}}
+ 
+ 
-  * Uses the [http://wiki.apache.org/pig/PigLatin#COGROUP:_Getting_the_relevant_data_together
GROUP] command to group the records by n-gram and hour. 
+  * Use the [http://wiki.apache.org/pig/PigLatin#COGROUP:_Getting_the_relevant_data_together
GROUP] command to group the records by n-gram and hour. 
+ {{{
+ hour_frequency1 = GROUP ngramed2 BY (ngram, hour);
+ }}}
+ 
+ 
-  * Uses the [http://wiki.apache.org/pig/PigBuiltins COUNT] function to get the count (occurrences)
of each n-gram. 
+  * Use the [http://wiki.apache.org/pig/PigBuiltins COUNT] function to get the count (occurrences)
of each n-gram. 
+ {{{
+ hour_frequency2 = FOREACH hour_frequency1 GENERATE flatten($0), COUNT($1) as count;
+ }}}
+ 
+ 
-  * Uses the [http://wiki.apache.org/pig/PigLatin#FOREACH_..._GENERATE:_Applying_transformations_to_the_data
FOREACH-GENERATE] command to assign names to the fields.
+  * Use the [http://wiki.apache.org/pig/PigLatin#FOREACH_..._GENERATE:_Applying_transformations_to_the_data
FOREACH-GENERATE] command to assign names to the fields.
+ {{{
+ hour_frequency3 = FOREACH hour_frequency2 GENERATE $0 as ngram, $1 as hour, $2 as count;
+ }}}
+ 
+ 
-  * Uses the [http://wiki.apache.org/pig/PigLatin#FILTER:_Getting_rid_of_data_you_are_not_interested_in_
FILTER] command to get the n-grams for hour ‘00’ 
+  * Use the [http://wiki.apache.org/pig/PigLatin#FILTER:_Getting_rid_of_data_you_are_not_interested_in_
FILTER] command to get the n-grams for hour ‘00’ 
+ {{{
+ hour00 = FILTER hour_frequency2 BY hour eq '00';
+ }}}
+ 
+ 
-  * Uses the [http://wiki.apache.org/pig/PigLatin#FILTER:_Getting_rid_of_data_you_are_not_interested_in_
FILTER] command to get the n-grams for hour ‘12’ 
+  * Uses the [http://wiki.apache.org/pig/PigLatin#FILTER:_Getting_rid_of_data_you_are_not_interested_in_
FILTER] command to get the n-grams for hour ‘12’
+ {{{
+ hour12 = FILTER hour_frequency3 BY hour eq '12';
+ }}}
+ 
+  
-  * Uses the [http://wiki.apache.org/pig/PigLatin#Joining JOIN] command to get the n-grams
that appear in both hours.
+  * Use the [http://wiki.apache.org/pig/PigLatin#Joining JOIN] command to get the n-grams
that appear in both hours.
+ {{{
+ same = JOIN hour00 BY $0, hour12 BY $0;
+ }}}
+ 
-  * Uses the [http://wiki.apache.org/pig/PigLatin#FOREACH_..._GENERATE:_Applying_transformations_to_the_data
FOREACH-GENERATE] command to record their frequency.
+  * Use the [http://wiki.apache.org/pig/PigLatin#FOREACH_..._GENERATE:_Applying_transformations_to_the_data
FOREACH-GENERATE] command to record their frequency.
+ {{{
+ same1 = FOREACH same GENERATE hour_frequency2::hour00::group::ngram as ngram, $2 as count00,
$5 as count12;
+ }}}
  
-  * Uses the [http://wiki.apache.org/pig/PigBuiltins PigStorage] function to store the results.
The output file contains a list of n-grams with the following fields: '''hour''', '''count00''',
'''count12'''
+  * Use the [http://wiki.apache.org/pig/PigBuiltins PigStorage] function to store the results.
The output file contains a list of n-grams with the following fields: '''hour''', '''count00''',
'''count12'''.
+ {{{
+ STORE same1 INTO '/tmp/tutorial-join-results' USING PigStorage();
+ }}}
  

Mime
View raw message