Mailing-List: contact pig-commits-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: pig-dev@hadoop.apache.org
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: quoted-printable
From: Apache Wiki <wikidiffs@apache.org>
To: Apache Wiki <wikidiffs@apache.org>
Date: Wed, 10 Feb 2010 21:57:00 -0000
Message-ID: <20100210215700.28242.91206@eos.apache.org>
Subject: 
 =?utf-8?q?=5BPig_Wiki=5D_Update_of_=22LoadStoreMigrationGuide=22_by_Prade?=
 =?utf-8?q?epKamath?=

Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Pig Wiki" for chang=
e notification.

The "LoadStoreMigrationGuide" page has been changed by PradeepKamath.
http://wiki.apache.org/pig/LoadStoreMigrationGuide?action=3Ddiff&rev1=3D10&=
rev2=3D11

--------------------------------------------------

  =

  The main change is that the new !LoadFunc API is based on a !InputFormat =
to read the data. Implementations can choose to use existing !InputFormats =
like !TextInputFormat or implement a new one.
   =

- =3D=3D Table mapping old API calls to new API calls in rough order of cal=
l sequence=3D=3D
+ =3D=3D Table mapping old API calls to new API calls in rough order of cal=
l sequence =3D=3D
  || '''Old Method in !LoadFunc''' || '''Equivalent New Method''' || '''New=
 Class/Interface in which method is present''' || '''Explanation''' ||
  || No equivalent method || setUDFContextSignature() || !LoadFunc || This =
method will be called by Pig both in the front end and back end to pass a u=
nique signature to the Loader. The signature can be used to store into the =
UDFContext} any information which the Loader needs to store between various=
 method invocations in the front end and back end. A use case is to store !=
RequiredFieldList passed to it in !LoadPushDown.pushProjection(!RequiredFie=
ldList) for use in the back end before returning tuples in getNext()||
  || No equivalent method || relativeToAbsolutePath() || !LoadFunc || Pig r=
untime will call this method to allow the Loader to convert a relative load=
 location to an absolute location. The default implementation provided in !=
LoadFunc handles this for hdfs files and directories. If the load source is=
 something else, loader implementation may choose to override this.||
@@ -22, +22 @@

  || getNext() || getNext() || !LoadFunc || The meaning of getNext() has no=
t changed and is called by Pig runtime to get the next tuple in the data ||
  || bytesToInteger(),...bytesToBag() ||  bytesToInteger(),...bytesToBag() =
|| !LoadCaster || The meaning of these methods has not changed and is calle=
d by Pig runtime to cast a !DataByteArray fields to the right type when nee=
ded. In the new API, a !LoadFunc implementation should give a !LoadCaster o=
bject back to pig as the return value of getLoadCaster() method so that it =
can be used for casting. If a null is returned then casting from !DataByteA=
rray to any other type (implicitly or explicitly) in the pig script will no=
t be possible ||
  =

+ An example of how a simple !LoadFunc implementation based on old interfac=
e can be converted to the new interfaces is shown in the Examples section b=
elow. =

+ =

+ =

+ =

+ =3D=3D Examples =3D=3D
+ =

+ =3D=3D=3D Loader =3D=3D=3D
+ =

-  An example of how a simple !LoadFunc implementation based on old interfa=
ce can be converted to the new interfaces will be shown below. The loader i=
mplementation in the example is a loader for text data with line delimiter =
as '\n' and '\t' as default field delimiter (which can be overridden by pas=
sing a different field delimiter in the constructor) - this is similar to c=
urrent !PigStorage loader in Pig. The new implementation uses an existing H=
adoop supported !Inputformat - !TextInputFormat as the underlying !InputFor=
mat.
+ The loader implementation in the example is a loader for text data with l=
ine delimiter as '\n' and '\t' as default field delimiter (which can be ove=
rridden by passing a different field delimiter in the constructor) - this i=
s similar to current !PigStorage loader in Pig. The new implementation uses=
 an existing Hadoop supported !Inputformat - !TextInputFormat as the underl=
ying !InputFormat.
  =

  =3D=3D Old Implementation =3D=3D
  {{{