Mailing-List: contact pig-commits-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: pig-dev@hadoop.apache.org
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: quoted-printable
From: Apache Wiki <wikidiffs@apache.org>
To: Apache Wiki <wikidiffs@apache.org>
Date: Wed, 10 Feb 2010 21:54:22 -0000
Message-ID: <20100210215422.27979.1490@eos.apache.org>
Subject: 
 =?utf-8?q?=5BPig_Wiki=5D_Update_of_=22LoadStoreMigrationGuide=22_by_Prade?=
 =?utf-8?q?epKamath?=

Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Pig Wiki" for chang=
e notification.

The "LoadStoreMigrationGuide" page has been changed by PradeepKamath.
http://wiki.apache.org/pig/LoadStoreMigrationGuide?action=3Ddiff&rev1=3D9&r=
ev2=3D10

--------------------------------------------------

  =

  The main change is that the new !LoadFunc API is based on a !InputFormat =
to read the data. Implementations can choose to use existing !InputFormats =
like !TextInputFormat or implement a new one.
   =

- =3D=3D Table mapping old API calls to new API calls =3D=3D
+ =3D=3D Table mapping old API calls to new API calls in rough order of cal=
l sequence=3D=3D
  || '''Old Method in !LoadFunc''' || '''Equivalent New Method''' || '''New=
 Class/Interface in which method is present''' || '''Explanation''' ||
+ || No equivalent method || setUDFContextSignature() || !LoadFunc || This =
method will be called by Pig both in the front end and back end to pass a u=
nique signature to the Loader. The signature can be used to store into the =
UDFContext} any information which the Loader needs to store between various=
 method invocations in the front end and back end. A use case is to store !=
RequiredFieldList passed to it in !LoadPushDown.pushProjection(!RequiredFie=
ldList) for use in the back end before returning tuples in getNext()||
+ || No equivalent method || relativeToAbsolutePath() || !LoadFunc || Pig r=
untime will call this method to allow the Loader to convert a relative load=
 location to an absolute location. The default implementation provided in !=
LoadFunc handles this for hdfs files and directories. If the load source is=
 something else, loader implementation may choose to override this.||
+ || determineSchema() || getSchema() || !LoadMetadata || determineSchema()=
 was used by old code to ask the loader to provide a schema for the data re=
turned by it - the same semantics are now achieved through getSchema() of t=
he !LoadMetadata interface. !LoadMetadata is an optional interface for load=
ers to implement - if a loader does not implement it, this will indicate to=
 the pig runtime that the loader cannot return a schema for the data ||
+ || fieldsToRead() || pushProject() || !LoadPushDown || fieldsToRead() was=
 used by old code to convey to the loader the exact fields required by the =
pig script -the same semantics are now achieved through pushProject() of th=
e !LoadPushDown interface. !LoadPushDown is an optional interface for loade=
rs to implement - if a loader does not implement it, this will indicate to =
the pig runtime that the loader is not capable of returning just the requir=
ed fields and will return all fields in the data. If a loader implementatio=
n is able to efficiently return only required fields, it should implement !=
LoadPushDown to improve query performance||
+ || No equivalent method || getInputFormat() ||!LoadFunc ||  This method w=
ill be called by Pig to get the !InputFormat used by the loader. The method=
s in the !InputFormat (and underlying !RecordReader) will be called by pig =
in the same manner (and in the same context) as by Hadoop in a map-reduce j=
ava program.||
+ || No equivalent method || setLocation() || !LoadFunc || This method is c=
alled by Pig to communicate the load location to the loader. The loader sho=
uld use this method to communicate the same information to the underlying !=
InputFormat. This method is called multiple times by pig - implementations =
should bear in mind that this method is called multiple times and should en=
sure there are no inconsistent side effects due to the multiple calls.||
  || bindTo() || prepareToRead() || !LoadFunc || bindTo() was the old metho=
d which would provide an !InputStream among other things to the !LoadFunc. =
The !LoadFunc implementation would then read from the !InputStream in getNe=
xt(). In the new API, reading of the data is through the !InputFormat provi=
ded by the !LoadFunc. So the equivalent call is prepareToRead() wherein the=
 !RecordReader associated with the !InputFormat provided by the !LoadFunc i=
s passed to the !LoadFunc. The !RecordReader can then be used by the implem=
entation in getNext() to return a tuple representing a record of data back =
to pig. ||
  || getNext() || getNext() || !LoadFunc || The meaning of getNext() has no=
t changed and is called by Pig runtime to get the next tuple in the data ||
  || bytesToInteger(),...bytesToBag() ||  bytesToInteger(),...bytesToBag() =
|| !LoadCaster || The meaning of these methods has not changed and is calle=
d by Pig runtime to cast a !DataByteArray fields to the right type when nee=
ded. In the new API, a !LoadFunc implementation should give a !LoadCaster o=
bject back to pig as the return value of getLoadCaster() method so that it =
can be used for casting. If a null is returned then casting from !DataByteA=
rray to any other type (implicitly or explicitly) in the pig script will no=
t be possible ||
- || fieldsToRead() || pushProject() || !LoadPushDown || fieldsToRead() was=
 used by old code to convey to the loader the exact fields required by the =
pig script -the same semantics are now achieved through pushProject() of th=
e !LoadPushDown interface. !LoadPushDown is an optional interface for loade=
rs to implement - if a loader does not implement it, this will indicate to =
the pig runtime that the loader is not capable of returning just the requir=
ed fields and will return all fields in the data. If a loader implementatio=
n is able to efficiently return only required fields, it should implement !=
LoadPushDown to improve query performance||
- || determineSchema() || getSchema() || !LoadMetadata || determineSchema()=
 was used by old code to ask the loader to provide a schema for the data re=
turned by it - the same semantics are now achieved through getSchema() of t=
he !LoadMetadata interface. !LoadMetadata is an optional interface for load=
ers to implement - if a loader does not implement it, this will indicate to=
 the pig runtime that the loader cannot return a schema for the data ||
- || No equivalent method || relativeToAbsolutePath() || Pig runtime will c=
all this method to allow the Loader to convert a relative load location to =
an absolute location. The default implementation provided in !LoadFunc hand=
les this for hdfs files and directories. If the load source is something el=
se, loader implementation may choose to override this.||
- || No equivalent method || getInputFormat() || This method will be called=
 by Pig to get the !InputFormat used by the loader. The methods in the !Inp=
utFormat (and underlying !RecordReader) will be called by pig in the same m=
anner (and in the same context) as by Hadoop in a map-reduce java program.||
- || No equivalent method || setLocation() || This method is called by Pig =
to communicate the load location to the loader. The loader should use this =
method to communicate the same information to the underlying !InputFormat. =
This method is called multiple times by pig - implementations should bear i=
n mind that this method is called multiple times and should ensure there ar=
e no inconsistent side effects due to the multiple calls.||
- || No equivalent method || setUDFContextSignature() || This method will b=
e called by Pig both in the front end and back end to pass a unique signatu=
re to the Loader. The signature can be used to store into the UDFContext} a=
ny information which the Loader needs to store between various method invoc=
ations in the front end and back end. A use case is to store RequiredFieldL=
ist passed to it in LoadPushDown.pushProjection(RequiredFieldList) for use =
in the back end before returning tuples in getNext()||
  =

-  An example of how a simple !LoadFunc implementation based on old interfa=
ce can be converted to the new interfaces will be shown below. The loader i=
mplementation in the example is a loader for text data with line delimiter =
as '\n' and '\t' as default field delimiter (which can be overridden by pas=
sing a different field delimiter in the constructor) - this is similar to c=
urrent !PigStorage loader in Pig.
+  An example of how a simple !LoadFunc implementation based on old interfa=
ce can be converted to the new interfaces will be shown below. The loader i=
mplementation in the example is a loader for text data with line delimiter =
as '\n' and '\t' as default field delimiter (which can be overridden by pas=
sing a different field delimiter in the constructor) - this is similar to c=
urrent !PigStorage loader in Pig. The new implementation uses an existing H=
adoop supported !Inputformat - !TextInputFormat as the underlying !InputFor=
mat.
  =

  =3D=3D Old Implementation =3D=3D
  {{{