ibatis-user-java mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Nathan Maves" <nathan.ma...@gmail.com>
Subject Re: select * causing " OutOfMemoryError: Java heap space"
Date Tue, 18 Mar 2008 14:56:40 GMT
Are we absolutely positive that we are using the same driver with hibernate
as we are with ibatis?


On Mon, Mar 17, 2008 at 4:18 PM, Clinton Begin <clinton.begin@gmail.com>
wrote:

> Wow, I just read that other thread on the Compass site.... This is not
> directly an iBATIS problem at all...
>
> at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1268)
> at com.mysql.jdbc.PreparedStatement.execute(PreparedStatement.java:843)
>
> This line tells me that the execution is never returning from the prepared
> statement.  iBATIS never has a chance to even see the ResultSet, let alone
> get to the RowHandler. So swapping out the GpsDevice here won't make any
> difference at all. This is odd behavior for a JDBC driver and tells me that
> something is very wrong with the configuration.
>
> Which version of the JDBC driver are you using?  What are the rest of the
> iBATIS convfiguration details?
>
> Clinton
>
>
>
> On Mon, Mar 17, 2008 at 4:12 PM, Clinton Begin <clinton.begin@gmail.com>
> wrote:
>
> > Thanks for the summary.  A few things:
> >
> > * PaginatedList was definitely the wrong solution for that, so I'm glad
> > it's gone.
> >
> > * A RowHandler is possibly the right solution, but will only store as
> > many records in memory as you tell it to.  iBATIS does not just arbitrarily
> > keep records around.  Quite the opposite actually.  Unless you keep a
> > reference or configure a cache, iBATIS will not keep the object around at
> > all.
> >
> > Looking at your implementation, there are a couple of things going on...
> >
> > session.create(o);
> > if(currentItem== pageSize){
> >   if (log.isDebugEnabled()) {
> >     log.debug("Indexing page number ["+ pageCount++ + "]");
> >   }
> > session.evictAll();
> >
> > Things to Try:
> >
> > 1)  These lines of code must be treated as guilty until proven innocent
> > (especially the logging).  The easiest way to do that is to delete them.
> > Create an empty handler/GpsDevice and run the test to see what happens.
> >
> > 2) Look at the iBATIS configuration files.  Is a cache configured for
> > this query?  What kind of cache?
> >
> > 3) You could try multiple queries of fixed sizes with
> > queryForList(String,Object,int offset,int limit) ... but I can't see how
> > that would be better than a RowHandler.
> >
> > 4) Next, just in case the driver's default fetch size is ridiculously
> > high (or unlimited), try setting <select ... fetchSize="100"/>.  You could
> > also try a scrollable SesultSet with <statement ... resultSetType /> of
> > SCROLL_INSENSITIVE or SCROLL_SENSITIVE.
> >
> > Finally, please know that I've used iBATIS to load tens of millions of
> > rows for ETL and even analysis (NetFlix Prize) and was able to do so with a
> > RowHandler or even simply multiple calls to queryForList(String,Object,int
> > offset,int limit).  I don't recall having to even think about the
> > configuration, although I did manage to optimize it to achieve a read/insert
> > performance of 15,000 records per second between two databases (on a single
> > machine too).
> >
> > I'm sure there's something about the configuration here that is causing
> > the problem, and it can be solved.
> >
> > I hope one of these helps.
> >
> > Clinton
> >
> >
> > On Mon, Mar 17, 2008 at 1:53 PM, nch <underscore_dot@yahoo.com> wrote:
> >
> > >
> > > Hi, there. :-)
> > >
> > > The thing is, I'm testing a search engine called Compass. One of the
> > > tests consists of indexing a Wikipedia dump. In this case only the text of
> > > the articles, so I downloaded it and imported it into a MySQL database
> > > (about 650000 records - 1.5GB).
> > >
> > > You can configure Compass to access a database and index it's contents
> > > by configuring what they call a GPS Device. You can do that through an ORM
> > > such as iBatis, Hibernate or JPA.
> > >
> > > Compass provides an implementation of such a GPS Device called
> > > SqlMapClientGpsDevice which uses iBatis queryForPaginatedList to get the
> > > results of a query and, so, index them. The query is just a "select * from
> > > articles_table".
> > >
> > > So I wired everything up and ran several tests with different amounts
> > > of data to be indexed and JVM stack space sizes. The result was an
> > > "OutOfMemoryError: java heap space" error message whenever the size of the
> > > selected rows was bigger than the amount of available heap space (I can not
> > > index the 650000 records having 1GB of stack).
> > >
> > > After doing some profiling I concluded iBatis was loading the whole
> > > amount of results into memory. So I decided to try Hibernate, which worked
> > > just fine.
> > >
> > > Finally I found queryForPaginatedList was deprecated and that I should
> > > use queryWithRowHandler instead, so I made my own implementation of the
> > > SqlMapClientGpsDevice based on queryWithRowHandler and tested it, but I had
> > > the same result (I profiled the application stack usage and I could see a
> > > line growing steadily and finally crash against the stack ceiling while
> > > performing the select statement).
> > >
> > > Shay Banon, Compass project leader, is planning to patch
> > > SqlMapClientGpsDevice by replacing queryForPaginatedList with
> > > queryWithRowHandler, but I'm doubtful it's gonna work.
> > >
> > > Finally you can, of course, devide the select stament into several
> > > selects (and it works) but, in my opinion, that shouldn't be necessary.
> > > You can read more on
> > > http://forum.compass-project.org/thread.jspa?threadID=215278
> > >
> > > Thanks!
> > >
> > >
> > > ----- Original Message ----
> > > From: Clinton Begin <clinton.begin@gmail.com>
> > > To: user-java@ibatis.apache.org
> > > Sent: Monday, March 17, 2008 2:22:39 PM
> > > Subject: Re: select * causing " OutOfMemoryError: Java heap space"
> > >
> > > Wow, this thread is interesting.   I suppose I could read the past
> > > emails, but could someone summarize the problem and progress to date?  I may
> > > be able to shed some light on what is happening.
> > >
> > > Clinton
> > >
> > > On Sun, Mar 16, 2008 at 11:49 PM, nch <underscore_dot@yahoo.com>
> > > wrote:
> > >
> > > >
> > > > Another thing I forgot. This works perfectly well if we replace
> > > > iBatis by Hibernate, so I don't think this is being caused by the MySQL
> > > > driver.
> > > >
> > > > ----- Original Message ----
> > > > From: nch <underscore_dot@yahoo.com>
> > > > To: user-java@ibatis.apache.org
> > > > Sent: Sunday, March 16, 2008 10:18:42 PM
> > > > Subject: Re: select * causing " OutOfMemoryError: Java heap space"
> > > >
> > > >
> > > > Well, I modified the mediumblob into a mediumtext and removed all
> > > > other fields except the id one.
> > > >
> > > > ----- Original Message ----
> > > > From: nch <underscore_dot@yahoo.com>
> > > > To: user-java@ibatis.apache.org
> > > > Sent: Sunday, March 16, 2008 10:06:34 PM
> > > > Subject: Re: select * causing " OutOfMemoryError: Java heap space"
> > > >
> > > >
> > > > You're probably right. I'm using the table called "text" which
> > > > contains a field of type "mediumblob".
> > > >
> > > > CREATE TABLE /*$wgDBprefix*/text (
> > > >   old_id int unsigned NOT NULL auto_increment,
> > > >   old_text mediumblob NOT NULL,
> > > >   old_flags tinyblob NOT NULL,
> > > >   PRIMARY KEY old_id (old_id)
> > > > ) /*$wgDBTableOptions*/ MAX_ROWS=10000000 AVG_ROW_LENGTH=10240;
> > > >
> > > >
> > > > You can find it's definition here:
> > > >
> > > > http://svn.wikimedia.org/viewvc/mediawiki/trunk/phase3/maintenance/tables.sql?view=markup
> > > >
> > > > I'll try to modify that field, first, and see what happens.
> > > > I've already tried "select * from your_wikipedia_dump limit $start$,
> > > > $size$" and it seems to work fine, but I see a possible issue with this,
you
> > > > can not remove any of the records you've already read during the whole
> > > > process. This might not be an option.
> > > >
> > > > Thank you.
> > > >
> > > > ----- Original Message ----
> > > > From: Larry Meadors <larry.meadors@gmail.com>
> > > > To: user-java@ibatis.apache.org
> > > > Sent: Sunday, March 16, 2008 4:19:43 PM
> > > > Subject: Re: select * causing " OutOfMemoryError: Java heap space"
> > > >
> > > > Yeah, I just noticed that - it looks like it's failing in the jdbc
> > > > driver when it tries to read a blob, maybe?
> > > >
> > > > I know that some versions of the postgresql jdbc driver load the
> > > > entire contents of a resultset into a massive byte[][] structure. I
> > > > wonder if that is what is happening here? I think you can get the
> > > > source for the MySQL driver and check that if you want to.
> > > >
> > > > If it is, you may need to structure your app to take smaller bites
> > > > of
> > > > this elephant.
> > > >
> > > > Try "select * from your_wikipedia_dump limit $start$, $size$"
> > > > instead
> > > > - that will limit the size of the results to $size$ rows, starting
> > > > on
> > > > row $start$ (the $start$ value is zero-based). If you experiment to
> > > > see how big $size$ can get, you should be able to get pretty decent
> > > > performance out of this.
> > > >
> > > > Psuedo-code would be like this...
> > > >
> > > > int start = 0;
> > > > int size = 1000; // see how big you can make this to improve
> > > > performance
> > > > while(still_more_data){
> > > >   still_more_data = index(start, size);
> > > >   start += size;
> > > > }
> > > >
> > > > Your index method will return true if the query returned any data.
> > > > If
> > > > the query returns no data, it'll return false.
> > > >
> > > > I can't imagine what Hibernate is doing differently here to make
> > > > this
> > > > work if the JDBC driver is failing when executing the query.
> > > >
> > > > Larry
> > > >
> > > >
> > > > On Sun, Mar 16, 2008 at 4:12 AM, nch <underscore_dot@yahoo.com>
> > > > wrote:
> > > > >
> > > > >
> > > > > Sure. Please, see attached.
> > > > > I don't think the problem is in the RowHandler, though, because
> > > > the
> > > > > OutOfMemoryError occurs before invoking RowHandle#handleRow.
> > > > >
> > > > > Cheers
> > > > >
> > > > >
> > > > > ----- Original Message ----
> > > > > From: Larry Meadors <larry.meadors@gmail.com>
> > > > > To: user-java@ibatis.apache.org
> > > > >
> > > > > Sent: Sunday, March 16, 2008 3:13:27 AM
> > > > > Subject: Re: select * causing " OutOfMemoryError: Java heap space"
> > > > >
> > > > >  Can you post the row handler you are using.
> > > > >
> > > > > Larry
> > > > >
> > > > >
> > > > > On Sat, Mar 15, 2008 at 1:14 PM, nch <underscore_dot@yahoo.com>
> > > > wrote:
> > > > > >
> > > > > >
> > > > > > Hi, Nathan.
> > > > > > I did so, but I'm still having the same issue. Perhaps I'm not
> > > > using it
> > > > > > correctly?
> > > > > > See my last post to this forum entry:
> > > > > >
> > > > > > http://forum.compass-project.org/thread.jspa?threadID=215278
> > > > > >
> > > > > > See the stack trace:
> > > > > >
> > > > > > MemoryError: Java heap space:
> > > > > > java.lang.OutOfMemoryError: Java heap space
> > > > > >        at com.mysql.jdbc.Buffer.getBytes(Buffer.java:198)
> > > > > >        at com.mysql.jdbc.Buffer.readLenByteArray(Buffer.java
> > > > :318)
> > > > > >        at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1366)
> > > > > >        at com.mysql.jdbc.MysqlIO.readSingleRowSet(MysqlIO.java
> > > > :2333)
> > > > > >        at com.mysql.jdbc.MysqlIO.getResultSet(MysqlIO.java:435)
> > > > > >        at
> > > > > > com.mysql.jdbc.MysqlIO.readResultsForQueryOrUpdate(MysqlIO.java
> > > > :2040)
> > > > > >        at com.mysql.jdbc.MysqlIO.readAllResults(MysqlIO.java
> > > > :1443)
> > > > > >        at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java
> > > > :1777)
> > > > > >        at com.mysql.jdbc.Connection.execSQL(Connection.java
> > > > :3249)
> > > > > >        at
> > > > > >
> > > > > com.mysql.jdbc.PreparedStatement.executeInternal(
> > > > PreparedStatement.java:1268)
> > > > > >        at
> > > > > > com.mysql.jdbc.PreparedStatement.execute(PreparedStatement.java
> > > > :843)
> > > > > >        at
> > > > > >
> > > > > org.apache.commons.dbcp.DelegatingPreparedStatement.execute(
> > > > DelegatingPreparedStatement.java:169)
> > > > > >        at
> > > > > >
> > > > > org.apache.commons.dbcp.DelegatingPreparedStatement.execute(
> > > > DelegatingPreparedStatement.java:169)
> > > > > >        at
> > > > > >
> > > > > com.ibatis.sqlmap.engine.execution.SqlExecutor.executeQuery(
> > > > SqlExecutor.java:186)
> > > > > >        at
> > > > > >
> > > > >
> > > > com.ibatis.sqlmap.engine.mapping.statement.GeneralStatement.sqlExecuteQuery
> > > > (GeneralStatement.java:205)
> > > > > >        at
> > > > > >
> > > > >
> > > > com.ibatis.sqlmap.engine.mapping.statement.GeneralStatement.executeQueryWithCallback
> > > > (GeneralStatement.java:173)
> > > > > >        at
> > > > > >
> > > > >
> > > > com.ibatis.sqlmap.engine.mapping.statement.GeneralStatement.executeQueryWithRowHandler
> > > > (GeneralStatement.java:133)
> > > > > >        at
> > > > > >
> > > > >
> > > > com.ibatis.sqlmap.engine.impl.SqlMapExecutorDelegate.queryWithRowHandler
> > > > (SqlMapExecutorDelegate.java:649)
> > > > > >        at
> > > > > >
> > > > >
> > > > com.ibatis.sqlmap.engine.impl.SqlMapSessionImpl.queryWithRowHandler(
> > > > SqlMapSessionImpl.java:156)
> > > > > >        at
> > > > > >
> > > > > org.myorg.compass.SqlMapClientGpsDeviceWithRowHandler.doIndex(
> > > > SqlMapClientGpsDeviceWithRowHandler.java:33)
> > > > > >        at
> > > > > >
> > > > >
> > > > org.compass.gps.device.AbstractGpsDevice$1.doInCompassWithoutResult(
> > > > AbstractGpsDevice.java:93)
> > > > > >        at
> > > > > >
> > > > > org.compass.core.CompassCallbackWithoutResult.doInCompass(
> > > > CompassCallbackWithoutResult.java:29)
> > > > > >        at
> > > > > > org.compass.core.CompassTemplate.execute(CompassTemplate.java
> > > > :132)
> > > > > >        at
> > > > > >
> > > > > org.compass.gps.impl.SingleCompassGps.executeForIndex(
> > > > SingleCompassGps.java:161)
> > > > > >        at
> > > > > > org.compass.gps.device.AbstractGpsDevice.index(
> > > > AbstractGpsDevice.java:91)
> > > > > >        at
> > > > > >
> > > > >
> > > > org.compass.spring.device.SpringSyncTransactionGpsDeviceWrapper$1.doInTransactionWithoutResult
> > > > (SpringSyncTransactionGpsDeviceWrapper.java:98)
> > > > > >        at
> > > > > >
> > > > >
> > > > org.springframework.transaction.support.TransactionCallbackWithoutResult.doInTransaction
> > > > (TransactionCallbackWithoutResult.java:33)
> > > > > >        at
> > > > > >
> > > > >
> > > > org.springframework.transaction.support.TransactionTemplate.execute(
> > > > TransactionTemplate.java:128)
> > > > > >        at
> > > > > >
> > > > >
> > > > org.compass.spring.device.SpringSyncTransactionGpsDeviceWrapper.index
> > > > (SpringSyncTransactionGpsDeviceWrapper.java:96)
> > > > > >        at
> > > > > >
> > > > > org.compass.gps.impl.SingleCompassGps$1.buildIndexIfNeeded(
> > > > SingleCompassGps.java:133)
> > > > > >        at
> > > > > >
> > > > >
> > > > org.compass.core.lucene.engine.manager.DefaultLuceneSearchEngineIndexManager$8.firstStep
> > > > (DefaultLuceneSearchEngineIndexManager.java:233)
> > > > > >        at
> > > > > >
> > > > >
> > > > org.compass.core.lucene.engine.manager.DefaultLuceneSearchEngineIndexManager.doOperate
> > > > (DefaultLuceneSearchEngineIndexManager.java:182)
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > ----- Original Message ----
> > > > > > From: Nathan Maves <nathan.maves@gmail.com>
> > > > > > To: user-java@ibatis.apache.org
> > > > > >
> > > > > > Sent: Saturday, March 15, 2008 5:43:04 PM
> > > > > > Subject: Re: select * causing " OutOfMemoryError: Java heap
> > > > space"
> > > > > >
> > > > > >  this is only my 2 cents but I would throw that class out.
> > > > Write your own
> > > > > > implementation which uses a row handler.  This is the type of
> > > > situation in
> > > > > > which a row handler could really help.  almost no memory would
> > > > be used.
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Thu, Mar 13, 2008 at 4:57 PM, nch <underscore_dot@yahoo.com>
> > > > wrote:
> > > > > > >
> > > > > > > Yes, I agree that dividing the query into several queries
> > > > should do as
> > > > > > long as none of them uses more memory than available. Gracias,
> > > > Carlos.
> > > > > > >
> > > > > > > Nevertheless, I looked into
> > > > > > org.compass.gps.device.ibatis.SqlMapClientGpsDevice#doIndex
and,
> > > > > > fundamentally, what it does is a queryForPaginatedList and
> > > > iterates
> > > > > through
> > > > > > the resulting PaginatedList with nextPage()
> > > > > > >
> > > > > > > I guess that the reason why pages are kept in memory after
> > > > performing
> > > > > > paginatedList.nextPage() is because a paginatedList.previousPage()
> > > > can be
> > > > > > issued but, should't it free pages if heap space is running
out?
> > > > > > >
> > > > > > >
> > > > > > > Carlos de Luna Saenz <cdelunasaenz@yahoo.com.mx>
wrote:
> > > > > > >
> > > > > > > Since we have a similar trouble around here i must say
that
> > > > you have lot
> > > > > > of options when going to compass..
> > > > > > > The first one is to index "object by object" by hand, the
> > > > second one is
> > > > > to
> > > > > > use the Gps wich will use the complete bunch of data and make
> > > > subindexes
> > > > > > (making parts of your "main index" with diferent "maps" for
each
> > > > subindex
> > > > > in
> > > > > > your gpsDriver.
> > > > > > > if your indexing is as strong as ours i should look forward
> > > > for numer 1
> > > > > or
> > > > > > number 3 depending of your needs. (More a Compass than iBatis
> > > > issue)
> > > > > > > Greetings
> > > > > > > Carlos de Luna
> > > > > > >
> > > > > > > ----- Mensaje original ----
> > > > > > > De: nch
> > > > > > >
> > > > > > > Para: user-java@ibatis.apache.org; larry.meadors@gmail.com
> > > > > > > Enviado: jueves, 13 de marzo, 2008 9:37:04
> > > > > > > Asunto: Re: select * causing " OutOfMemoryError: Java heap
> > > > space"
> > > > > > >
> > > > > > >
> > > > > > > Sorry. Yes, I need all the data at one time, because the
> > > > Compass API
> > > > > seems
> > > > > > to be meant in that way.
> > > > > > >
> > > > > > > Larry Meadors wrote: OK, so the answer to the question
is...[
> > > > yes, i
> > > > > need
> > > > > > all the data at
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > one time | no, i do not need all of the data at one time
].
> > > > > > >
> > > > > > > Larry
> > > > > > >
> > > > > > >
> > > > > > > On Thu, Mar 13, 2008 at 9:04 AM, nch wrote:
> > > > > > > >
> > > > > > > > Hi!
> > > > > > > >
> > > > > > > > I'm doing tests both under tomcat and jetty. I set
1024MB of
> > > > heap
> > > > > space
> > > > > > in
> > > > > > > > both cases. I am using a profiler to look into what's
> > > > happening during
> > > > > > > > execution of the indexing process and I can tell that's
the
> > > > exact
> > > > > amount
> > > > > > it
> > > > > > > > displays for the VM.
> > > > > > > >
> > > > > > > > As I describe in the Compass user forum (see link
bellow),
> > > > I'm
> > > > > indexing
> > > > > > a
> > > > > > > > Wikipedia dump of about 650000-675000 records. That's
sort
> > > > of 1-1.5GB
> > > > > of
> > > > > > > > data.
> > > > > > > >
> > > > > > > > Hibernate can handle all that data, I guess it's using
some
> > > > sort of
> > > > > > > > pagination. iBatis seems to divide the select statement
into
> > > > several
> > > > > > select
> > > > > > > > statements, but it also seems each of such statements
> > > > surpasses the
> > > > > > limit of
> > > > > > > > 1024MB when trying to index only 300000 of the database
> > > > records.
> > > > > > > >
> > > > > > > > Thank you
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > Chris Lamey wrote:
> > > > > > > > What is you JVM heap size set to and how much data
is
> > > > returned by
> > > > > select
> > > > > > *
> > > > > > > > from table?
> > > > > > > >
> > > > > > > > If you're trying to pull back 1G worth of data into
a JVM
> > > > with a heap
> > > > > > size
> > > > > > > > set to 64M, you will hit the heap limit pretty quick.
> > > > > > > >
> > > > > > > >
> > > > > > > > -----Original Message-----
> > > > > > > > From: nch [mailto:underscore_dot@yahoo..com]
> > > > > > > > Sent: Thu 3/13/2008 8:03 AM
> > > > > > > > To: user-java@ibatis.apache.org
> > > > > > > > Subject: select * causing " OutOfMemoryError: Java
heap
> > > > space"
> > > > > > > >
> > > > > > > >
> > > > > > > > Hi, everybody.
> > > > > > > > I'm testing the Compass search engine and, in particular,
> > > > how to index
> > > > > a
> > > > > > big
> > > > > > > > set of documents from a table in a MySQL database.
In order
> > > > to do this
> > > > > I
> > > > > > > > issue a "select * from table_name" using iBatis, but
this
> > > > seems to be
> > > > > > > > causing the application to use all heap space available.
> > > > > > > > I added a new entry into Compass user forum with details:
> > > > > > > > http://forum.compass-project.org/thread.jspa?threadID=215278
> > > > > > > >
> > > > > > > > Can you figure out why is this happening?
> > > > > > > >
> > > > > > > > Many thanks
> > > > > > > >
> > > > > > > >
> > > > > > > > ---------------------------------
> > > > > > > > Be a better friend, newshound, and know-it-all with
Yahoo!
> > > > Mobile. Try
> > > > > > it
> > > > > > > > now.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > ________________________________
> > > > > > > > Never miss a thing. Make Yahoo your homepage.
> > > > > > >
> > > > > > >
> > > > > > > Never miss a thing. Make Yahoo your homepage.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > ____________________________________________________________________________________
> > > > > > > ¡Capacidad ilimitada de almacenamiento en tu correo!
> > > > > > > No te preocupes más por el espacio de tu cuenta con Correo
> > > > Yahoo!:
> > > > > > > http://correo.espanol.yahoo.com/
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > ________________________________
> > > > > > Never miss a thing. Make Yahoo your homepage.
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >  ________________________________
> > > > > > Never miss a thing. Make Yahoo your homepage.
> > > > >
> > > > >
> > > > >  ________________________________
> > > > >
> > > > > Be a better friend, newshound, and know-it-all with Yahoo! Mobile.
> > > > Try it
> > > > > now.
> > > >
> > > >
> > > > ------------------------------
> > > > Looking for last minute shopping deals? Find them fast with Yahoo!
> > > > Search.<http://us.rd.yahoo.com/evt=51734/*http://tools.search.yahoo.com/newsearch/category.php?category=shopping>
> > > >
> > > >
> > > > ------------------------------
> > > > Be a better friend, newshound, and know-it-all with Yahoo! Mobile. Try
> > > > it now.<http://us.rd.yahoo.com/evt=51733/*http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ%20>
> > > >
> > > >
> > > > ------------------------------
> > > > Looking for last minute shopping deals? Find them fast with Yahoo!
> > > > Search.<http://us.rd.yahoo.com/evt=51734/*http://tools.search.yahoo.com/newsearch/category.php?category=shopping>
> > > >
> > >
> > >
> > >
> > > ------------------------------
> > > Never miss a thing. Make Yahoo your homepage.<http://us.rd.yahoo.com/evt=51438/*http://www.yahoo.com/r/hs>
> > >
> >
> >
>
Mime
View raw message