ibatis-user-java mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From <robert.kasani...@accenture.com>
Subject RE: how to map huge resultsets?
Date Thu, 04 Oct 2007 14:06:45 GMT
Spring-Batch processes items (item = arbitrary domain object). It
iteratively asks "item provider" for the next object and passes them to
"item processor" for processing. Item provider and processor are just
interfaces and there are provided implementations for typical inputs and
outputs (flat file, xml, database).

Robert

-----Original Message-----
From: larry.meadors@gmail.com [mailto:larry.meadors@gmail.com] On Behalf
Of Larry Meadors
Sent: Thursday, October 04, 2007 15:55
To: user-java@ibatis.apache.org
Subject: Re: how to map huge resultsets?

So does the Spring-Batch framework only process Lists?

What are the options for feeding it?

Larry


On 10/4/07, robert.kasanicky@accenture.com
<robert.kasanicky@accenture.com> wrote:
>
>
>
>
> Thanks for the insights Jeff.
>
>
>
> Shifting a paradigm and letting iBATIS control the flow is not an
option
> this case, but I think iBATIS can still be used in a slightly
different
> "driving query" scenario. In this case a "driving query" is run first
which
> retrieves all primary keys. When framework asks for next item, the
item
> provider takes the next key and asks iBATIS for the corresponding
object -
> it is not what I was originally looking for, but still useful.
>
>
>
> Btw. the motivation for my post is to figure out how ibatis can be
> integrated as an input source for the emerging Spring-Batch framework.
It
> provides support for ordinary sql with manual mapping, but supporting
ORM
> frameworks (especially Hibernate & iBATIS) would be a valuable
addition.
>
>
>
> Robert
>
>
>
>  ________________________________
>
>
> From: Jeff Butler [mailto:jeffgbutler@gmail.com]
>  Sent: Thursday, October 04, 2007 14:15
>
>  To: user-java@ibatis.apache.org
>  Subject: Re: how to map huge resultsets?
>
>
>
>
>
> Your question is not that confusing.  Here are the answers:
>
>
>
>
>
> 1. No (although the approach suggested by Christopher Lamey is very
close if
> you are willing to shift your paradigm)
>
>
>
>
>
> 2. If the result set is truly huge, then approach #1 is a bad idea
because
> iBATIS will have to read through huge amounts of data to get to the
latter
> parts of the result set in subsequent calls (this is why Larry asked
about
> selecting by primary key rather than returning such a huge result set)
>
>
>
>
>
> I guess you'll have to stick with Hibernate :)
>
>
>
>
>
> Jeff Butler
>
>
>
>
> On 10/4/07, robert.kasanicky@accenture.com
<robert.kasanicky@accenture.com >
> wrote:
>
> Seems like my question was quite confusing, so I'll try to explain
what
>  I need once again from scratch.
>
>  1. I need to process a potentially huge number of rows (so
>  queryForList(query) is not an option because it would try to load all
>  objects into memory at once).
>
>  2. I need to control the iteration over the query results, so I can't
>  use queryWithRowHandler (in this case iBATIS iterates and I only tell
>  iBATIS how each record should be processed - I need to be able to ask
>  iBATIS for the next record instead - it is the internal vs. external
>  iterator difference, or SAX vs. StAX in case of XML processing or how
>  collections are typically iterated in ruby/groovy vs. Java or ... you
>  name it). Simply framework iterates, iBATIS knows the query and must
>  provide next record when it is asked for it.
>
>  First approach:
>  It is possible to implement these requirements by using
>  queryForList(query, skipSize, maxSize) and  query the database
>  TOTAL_ROW_COUNT / maxSize times (issuing a new query when records
from
>  current list have been processed - this is invisible to the user
>  (framework) who just iteratively asks for next record for
processing).
>
>  Second approach:
>  Hibernate allows to get a ScrollableResults object for a query
allowing
>  the user to move the cursor and ask for an object that corresponds to
>  the current row.
>
>  Questions:
>  1. is the second approach possible with iBATIS?
>  2. first vs. second approach pros & cons?
>
>  Thanks
>  Robert
>
>  -----Original Message-----
>  From: larry.meadors@gmail.com [mailto:larry.meadors@gmail.com] On
Behalf
>  Of Larry Meadors
>  Sent: Thursday, October 04, 2007 01:18
>  To: user-java@ibatis.apache.org
>  Subject: Re: how to map huge resultsets?
>
>  I'm confused, how would grabbing chunks of a huge result set be more
>  efficient than grabbing the records by PK?
>
>  Couldn't you just have a select that grabbed the requested item, and
>  passed it back to the consumer?
>
>  Larry
>
>
>  On 10/3/07, robert.kasanicky@accenture.com
>  <robert.kasanicky@accenture.com> wrote:
>  > My problem with RowHandler is that iBATIS controls the iteration. I
>  just say
>  >
>  > sqlMap.queryWithRowHandler ("getAllItems", rowHandler);
>  >
>  > and all items get processed by the rowHandler.
>  >
>  > But in my case I need to make iBATIS return items one-by-one when
it
>  is asked to do so because the framework controls the iteration.
>  > This is a very simplified basic logic of the framework:
>  >
>  > while (itemProvider.hasNext()) {
>  >  Object item = itemProvider.next ();
>  >  process(item);
>  > }
>  >
>  >
>  >
>  > -----Original Message-----
>  > From: Christopher Lamey [mailto:clamey@localmatters.com]
>  > Sent: Wed 10/3/2007 11:55 PM
>  > To: user-java@ibatis.apache.org
>  > Subject: Re: how to map huge resultsets?
>  >
>  > Hmm...I don't see how having an external framework prevents you
from
>  using a
>  > RowHandler.  Your item provider could implement the RowHandler
>  interface and
>  > the external code wouldn't know or care about it.  Or your item
>  provider
>  > could wrap something that does implement RowHandler so the external
>  code
>  > doesn't know it exists.  The main point is that you can pull mapped
>  objects
>  > on a row by row basis from the database.
>  >
>  > How is a RowHandler different that what you were describing in your
>  first
>  > mail?
>  >
>  > On 10/3/07 3:33 PM, "robert.kasanicky@accenture.com"
>  > <robert.kasanicky@accenture.com > wrote:
>  >
>  > >
>  > > Thanks for your reply.
>  > >
>  > > I can't use the rowhandler callback because the iteration is
>  external to
>  > > iBATIS. In my case a batch framework iteratively asks for an item
>  and
>  > > processes it - and I am trying to implement an iBATIS item
provider
>  (I realize
>  > > now I should have explained this in the initial post).
>  > >
>  > > Robert
>  > >
>  > >
>  > > -----Original Message-----
>  > > From: Christopher Lamey [mailto:clamey@localmatters.com]
>  > > Sent: Wed 10/3/2007 11:06 PM
>  > > To: user-java@ibatis.apache.org
>  > > Subject: Re: how to map huge resultsets?
>  > >
>  > > Hello,
>  > >
>  > > You should take a look at the RowHandler interface and the
>  > > queryWithRowHandler calls in SqlMapClient (page 61 of the pdf).
>  Basically,
>  > > the RowHandler gets invoked for every row returned rather than
>  mapping all
>  > > the rows into objects in a collection.
>  > >
>  > > Cheers,
>  > > Chris
>  > >
>  > >
>  > > On 10/3/07 2:37 PM, "robert.kasanicky@accenture.com"
>  > > <robert.kasanicky@accenture.com > wrote:
>  > >
>  > >> Hello,
>  > >>
>  > >> I am wondering whether it possible to implement the following
>  scenario with
>  > >> iBATIS:
>  > >>
>  > >>    1. run an iBATIS-managed select
>  > >>    2. get a scrollable result set instead of a list of mapped
>  objects
>  > >>    3. manually scroll the result set and ask iBATIS for object
>  corresponding
>  > >> to current row
>  > >>
>  > >> Hibernate provides this possibility
>  > >>
> (http://www.hibernate.org/hib_docs/reference/en/html/batch.html
> ) so
>  I thought
>  > >> it would be feasible with iBATIS too, but I couldn't figure out
a
>  way. The
>  > >> motivation is a batch scenario where the select returns a huge
>  number of rows
>  > >> so all mapped objects can't be loaded into memory at once.
>  > >>
>  > >> The iBATIS way I am aware of is to use queryForList(String
>  statementName, int
>  > >> skipResults, int maxResults), but this means querying the
database
>  > >> (TOTAL_NUMBER_OF_ROWS / maxResults) times.
>  > >>
>  > >> Can somebody give advice about pros & cons of the two
approaches?
>  > >>
>  > >> Thanks
>  > >> Robert
>  > >>
>  > >>
>  > >> This message is for the designated recipient only and may
contain
>  privileged,
>  > >> proprietary, or otherwise private information.  If you have
>  received it in
>  > >> error, please notify the sender immediately and delete the
>  original.  Any
>  > >> other use of the email by you is prohibited.
>  > >
>  > >
>  > >
>  > >
>  > > This message is for the designated recipient only and may contain
>  privileged,
>  > > proprietary, or otherwise private information.  If you have
received
>  it in
>  > > error, please notify the sender immediately and delete the
original.
>  Any
>  > > other use of the email by you is prohibited.
>  >
>  >
>  >
>  >
>  > This message is for the designated recipient only and may contain
>  privileged, proprietary, or otherwise private information.  If you
have
>  received it in error, please notify the sender immediately and delete
>  the original.  Any other use of the email by you is prohibited.
>  >
>  >
>
>
>  This message is for the designated recipient only and may contain
> privileged, proprietary, or otherwise private information.  If you
have
> received it in error, please notify the sender immediately and delete
the
> original.  Any other use of the email by you is prohibited.
>
>
>
>
>
>
>
> This message is for the designated recipient only and may contain
> privileged, proprietary, or otherwise private information. If you have
> received it in error, please notify the sender immediately and delete
the
> original. Any other use of the email by you is prohibited.


This message is for the designated recipient only and may contain privileged, proprietary,
or otherwise private information.  If you have received it in error, please notify the sender
immediately and delete the original.  Any other use of the email by you is prohibited.

Mime
View raw message