ibatis-user-java mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Larry Meadors" <lmead...@apache.org>
Subject Re: how to map huge resultsets?
Date Thu, 04 Oct 2007 13:54:49 GMT
So does the Spring-Batch framework only process Lists?

What are the options for feeding it?

Larry


On 10/4/07, robert.kasanicky@accenture.com
<robert.kasanicky@accenture.com> wrote:
>
>
>
>
> Thanks for the insights Jeff.
>
>
>
> Shifting a paradigm and letting iBATIS control the flow is not an option
> this case, but I think iBATIS can still be used in a slightly different
> "driving query" scenario. In this case a "driving query" is run first which
> retrieves all primary keys. When framework asks for next item, the item
> provider takes the next key and asks iBATIS for the corresponding object –
> it is not what I was originally looking for, but still useful.
>
>
>
> Btw. the motivation for my post is to figure out how ibatis can be
> integrated as an input source for the emerging Spring-Batch framework. It
> provides support for ordinary sql with manual mapping, but supporting ORM
> frameworks (especially Hibernate & iBATIS) would be a valuable addition.
>
>
>
> Robert
>
>
>
>  ________________________________
>
>
> From: Jeff Butler [mailto:jeffgbutler@gmail.com]
>  Sent: Thursday, October 04, 2007 14:15
>
>  To: user-java@ibatis.apache.org
>  Subject: Re: how to map huge resultsets?
>
>
>
>
>
> Your question is not that confusing.  Here are the answers:
>
>
>
>
>
> 1. No (although the approach suggested by Christopher Lamey is very close if
> you are willing to shift your paradigm)
>
>
>
>
>
> 2. If the result set is truly huge, then approach #1 is a bad idea because
> iBATIS will have to read through huge amounts of data to get to the latter
> parts of the result set in subsequent calls (this is why Larry asked about
> selecting by primary key rather than returning such a huge result set)
>
>
>
>
>
> I guess you'll have to stick with Hibernate :)
>
>
>
>
>
> Jeff Butler
>
>
>
>
> On 10/4/07, robert.kasanicky@accenture.com <robert.kasanicky@accenture.com >
> wrote:
>
> Seems like my question was quite confusing, so I'll try to explain what
>  I need once again from scratch.
>
>  1. I need to process a potentially huge number of rows (so
>  queryForList(query) is not an option because it would try to load all
>  objects into memory at once).
>
>  2. I need to control the iteration over the query results, so I can't
>  use queryWithRowHandler (in this case iBATIS iterates and I only tell
>  iBATIS how each record should be processed - I need to be able to ask
>  iBATIS for the next record instead - it is the internal vs. external
>  iterator difference, or SAX vs. StAX in case of XML processing or how
>  collections are typically iterated in ruby/groovy vs. Java or ... you
>  name it). Simply framework iterates, iBATIS knows the query and must
>  provide next record when it is asked for it.
>
>  First approach:
>  It is possible to implement these requirements by using
>  queryForList(query, skipSize, maxSize) and  query the database
>  TOTAL_ROW_COUNT / maxSize times (issuing a new query when records from
>  current list have been processed - this is invisible to the user
>  (framework) who just iteratively asks for next record for processing).
>
>  Second approach:
>  Hibernate allows to get a ScrollableResults object for a query allowing
>  the user to move the cursor and ask for an object that corresponds to
>  the current row.
>
>  Questions:
>  1. is the second approach possible with iBATIS?
>  2. first vs. second approach pros & cons?
>
>  Thanks
>  Robert
>
>  -----Original Message-----
>  From: larry.meadors@gmail.com [mailto:larry.meadors@gmail.com] On Behalf
>  Of Larry Meadors
>  Sent: Thursday, October 04, 2007 01:18
>  To: user-java@ibatis.apache.org
>  Subject: Re: how to map huge resultsets?
>
>  I'm confused, how would grabbing chunks of a huge result set be more
>  efficient than grabbing the records by PK?
>
>  Couldn't you just have a select that grabbed the requested item, and
>  passed it back to the consumer?
>
>  Larry
>
>
>  On 10/3/07, robert.kasanicky@accenture.com
>  <robert.kasanicky@accenture.com> wrote:
>  > My problem with RowHandler is that iBATIS controls the iteration. I
>  just say
>  >
>  > sqlMap.queryWithRowHandler ("getAllItems", rowHandler);
>  >
>  > and all items get processed by the rowHandler.
>  >
>  > But in my case I need to make iBATIS return items one-by-one when it
>  is asked to do so because the framework controls the iteration.
>  > This is a very simplified basic logic of the framework:
>  >
>  > while (itemProvider.hasNext()) {
>  >  Object item = itemProvider.next ();
>  >  process(item);
>  > }
>  >
>  >
>  >
>  > -----Original Message-----
>  > From: Christopher Lamey [mailto:clamey@localmatters.com]
>  > Sent: Wed 10/3/2007 11:55 PM
>  > To: user-java@ibatis.apache.org
>  > Subject: Re: how to map huge resultsets?
>  >
>  > Hmm...I don't see how having an external framework prevents you from
>  using a
>  > RowHandler.  Your item provider could implement the RowHandler
>  interface and
>  > the external code wouldn't know or care about it.  Or your item
>  provider
>  > could wrap something that does implement RowHandler so the external
>  code
>  > doesn't know it exists.  The main point is that you can pull mapped
>  objects
>  > on a row by row basis from the database.
>  >
>  > How is a RowHandler different that what you were describing in your
>  first
>  > mail?
>  >
>  > On 10/3/07 3:33 PM, "robert.kasanicky@accenture.com"
>  > <robert.kasanicky@accenture.com > wrote:
>  >
>  > >
>  > > Thanks for your reply.
>  > >
>  > > I can't use the rowhandler callback because the iteration is
>  external to
>  > > iBATIS. In my case a batch framework iteratively asks for an item
>  and
>  > > processes it - and I am trying to implement an iBATIS item provider
>  (I realize
>  > > now I should have explained this in the initial post).
>  > >
>  > > Robert
>  > >
>  > >
>  > > -----Original Message-----
>  > > From: Christopher Lamey [mailto:clamey@localmatters.com]
>  > > Sent: Wed 10/3/2007 11:06 PM
>  > > To: user-java@ibatis.apache.org
>  > > Subject: Re: how to map huge resultsets?
>  > >
>  > > Hello,
>  > >
>  > > You should take a look at the RowHandler interface and the
>  > > queryWithRowHandler calls in SqlMapClient (page 61 of the pdf).
>  Basically,
>  > > the RowHandler gets invoked for every row returned rather than
>  mapping all
>  > > the rows into objects in a collection.
>  > >
>  > > Cheers,
>  > > Chris
>  > >
>  > >
>  > > On 10/3/07 2:37 PM, "robert.kasanicky@accenture.com"
>  > > <robert.kasanicky@accenture.com > wrote:
>  > >
>  > >> Hello,
>  > >>
>  > >> I am wondering whether it possible to implement the following
>  scenario with
>  > >> iBATIS:
>  > >>
>  > >>    1. run an iBATIS-managed select
>  > >>    2. get a scrollable result set instead of a list of mapped
>  objects
>  > >>    3. manually scroll the result set and ask iBATIS for object
>  corresponding
>  > >> to current row
>  > >>
>  > >> Hibernate provides this possibility
>  > >>
> (http://www.hibernate.org/hib_docs/reference/en/html/batch.html
> ) so
>  I thought
>  > >> it would be feasible with iBATIS too, but I couldn't figure out a
>  way. The
>  > >> motivation is a batch scenario where the select returns a huge
>  number of rows
>  > >> so all mapped objects can't be loaded into memory at once.
>  > >>
>  > >> The iBATIS way I am aware of is to use queryForList(String
>  statementName, int
>  > >> skipResults, int maxResults), but this means querying the database
>  > >> (TOTAL_NUMBER_OF_ROWS / maxResults) times.
>  > >>
>  > >> Can somebody give advice about pros & cons of the two approaches?
>  > >>
>  > >> Thanks
>  > >> Robert
>  > >>
>  > >>
>  > >> This message is for the designated recipient only and may contain
>  privileged,
>  > >> proprietary, or otherwise private information.  If you have
>  received it in
>  > >> error, please notify the sender immediately and delete the
>  original.  Any
>  > >> other use of the email by you is prohibited.
>  > >
>  > >
>  > >
>  > >
>  > > This message is for the designated recipient only and may contain
>  privileged,
>  > > proprietary, or otherwise private information.  If you have received
>  it in
>  > > error, please notify the sender immediately and delete the original.
>  Any
>  > > other use of the email by you is prohibited.
>  >
>  >
>  >
>  >
>  > This message is for the designated recipient only and may contain
>  privileged, proprietary, or otherwise private information.  If you have
>  received it in error, please notify the sender immediately and delete
>  the original.  Any other use of the email by you is prohibited.
>  >
>  >
>
>
>  This message is for the designated recipient only and may contain
> privileged, proprietary, or otherwise private information.  If you have
> received it in error, please notify the sender immediately and delete the
> original.  Any other use of the email by you is prohibited.
>
>
>
>
>
>
>
> This message is for the designated recipient only and may contain
> privileged, proprietary, or otherwise private information. If you have
> received it in error, please notify the sender immediately and delete the
> original. Any other use of the email by you is prohibited.

Mime
View raw message