lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Sokolov <msoko...@gmail.com>
Subject Re: Not-indexed, Stored Thumbnails or NoSQL?
Date Mon, 03 Dec 2018 11:37:23 GMT
Also, you should know that stored fields are handled in a block for each
document, so when you retrieve your first page you are also in some sense
paying for skipping over (loading into RAM and decoding) the large blobs
too. As you scale, and consider other storage options, that is something to
keep in mind.

On Sun, Dec 2, 2018, 7:17 AM Arjen van der Meijden <acmmailing@tweakers.net
wrote:

> I'd think it depends on your application.
>
> If its a web-application and you're generating html, it may be better
> for the (client side) performance to have those images load via a
> webserver that can directly access the images as files (altough you
> could generate the images inline with base64). If its some application
> that has to load and display an image itself, than having easy control
> over the entire document will likely outweigh most potential advantages
> of a second database.
>
> Btw, Lucene can be considered a NoSQL-storage ;) If you really do get
> milllions of documents, it may be interesting to store them elsewhere if
> otherwise the database gets too large (but see Uwe's reply for ways to
> reduce the storage-overhead).
>
> Best regards,
>
> Arjen
>
> On 2-12-2018 10:20, Joe MA wrote:
> > Greetings,
> >
> > I have an index where I import documents such as powerpoint, PDF, and so
> forth.  One nice feature I added is that for each document, I store a
> thumbnail of the first page as an encoded String (uuencode) using a
> stored,not-indexed field.  This thumbnail gets displayed when the user
> finds a document.
> >
> > I am wondering if, as the size of the index grows to perhaps hundreds of
> thousands if not millions of documents,  how efficient is this?  Is it a
> good idea?
> > These encoded strings could be several hundred bytes in size, and of
> course are completely unique for each file indexed, and provide no 'search'
> value.  On the surface, it seems like there could be a better way to do
> this given the size, as well as the extra retrieval time for Lucene to pull
> these fields for found documents.
> >
> > Since I also have a unique hash for each document in the index, it would
> not be too difficult to set up a separate, independent NoSQL key/value
> store with the thumbnail images, such as MongoDB or similar, and then
> retrieve the thumbnails from that store instead of keeping them in the
> Lucene index.  Does this seem like a better approach? Or is Lucene stored
> field retrieval efficient enough that there would be no benefit to doing
> this?  Any other ideas?
> >
> > Thanks in advance,
> > J
> >
> >
> >
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message