Return-Path: Delivered-To: apmail-lucene-java-user-archive@www.apache.org Received: (qmail 80920 invoked from network); 6 Jun 2006 09:11:40 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (209.237.227.199) by minotaur.apache.org with SMTP; 6 Jun 2006 09:11:40 -0000 Received: (qmail 69245 invoked by uid 500); 6 Jun 2006 09:11:34 -0000 Delivered-To: apmail-lucene-java-user-archive@lucene.apache.org Received: (qmail 69217 invoked by uid 500); 6 Jun 2006 09:11:33 -0000 Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: java-user@lucene.apache.org Delivered-To: mailing list java-user@lucene.apache.org Received: (qmail 69206 invoked by uid 99); 6 Jun 2006 09:11:33 -0000 Received: from asf.osuosl.org (HELO asf.osuosl.org) (140.211.166.49) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Jun 2006 02:11:33 -0700 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests= X-Spam-Check-By: apache.org Received-SPF: pass (asf.osuosl.org: local policy) Received: from [209.97.205.201] (HELO mail.seseit.com) (209.97.205.201) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Jun 2006 02:11:31 -0700 Received: by mail.seseit.com (Postfix, from userid 65534) id E33DB187B3; Tue, 6 Jun 2006 09:11:09 +0000 (GMT) Received: from max (spc1-lewi6-0-0-cust524.asfd.broadband.ntl.com [81.103.210.13]) by mail.seseit.com (Postfix) with ESMTP id BA4CF187B1 for ; Tue, 6 Jun 2006 09:11:07 +0000 (GMT) From: "Rob Staveley (Tom)" To: Subject: Avoiding java.lang.OutOfMemoryError in an unstored field Date: Tue, 6 Jun 2006 10:11:10 +0100 MIME-Version: 1.0 X-Mailer: Microsoft Office Outlook, Build 11.0.6353 In-Reply-To: <0B1C92228C5C8D45AA7450F78B3FB9DE790761@staf-exm-01.ardentia.co.uk> Content-Type: multipart/signed; protocol="application/x-pkcs7-signature"; micalg=SHA1; boundary="----=_NextPart_000_00D8_01C68951.88294480" Thread-Index: AcaJIaW8gyX3KzksSEafb8g1wOK2ewAHS2DgAAIMpGA= X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.2869 Message-Id: <20060606091107.BA4CF187B1@mail.seseit.com> X-Spam-Checker-Version: SpamAssassin 3.0.3 (2005-04-27) on mini.seseit.net X-Spam-Level: X X-Virus-Checked: Checked by ClamAV on apache.org X-Old-Spam-Status: No, score=1.7 required=5.0 tests=MSGID_FROM_MTA_ID autolearn=no version=3.0.3 X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N ------=_NextPart_000_00D8_01C68951.88294480 Content-Type: text/plain; charset="US-ASCII" Content-Transfer-Encoding: 7bit Sometimes I need to index large documents. I've got just about as much heap as my application is allowed (-Xmx512m) and I'm using the unstored org.apache.lucene.document.Field constructed with a java.io.Reader, but I'm still suffering from java.lang.OutOfMemoryError when I index some large documents. Are org.apache.lucene.document.Field and org.apache.lucene.document.Document always loaded entirely in memory? ------=_NextPart_000_00D8_01C68951.88294480 Content-Type: application/x-pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIKaDCCAj0w ggGmAhEAzbp/VvDf5LxU/iKss3KqVTANBgkqhkiG9w0BAQIFADBfMQswCQYDVQQGEwJVUzEXMBUG A1UEChMOVmVyaVNpZ24sIEluYy4xNzA1BgNVBAsTLkNsYXNzIDEgUHVibGljIFByaW1hcnkgQ2Vy dGlmaWNhdGlvbiBBdXRob3JpdHkwHhcNOTYwMTI5MDAwMDAwWhcNMjgwODAxMjM1OTU5WjBfMQsw CQYDVQQGEwJVUzEXMBUGA1UEChMOVmVyaVNpZ24sIEluYy4xNzA1BgNVBAsTLkNsYXNzIDEgUHVi bGljIFByaW1hcnkgQ2VydGlmaWNhdGlvbiBBdXRob3JpdHkwgZ8wDQYJKoZIhvcNAQEBBQADgY0A MIGJAoGBAOUZv22jVmEtmUhx9mfeuY3rt56GgAqRDvo4Ja9GiILlc6igmyRdDR/MZW4MsNBWhBiH mgabEKFz37RYOWtuwfYV1aioP6oSBo0xrH+wNNePNGeICc0UEeJORVZpH3gCgNrcR5EpuzbJY1zF 4Ncth3uhtzKwezC6Ki8xqu6jZ9rbAgMBAAEwDQYJKoZIhvcNAQECBQADgYEATD+4i8Zo3+5DMw5d 6abLB4RNejP/khv0Nq3YlSI2aBFsfELM85wuxAc/FLAPT/+Qknb54rxK6Y/NoIAK98Up8YIiXbix 3YEjo3slFUYweRb46gVLlH8dwhzI47f0EEA8E8NfH1PoSOSGtHuhNbB7Jbq4046rPzidADQAmPPR cZQwggNiMIICy6ADAgECAhAL2gsXwT+JjqsJdHq0zi4zMA0GCSqGSIb3DQEBAgUAMF8xCzAJBgNV BAYTAlVTMRcwFQYDVQQKEw5WZXJpU2lnbiwgSW5jLjE3MDUGA1UECxMuQ2xhc3MgMSBQdWJsaWMg UHJpbWFyeSBDZXJ0aWZpY2F0aW9uIEF1dGhvcml0eTAeFw05ODA1MTIwMDAwMDBaFw0wODA1MTIy MzU5NTlaMIHMMRcwFQYDVQQKEw5WZXJpU2lnbiwgSW5jLjEfMB0GA1UECxMWVmVyaVNpZ24gVHJ1 c3QgTmV0d29yazFGMEQGA1UECxM9d3d3LnZlcmlzaWduLmNvbS9yZXBvc2l0b3J5L1JQQSBJbmNv cnAuIEJ5IFJlZi4sTElBQi5MVEQoYyk5ODFIMEYGA1UEAxM/VmVyaVNpZ24gQ2xhc3MgMSBDQSBJ bmRpdmlkdWFsIFN1YnNjcmliZXItUGVyc29uYSBOb3QgVmFsaWRhdGVkMIGfMA0GCSqGSIb3DQEB AQUAA4GNADCBiQKBgQC7WkSKBBa7Vf0DeootlE8VeDa4DUqyb5xUv7zodyqdufBou5XZMUFweoFL uUgTVi3HCOGEQqvAopKrRFyqQvCCDgLpL/vCO7u+yScKXbawNkIztW5UiE+HSr8Z2vkV6A+Hthzj zMaajn9qJJLj/OBluqexfu/J2zdqyErICQbkmQIDAQABo4GwMIGtMA8GA1UdEwQIMAYBAf8CAQAw RwYDVR0gBEAwPjA8BgtghkgBhvhFAQcBATAtMCsGCCsGAQUFBwIBFh93d3cudmVyaXNpZ24uY29t L3JlcG9zaXRvcnkvUlBBMDEGA1UdHwQqMCgwJqAkoCKGIGh0dHA6Ly9jcmwudmVyaXNpZ24uY29t L3BjYTEuY3JsMAsGA1UdDwQEAwIBBjARBglghkgBhvhCAQEEBAMCAQYwDQYJKoZIhvcNAQECBQAD gYEAAn2eb0VLOKC43ulTZCG85Ewrjx7+kkCs2Ao5aqEyISwHm6tZ/tJiGn1VOLA3c9z0B2ZjYr3h U3BSh+eo2FLpWy2q4d7PrDFU1IsZyNgjqO8EKzJ9LBgcyHyJqC538kTRZQpNdLXu0xuSc3QuiTs1 E3LnQDGa07LEq+dWvovj+xUwggS9MIIEJqADAgECAhB1uB0L6Y5EVCCaaM0jkP2tMA0GCSqGSIb3 DQEBBQUAMIHMMRcwFQYDVQQKEw5WZXJpU2lnbiwgSW5jLjEfMB0GA1UECxMWVmVyaVNpZ24gVHJ1 c3QgTmV0d29yazFGMEQGA1UECxM9d3d3LnZlcmlzaWduLmNvbS9yZXBvc2l0b3J5L1JQQSBJbmNv cnAuIEJ5IFJlZi4sTElBQi5MVEQoYyk5ODFIMEYGA1UEAxM/VmVyaVNpZ24gQ2xhc3MgMSBDQSBJ bmRpdmlkdWFsIFN1YnNjcmliZXItUGVyc29uYSBOb3QgVmFsaWRhdGVkMB4XDTA1MTAwNDAwMDAw MFoXDTA2MTAxODIzNTk1OVowggEZMRcwFQYDVQQKEw5WZXJpU2lnbiwgSW5jLjEfMB0GA1UECxMW VmVyaVNpZ24gVHJ1c3QgTmV0d29yazFGMEQGA1UECxM9d3d3LnZlcmlzaWduLmNvbS9yZXBvc2l0 b3J5L1JQQSBJbmNvcnAuIGJ5IFJlZi4sTElBQi5MVEQoYyk5ODEeMBwGA1UECxMVUGVyc29uYSBO b3QgVmFsaWRhdGVkMTMwMQYDVQQLEypEaWdpdGFsIElEIENsYXNzIDEgLSBOZXRzY2FwZSBGdWxs IFNlcnZpY2UxGzAZBgNVBAMUElJvYiBTdGF2ZWxleSAoVG9tKTEjMCEGCSqGSIb3DQEJARYUcnN0 YXZlbGV5QHNlc2VpdC5jb20wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCm4C+s/DP/ NIW0DrHFueJPsZAZxTzi8Aw2bXOUQbHRDNlSO7nWoDPlkTs5RQl0tsJUDlB5TubjJGWKGBdRePXq ueK/8yj23cBTzKRTU2gNG6cmmc/f/HwHcuc4MCpW9+okvFCPeaFNzuBPPvegjewz3BL3ewfjeOKr ly//fr/8JBVW19EgK070C2COZgegYNQ5NeV4Y4oDjqETADQDDnYuTe7EMBuSMysmOSgkCfVzEG4h 8N+1ImW6hWe+bH6Rify+q9eZAQint7sSFGZv3ZbBgmE654FKrvE7O8BOOP807/yaJUBlpDkF/X8G UwvYNwhH/zbHN1BSCD3tstcSIUGHAgMBAAGjgcswgcgwCQYDVR0TBAIwADBEBgNVHSAEPTA7MDkG C2CGSAGG+EUBBxcDMCowKAYIKwYBBQUHAgEWHGh0dHBzOi8vd3d3LnZlcmlzaWduLmNvbS9ycGEw CwYDVR0PBAQDAgWgMB0GA1UdJQQWMBQGCCsGAQUFBwMEBggrBgEFBQcDAjAUBgpghkgBhvhFAQYH BAYWBE5vbmUwMwYDVR0fBCwwKjAooCagJIYiaHR0cDovL2NybC52ZXJpc2lnbi5jb20vY2xhc3Mx LmNybDANBgkqhkiG9w0BAQUFAAOBgQCtsGmCX3v78Qn3I5DAo9mJDenolmgmBjmlaSJr1e5nSBLv CEflpx3FaZuirDQNJ/sTxtUCJhlv/kTLuczeJkSDj1ALhqeYH1GX9T5zFY+3GCQqTJcTIarHSqZC PQzwv3dwZCRE0fuELb6oobi4kGA5PXbu3+P7upHSBAmN81O00TGCBL8wggS7AgEBMIHhMIHMMRcw FQYDVQQKEw5WZXJpU2lnbiwgSW5jLjEfMB0GA1UECxMWVmVyaVNpZ24gVHJ1c3QgTmV0d29yazFG MEQGA1UECxM9d3d3LnZlcmlzaWduLmNvbS9yZXBvc2l0b3J5L1JQQSBJbmNvcnAuIEJ5IFJlZi4s TElBQi5MVEQoYyk5ODFIMEYGA1UEAxM/VmVyaVNpZ24gQ2xhc3MgMSBDQSBJbmRpdmlkdWFsIFN1 YnNjcmliZXItUGVyc29uYSBOb3QgVmFsaWRhdGVkAhB1uB0L6Y5EVCCaaM0jkP2tMAkGBSsOAwIa BQCgggKyMBgGCSqGSIb3DQEJAzELBgkqhkiG9w0BBwEwHAYJKoZIhvcNAQkFMQ8XDTA2MDYwNjA5 MTEwOVowIwYJKoZIhvcNAQkEMRYEFPzAX6Gjn2pgmCB5QaiN3/uznSAbMGcGCSqGSIb3DQEJDzFa MFgwCgYIKoZIhvcNAwcwDgYIKoZIhvcNAwICAgCAMA0GCCqGSIb3DQMCAgFAMAcGBSsOAwIHMA0G CCqGSIb3DQMCAgEoMAcGBSsOAwIaMAoGCCqGSIb3DQIFMIHyBgkrBgEEAYI3EAQxgeQwgeEwgcwx FzAVBgNVBAoTDlZlcmlTaWduLCBJbmMuMR8wHQYDVQQLExZWZXJpU2lnbiBUcnVzdCBOZXR3b3Jr MUYwRAYDVQQLEz13d3cudmVyaXNpZ24uY29tL3JlcG9zaXRvcnkvUlBBIEluY29ycC4gQnkgUmVm LixMSUFCLkxURChjKTk4MUgwRgYDVQQDEz9WZXJpU2lnbiBDbGFzcyAxIENBIEluZGl2aWR1YWwg U3Vic2NyaWJlci1QZXJzb25hIE5vdCBWYWxpZGF0ZWQCEHW4HQvpjkRUIJpozSOQ/a0wgfQGCyqG SIb3DQEJEAILMYHkoIHhMIHMMRcwFQYDVQQKEw5WZXJpU2lnbiwgSW5jLjEfMB0GA1UECxMWVmVy aVNpZ24gVHJ1c3QgTmV0d29yazFGMEQGA1UECxM9d3d3LnZlcmlzaWduLmNvbS9yZXBvc2l0b3J5 L1JQQSBJbmNvcnAuIEJ5IFJlZi4sTElBQi5MVEQoYyk5ODFIMEYGA1UEAxM/VmVyaVNpZ24gQ2xh c3MgMSBDQSBJbmRpdmlkdWFsIFN1YnNjcmliZXItUGVyc29uYSBOb3QgVmFsaWRhdGVkAhB1uB0L 6Y5EVCCaaM0jkP2tMA0GCSqGSIb3DQEBAQUABIIBACqBlGa0PH3eApnNENjUI3VZw9HIC0Syk4md Jz57E6M5LIaJa504x6wknzHMgWwEG7d1bI3yH+eciWXfwx5twjIJ4lcxkVt7K1LpwbH1UBOH/mR0 ECuqU3PpPYERq1JHi+PO7FQx69Nphb2Uap4bO9y7k+CHursHjC4bu6dHKhPV4fusieIrGeVznD2i YKzoPmPhJGXA663QAnMC029bc+g04v/73Jz8/330SD0HoE18NdsDr8nFgM2FnqT1yBJ+02g4BiMH QFTqd+oYL4yEzDF2Q0HVpmPd/2IEkwbFdyqeNRIzFvBIjMtQpZPGRGvtQMT6aRuyerECqnrmfvcy 4swAAAAAAAA= ------=_NextPart_000_00D8_01C68951.88294480--