ponymail-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From s...@apache.org
Subject incubator-ponymail git commit: Bug: MboxoReader needs to return all data for a read() call
Date Tue, 03 Oct 2017 14:58:52 GMT
Repository: incubator-ponymail
Updated Branches:
  refs/heads/master 6768e751e -> 500827ada


Bug: MboxoReader needs to return all data for a read() call

This fixes #415

Project: http://git-wip-us.apache.org/repos/asf/incubator-ponymail/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-ponymail/commit/500827ad
Tree: http://git-wip-us.apache.org/repos/asf/incubator-ponymail/tree/500827ad
Diff: http://git-wip-us.apache.org/repos/asf/incubator-ponymail/diff/500827ad

Branch: refs/heads/master
Commit: 500827ada8e9a4852313749360437c3d1ef2ede1
Parents: 6768e75
Author: Sebb <sebb@apache.org>
Authored: Tue Oct 3 15:58:47 2017 +0100
Committer: Sebb <sebb@apache.org>
Committed: Tue Oct 3 15:58:47 2017 +0100

----------------------------------------------------------------------
 CHANGELOG.md         |  1 +
 tools/import-mbox.py |  9 +--------
 tools/mboxo_patch.py | 10 ++++++----
 3 files changed, 8 insertions(+), 12 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/incubator-ponymail/blob/500827ad/CHANGELOG.md
----------------------------------------------------------------------
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 07620eb..6fbee88 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,4 +1,5 @@
 ## CHANGES in 0.10:
+- Bug: MboxoReader needs to return all data for a read() call (#415)
 - Bug: import-mbox should not check database during a dry run (#413)
 - Archiver loses some message information (#402)
 - Mail parsing can change source spacing (#410)

http://git-wip-us.apache.org/repos/asf/incubator-ponymail/blob/500827ad/tools/import-mbox.py
----------------------------------------------------------------------
diff --git a/tools/import-mbox.py b/tools/import-mbox.py
index 9c50100..834a8f4 100755
--- a/tools/import-mbox.py
+++ b/tools/import-mbox.py
@@ -312,14 +312,7 @@ class SlurpThread(Thread):
                     # so the source agrees with the summary info
                     if message.__class__.__name__ == 'MboxoFactory':
                         file=MboxoReader(file)
-                    # Temporary work-round for #415 until MboxoReader is fixed
-                    buf=bytearray()
-                    while True:
-                        tmp=file.read()
-                        if len(tmp) <= 0:
-                            break
-                        buf.extend(tmp)
-                    raw_msg=bytes(buf)
+                    raw_msg=file.read()
                     file.close()
                     if args.dups:
                         try:

http://git-wip-us.apache.org/repos/asf/incubator-ponymail/blob/500827ad/tools/mboxo_patch.py
----------------------------------------------------------------------
diff --git a/tools/mboxo_patch.py b/tools/mboxo_patch.py
index f1de613..1349e7e 100644
--- a/tools/mboxo_patch.py
+++ b/tools/mboxo_patch.py
@@ -57,14 +57,16 @@ class MboxoReader(mailbox._PartialFile):
         # get the next chunk, resetting if necessary 
         if self.remain != 0:
             super().seek(whence=1, offset=-self.remain)
+        # if size is None or negative, then read returns everything.
+        # in which case there is no need to wory about matching across reads
+        limited_read = size and size >= 0
         # ensure we get enough to match successfully when refilling
-        # size can be None; assume large enough if so
-        if size:
-            size = size if size > FROM_MANGLED_LEN else FROM_MANGLED_LEN
+        if limited_read and size < FROM_MANGLED_LEN:
+            size = FROM_MANGLED_LEN
         bytes = super()._read(size, read_method)
         bufflen=len(bytes)
         # did we get anything new?
-        if bufflen > self.remain:
+        if limited_read and bufflen > self.remain:
             # is there a potential cross-boundary match?
             if bytes.endswith(FROMS):
                 # yes, work out what to keep


Mime
View raw message