corinthia-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From jan i <>
Subject Re: Checking malloc success and adding perror()
Date Mon, 23 Feb 2015 19:25:04 GMT
On 23 February 2015 at 12:47, Edward Zimmermann <>

> Hi,
> Been sort-of out the the discussion-- was on vacation last week-- so
> excuse me, in advance, if I bring up a point already made.
hi again, nice to hear from you again.

> First of all.. Corthinia is supposed to be C++? If so we don't want to use
> malloc. If its plain C, of course, malloc is probably our first choice for
> memory allocation.
No DocFormats (the library part) is strictly C99. The application of top
can be other languages.

> With the issue of x= malloc(y). This gets more complicated. Linux, for
> example, adopted from AIX an overbooking model of memory allocation. They
> literally sell more tickets than they have (they call it optimistic, I call
> it !#@?) and hope that when the page address is ready to be touched there
> is enough free memory one can map in. When Linux runs really out of
> memory--- and that is not just memory but pointers (in 32-bit Linux I can
> easily demonstrate how with little effort I can awaken the OOM Killer to
> wreck havoc) via kernel segment maps.
> AIX in all fairness had a memory low signal: SIGDANGER (Signal 33). Well
> designed AIX programs trap this signal to be able to try to recover
> sufficient pages to continue (if not it’s the same kind of seemingly random
> process sniping we see in Linux)... SIGDANGER in Linux? Nope.
> Generally in Linux the call malloc(x) can return a non-null pointer even
> when it has no memory. We can configure the kernel to do things a bit
> differently via the "vm.memory_overcommit" variable. See
we could, but I do not think we want to inerfere wih kernel parameters.

> Can we configure the OOM Killer to be a little nicer? Yes but really only
> process by process.. And when a process gets killed it's done so in a very
> silent way.. so if someone is "not in the know" to spot the clues.. it can
> get quite mysterious why some programs stop working...
> But let us even pretend that we get a NULL... What is the watermark to
> have gotten it? Can we recover? Any buffers we can quickly dispose of?
> Recovery is not that easy!
You are some steps ahead, right now we replace malloc with xmalloc to have
a central place for error handling.

> Should we pretend that we can get a NULL? Of course. It's good programming
> practice. Should we wrap malloc with an xmalloc for such testing? No. On
> systems where malloc might  return a NULL we should have for different
> objects alternative strategies for dealing with an allocation failure. A
> routine, for example, that wants to create a scratch buffer of x length but
> could work, albeit slower, with less we might make smaller. Etc etc. etc.
of course we need to check for that, try on a Android or IoS system to load
a huge documents, and you will most surely reach the limit.

> Optimizing memory pools? If we have a lot of object of the exact same size
> it's the standard paradigm. The most costly thing with most mallocs is not
> the malloc but the free. Being able to keep a pool around saves a lot of
> time. Its downside, however, is that the memory gets hogged and not
> returned generally until process end. We can also talk about the downsides
> to garbage collection models etc. etc...

> Windows since version 7 has a standard allocator with excellent
> performance. Current generation Linux, Solaris, BSD etc. too have gotten
> pretty good mallocs "out of the box" in their libc these days. Unless we
> are doing some hardcode mode allocations I don't think we should bother
> with pools. These mallocs are pretty darn fast. Yes I can write a special
> malloc with its own strategy that might provide better performance--- my
> search engine got its own tuned malloc--- but I don't think that is
> something Corithia needs (we don't I think need to worry about timer ticks).
in windows malloc only works well when you allocate in chunks of 4k (NTFS
size). But both in windows and linux calling both malloc and free will
cause a context switch.

IoS and Andriod work with preemptive context switching so here it is even
more expensive.

> I'd suggest we keep to malloc and IF NEEDED-- and only if and when
> needed--- we use a drop-in replacement (and chances are that we'll NEVER
> need it much less want one).
Right now we want to replace the malloc calls with xmalloc, so we do not
need tons of "if NULL" distributed in the code. Replacing malloc is a
second discussion.

> Part of the problem is that we might have different "best" approaches for
> different operating systems. IOS, Android, Linux, BSD, ... making "best"
> not really the "best" goal..
Well with my kernel knowledge, all of them benefit from us allocating a
chunk during startup, eating more if needed, and freeing it all when we are
finished. Please remember the typical use will be open a docment, do
something, save a document and stop the application.

> Perror?  No.  Calling directly a function that is intended to write to a
> console is, in general, a bad thing.
Why do you see that as bad ? I thought it wrote to stderr which can be
redirected, but anyhow what do you suggest instead.

jan i.

> -----Ursprüngliche Nachricht-----
> Von: Peter Kelly []
> Gesendet: Donnerstag, 19. Februar 2015 13:41
> An:
> Betreff: Re: Checking malloc success and adding perror()
> > On 19 Feb 2015, at 7:06 pm, Dennis E. Hamilton <>
> wrote:
> >
> > +1 about a cheap check and common abort procedure for starters.
> >
> > I think figuring out what to do about cleanup and exception unwinding,
> and even what exception handling to use (if any) is a further
> platform-development issue that could be masked with simple
> still-inlineable code, but needs much more architectural thought.
> I’m fine with us using wrapper functions for these which do the checks -
> though please let’s use xmalloc, xcalloc, xrealloc, and xstrdup instead of
> DFPlatform* (it’s nothing to do with platform abstraction, and these names
> are easier to type). (as a side note we can probably cut down on prefix
> usage a lot as long as we don’t export symbols; this was just to avoid name
> clashes with other libraries)
> In my previous mail I really just wanted to point out that by itself, this
> doesn’t really solve anything - the issue is in reality far more
> complicated than a simple NULL pointer check.
> I can think of two ways we could deal with the issue of graceful handling:
> 1) Allow the application to supply a callback, as Jan suggested
> 2) Adopt a “memory pool” type strategy where we create an memory pool
> object at the start of conversion which tracks all allocations that occur
> between the beginning and end of a top-level API call like DFGet, and put
> setjmp/longjmp-style exception handling in these API calls.
> The second approach is in fact already used to a limited extent with the
> DOM API. Every document maintains its own memory pool for storing Node
> objects (and the text values of nodes)… this is freed when the document’s
> retainCount drops to zero. I did this because it was much faster than
> traversing through the tree and releasing nodes individually (at least in
> comparison to have nodes as Objective C objects - the ObjC runtime was
> undoubtedly part of that overhead).
> —
> Dr Peter M. Kelly
> PGP key: <>
> (fingerprint 5435 6718 59F0 DD1F BFA0 5E46 2523 BAA1 44AE 2966)

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message