Since malloc and sbrk are allocating VM, I think there should be a mechanism, that kindly signals processes to free memory, before getting into a severe OOM situation. Not as a reaction of acute memory demands, but a kernel process, that constantly monitors the system and every now and then nudges the processes.

A good criterion for that would be the ratio of a process allocated pages / pages recently accessed. For garbage collected, but also leaking processes this number may grow rather quickly. In the case of a process, which actually freed up memory in that situation, the kernel could give bonus points, not to kill it in OOM. It could also track statistics about the average of the additionally mapped pages during memory deallocation, the fewer the better.

I am pro-Linux, but that any of the kernel developers defend this OOM mechanism is an embarrassment. It’s so obviously wrong to arbitrarily kill processes by any heuristic. Even if the heuristic were improved, there’s no reason (ever) to be killing off applications which were behaving correctly. Sounds like that was designed by a banker, doesn’t it?

Process badness?? Who are we kidding here, this is pure kernel badness. This is a glaring flaw on an otherwise reliable platform.

The solution should address the real problems:
1. The system should never over-commit. Not enough memory to fulfill a new memory request, then fail *deterministically* where applications can deal with it.

2. If this prevents a process from forking, then so be it. At least the system remains stable, and the failed request can be handled gracefully.

3. Maybe the fork/exec combination is fundamentally flawed. Even when it works it’s not particularly efficient, this could be an opportunity to improve the system calls.

4. Some feel that over-commit is desirable. I personally disagree that this is acceptable in any production setting, but if it’s necessary then it should be explicit. The kernel must always honor the memory contracts between itself and processes, no exceptions. If an application wants/needs to over commit ram it probably won’t use, then let it do so explicitly and at it’s own risk.

Conceptually a modified fork call could keep the parent safe from overcommit, and allow it’s child to over commit until it runs exec, the new executable could do whatever it pleases with regards to overcommit, but by default all processes have the right to expect the kernel will honor it’s obligations.

I simply cannot believe my ears when i hear Linux kernel hackers debating OOM process killer selection heuristics rather than how to actually fix the problem.

Your syscall idea is like an mlock() that reserves either or both of RAM and swap instead of only RAM, or like a MAP_RESERVE flag for Linux’s mmap() to compliment the MAP_NORESERVE flag. I think there’s some merit to that suggestion.

The only other thing I’d note is that heap implementations typically give each thread its own arena so as to minimize lock contention on SMP systems. Each of them would need an independent amortized syscall; what I’m getting at is that if this were a workable solution in general, /proc/sys/vm/overcommit_memory==2 would be the default.

Of course, it’s probably a perfectly workable default for people running Postgres

Log Buffer #171

Knows how much the application is using, or knows how much malloc has told the application that it can use?

I thought the problem was VM space versus actually allocated memory.

If, in addition to allocating VM space, malloc explicitly asked the OS to reserve memory (say, every 1024 pages or so to amortize the cost of a syscall) before returning a valid pointer, then the OS would have an opportunity to say “there’s no way I can reserve those pages for you”, and malloc could return NULL.

I understand there are still COW issues with fork and dlopen, but I’m not looking for a perfect solution. In those cases the OS has a pretty good idea how much it has extended itself (number of processes sharing that page that might try to write).

The main problem is with malloc, where (as you say) there is such a huge disconnect between the VM size and the memory that the application has requested that the OS doesn’t know what kind of trouble its in early enough, and even if it does, malloc has no way of knowing the OS may not be able to satisfy the requests.