Improve bigmem tests

[serhiy-storchaka]

BPO	31695
Nosy	@pitrou, @vstinner, @serhiy-storchaka

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2017-10-04.19:22:19.577>
labels = ['type-feature', 'tests']
title = 'Improve bigmem tests'
updated_at = <Date 2017-10-07.15:45:09.628>
user = 'https://github.com/serhiy-storchaka'

bugs.python.org fields:

activity = <Date 2017-10-07.15:45:09.628>
actor = 'pitrou'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Tests']
creation = <Date 2017-10-04.19:22:19.577>
creator = 'serhiy.storchaka'
dependencies = []
files = []
hgrepos = []
issue_num = 31695
keywords = []
message_count = 2.0
messages = ['303729', '303885']
nosy_count = 3.0
nosy_names = ['pitrou', 'vstinner', 'serhiy.storchaka']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue31695'
versions = []

[serhiy-storchaka]

I think that the process of running bigmem tests should be improved.

1. Every bigmem test should run in separate process. It's effect on memory fragmentation shouldn't affect other tests.

2. Only one bigmem test should be run at the same time. Otherwise several parallel bigmem tests can exhaust RAM and lead to swapping. In multiprocessing mode the process running a bigmem test should send a signal to other processes, wait until they finished their current tests and paused, run a bigmem test, and send a signal to other processes that they can continue.

3. The hard limit of addressed memory should be set for every bigmem test, according to the declared requirements. It is better to crash one test and report the failure, than hang on swapping.

4. All other tests should be run with the limit set to 2 GiB (or 1 GiB?). This will help to find memory consuming tests which should be made bigmem tests.

5. The maxrss of the process running a bigmem test should be reported in verbose mode after finishing a test.

[pitrou]

+1 to all this :-)

[cmaloney]

Currently bigmemtest in verbose mode only reports memory usage on Linux by way of reading /proc. I do a lot of work on macOS and would like to be able to see quickly the memory usage there. Hoping to add Windows and macOS support to help improve bigmemtest. With that also fix 5 above, printing the maxrss after finishing each test.

To support that I'd like to add a private C extension function that uses platform-specific APIs to get process memory usage and update _MemoryWatchdog / memory_watchdog.py to use that information in its print. That makes it so the platform-specific code lives entirely inside C conditional compilation.

The code currently pre-opens the /proc file and passes that through as stdin to reduce PID recycling issues if the parent exits. Unfortunately there is no cross-platform pidfd type mechanism to make this really simple. Instead the new code relies on a pipe to the parent to determine "has the parent been lost" which lets us detect parent process termination and prevent issues if the PID is recycled. It also means we should exit more quickly when the parent intentionally exits and gives a good point to add the end of test print.

[vstinner]

This issue has not been touched for 9 years, I suggest you opening a new issue for your idea. We can add a private function in a test module such as _testcapi. I didn't understand the pidfd issue, I should see the code to understand :-)