Constructing realtime simulations engine: breaking shared memory limits in high-level scripting languages

In critical applications, particularly in areas of Realtime Simulations, usages of Unix-based Shared Memory (Shared Memory) concepts help greatly in managing the system’s resources, to be called into play at any moment by the other subsystems. The fact that it is memory-based makes inter-process transactions very fast, as befitting the requirements of critical systems. Coupled with some semi-low-level compiled languages like C/C++ that incorporates the Unix Shared Memory easily, Shared Memories formed the very basis of most Realtime Simulations Engines. In modern times however, both the computer Operating System (OS) as well as programming languages have evolved: the former to cater more for distributed processes and global internet communications, while the latter into more rapid proto-typing as well as very high-level hierarchy in the man-machine interface, which resulted in many, many new languages being used, the most popular of which would be Python. In the OS side, issues about integrity, safety and privacy drive OS manufacturers to do away with many would-be leeways in which security issues (or lack thereof) could creep in, and the usage of Shared Memories would be the primeval talking point. In short, while operating systems like Windows and macOS still support shared memory mechanisms (e.g., POSIX SHM, memory-mapped files), these are limited to processes on a single host. Creating true shared-memory spaces across multiple networked systems is no longer natively supported. In this paper, we investigate the constraints of Python’s Shared Memory model in distributed contexts and, realizing of its futility in fully utilizing it, propose four practical alternatives to overcome the hurdles, eventually opting one of them after being convinced that it is the nearest in behaviour to the Shared Memory model. This work serves as a guide for developers and engineers facing similar constraints in using Python for distributed high-frequency data sharing.