Building Argus: A Zero-Copy Data Pipeline in C++

What I’m Building

I’m starting work on a new project called Argus Engine. It’s a zero-copy data pipeline built for Windows (because you use the OS you have, not the OS you wish you had) using modern C++23.

The goal is to process high-volume, continuous data streams with deterministic, sub-millisecond latency. Standard IPC methods like sockets or named pipes usually involve copying payloads back and forth across the kernel boundary, which kills performance when you are trying to push gigabytes of data per second. The core aim of this project is to bypass the kernel entirely, allowing the compute engine to directly read the producer’s memory space.

The Architecture Plan

To pull this off, the system is split into two isolated processes:

1. Process A (The Producer): This acts as the hardware simulator. It generates raw data and writes it directly into a pre-allocated shared memory region.
2. Process B (The Consumer): This is the compute engine. It reads that exact same memory block and processes the data using SIMD hardware acceleration.

Because both processes map the same physical RAM pages into their address space, we completely avoid syscalls and memory allocations on the hot path.

What’s Done So Far: Shared Memory

The first step was getting Windows to actually map a single memory block across two separate processes. I just finished writing the core WindowsSharedMemory class to handle this.

Here is what the initialization looks like:

std::expected<WindowsSharedMemory, SharedMemoryError>
WindowsSharedMemory::Initialize(const std::wstring& name, size_t size, Mode mode) noexcept
{
    void* rawHandle = nullptr;

    if (mode == Mode::Create)
    {
        const ULARGE_INTEGER mappingSize{.QuadPart = size};

        rawHandle = ::CreateFileMappingW(
            INVALID_HANDLE_VALUE, nullptr, PAGE_READWRITE, mappingSize.HighPart, mappingSize.LowPart, name.c_str());

        if (rawHandle && ::GetLastError() == ERROR_ALREADY_EXISTS)
        {
            ::CloseHandle(rawHandle);
            return std::unexpected(SharedMemoryError::AlreadyExists);
        }
    }
    else
    {
        rawHandle = ::OpenFileMappingW(FILE_MAP_ALL_ACCESS, FALSE, name.c_str());
    }

    if (!rawHandle)
    {
        return std::unexpected(SharedMemoryError::MappingCreationFailed);
    }

    UniqueHandle handle(rawHandle);

    void* rawMappedView = ::MapViewOfFile(handle.get(), FILE_MAP_ALL_ACCESS, 0, 0, size);

    if (!rawMappedView)
    {
        return std::unexpected(SharedMemoryError::ViewMappingFailed);
    }

    UniqueMapView mappedView(rawMappedView);

    return WindowsSharedMemory(std::move(handle), std::move(mappedView), size);
}

Once initialized, the consumer process just calls .GetView() to get a std::span<std::byte>, which points directly to the mapped memory.

I also set up Google Test and wrote a few quick tests to make sure the mappings actually work.

What’s Next: The Lock-Free Ring Buffer

Now that both processes can read and write to the same memory, they need a way to synchronize. Process A needs to signal to Process B when a new frame is ready to be read, without causing data races. Using standard mutexes or locks would force an OS context switch, completely ruining the latency target.

So, the next thing I will be working on is the LockFreeRingBuffer. I plan to build a wait-free Single-Producer Single-Consumer (SPSC) queue.

You can check out the code and follow the progress on GitHub here: Krish2882005/Argus-Engine