The Subaru DENSO head unit studied in this article is a system made up of multiple operating systems and dedicated hardware: a QNX Hypervisor, QNX Neutrino and Linux run on an Intel x86 platform, and the applications depend on graphics, audio, networking and an authentication chip. Debugging directly on such a production device runs into the problem of debugging conditions first: access to a shell, serial port and debugger is usually restricted, and a service crash or system reboot may interrupt normal use of the head unit. To repeatedly observe programs, compare configurations and collect crash scenes, a lab environment independent of the car is much more convenient.

Some of the accompanying scripts used in this article are at https://github.com/sparkcyf/subaru-denso-tools.git

QEMU/KVM gives us a way to move the firmware onto a computer and run it there. This environment can be built up layer by layer around the research question: to study a network service in Linux, run the factory kernel and root filesystem first and control the program over serial; when a hardware dependency such as the authentication chip appears, add an experimental stand-in at the corresponding interface; to observe QNX boot, drivers and virtual machine management, fill in the outer environment. The screen, GPU, audio and vehicle bus each have their own drivers and dependencies and can be connected as the research goal requires. This article proceeds through the three stages below:

StageWhat bootsMain questions to observe
Standalone LinuxFactory Linux kernel and ext4 rootfsUser-space services, network requests, process state and crash scenes
QNX bootA trimmed QNX host image and the necessary driversBoot scripts, serial, device discovery and storage access
Nested LinuxQNX Hypervisor under QEMU/KVM, with qvm booting LinuxVirtualization capability, guest configuration and the inner boot process

We start from a community-shared 2024 update package, work through unpacking, image identification and booting, and then connect logging, core dumps and debugging tools to this environment. What takes the most time is often a runtime condition that the original platform provides by default and that only becomes visible after the move to a virtual machine: a set of kernel parameters, a local network interface, the write-back length of an interface function, or the virtualization capability the host provides. Each section gives the cleaned-up procedure first, then reviews a few typical failures: how to find clues in logs and call stacks, and how to confirm a judgement with later output.

The article is aimed at readers who are familiar with PVE, QEMU or ordinary Linux virtual machines and are just starting with embedded firmware; concepts such as IFS, interface stubs and nested virtualization are explained as they come up.

1. First, Understand What We Are Running

1.1 The Head Unit’s Software and Hardware Architecture

This article uses a Subaru DENSO CP1.5-series head unit on an Intel x86 platform; Apollo Lake boot code and drivers can be found in the firmware. The software consists of QNX Hypervisor/QNX Neutrino together with Linux: QNX handles low-level boot, drivers and some user-space services, and qvm creates a virtual machine in that environment to run the Linux kernel and its applications. The debugging that follows unfolds along this relationship:

Intel platform of the real head unit
└── QNX Hypervisor / Neutrino
    ├── QNX boot, drivers and user-space services
    └── qvm
        └── Linux guest
            ├── Linux 4.9.52
            ├── ext4 rootfs
            └── HMI, networking and other application services

Getting a few names straight first will make reading the files much clearer later. The Linux distribution in this sample reports ADIT embedded Linux Platform / Genivi6, the name we use in this article; on the QNX side, Neutrino 7 refers to the runtime environment and QNX6 to the on-disk filesystem format. The update package’s FHI-sd.ifs also provides a QNX environment that can serve as an auxiliary lab environment, from which we extract the generic serial driver and PCI module below.

1.2 We Can Start by Taking Linux Out of Its Original Virtual Machine

If the goal is to observe a particular service inside Linux, we can swap in a different layer of virtual hardware:

Linux development machine
└── QEMU (TCG or KVM)
    └── Factory Linux kernel + factory ext4 rootfs

This Linux kernel already ran inside a virtual machine and already includes virtio block and network drivers. QEMU can provide matching interfaces so the factory kernel keeps using the factory rootfs, and we can first debug services, inspect networking and analyze crashes in a familiar Linux environment. When the work turns to QNX drivers, qvm or the host boot flow, we add the outer layer, forming the nested relationship below:

Intel physical development machine / PVE (L0)
└── QEMU + KVM
    └── QNX Hypervisor (L1)
        └── qvm
            └── Linux (L2)

The first route is good for quickly setting up a user-space experiment; the second lets us observe how host and guest run together. This article follows that order too, and the image identification, serial and crash-collection methods from the first half are still used in the second.

1.3 Restore What Debugging Needs, When It Needs It

Watching a TCP service depends mainly on the kernel, dynamic linker, shared libraries, runtime directories and network interfaces; display, audio, camera and vehicle bus can be added step by step as the research goal requires. Graphics, for example: this head unit uses Apollo Lake graphics hardware and the GVT-g interface, and getting a picture also involves drivers, graphics libraries and application configuration, which is work worth a separate article. We first control the system through the serial port and judge service state through normal network requests, and only when the execution path actually hits a hardware dependency do we choose between adding a driver, adjusting configuration or writing an interface stub.

Running firmware in a different execution environment is usually called firmware rehosting. Its practical value is that the emulation scope can be arranged around a specific question: to study a protocol, first make the protocol service accept input; to study boot, fill in the corresponding drivers and scripts. For this goal-driven way of rebuilding an environment, see Jetset’s research; automotive firmware has work such as MetaEmu.

2. Preparing the Public Sample and a Lab Directory

2.1 Where the Sample Comes From

Readers can start from the community-shared June 2024 update directory, also listed under the June 2024, CP1.5 update entry on the Gen6 Firmware page. The directory’s FM1WxM073-980.zip contains content for different head unit configurations; this article uses the ALLUPD-NAVI.pkg inside it, whose groups are version FM1WMM073-980. Completing the steps below with this public sample builds the base lab environment described here.

After downloading, record the file size and hash, then confirm the group version with the unpacking tool. The fingerprint of the local sample in this article is below, which you can use to pick the same baseline; with a different version, compare item by item along image type, kernel version and boot configuration:

File name: ALLUPD-NAVI.pkg
Size: 4,667,449,176 bytes
SHA-256: c95d1b431d56747fcdb9f6e06ebdee6d8e83bcce3bdc3729c12fc63d8b48b9ab
Group version: FM1WMM073-980

2.2 Host Environment

The first half is written for an x86-64 Linux environment on Debian 12; package names on Ubuntu are usually similar. Install:

sudo apt-get update
sudo apt-get install \
  qemu-system-x86 qemu-utils python3 file binutils e2fsprogs fdisk less \
  gdb socat unzip gcc libc6-dev make git

Set aside at least 25 GiB of free disk space. Unpacking the update package, exporting images and saving crash scenes all take space; if you keep several complete image copies, reserve more.

Record the environment:

uname -a
qemu-system-x86_64 --version
qemu-img --version
ls -l /dev/kvm

My earliest Linux experiments used TCG in QEMU 7.2, completing kernel boot and service debugging inside a development virtual machine. TCG executes guest instructions in software, which is good for getting the flow working first; KVM uses processor virtualization to execute same-architecture guests and runs faster. In both cases QEMU provides the memory layout, devices and external interfaces. The first half uses TCG, so readers can start in an existing Linux environment; the second half, where QNX creates another virtual machine, moves to an Intel KVM host that meets the nested virtualization requirements.

2.3 File Layout

Put the article’s scripts/ and examples/ in the lab directory of your choice, then run:

export LAB="$HOME/subaru-qemu-lab"
mkdir -p "$LAB"
cd "$LAB"
mkdir -p originals pkg work logs exports tools

The commands below assume the host’s current directory is $LAB. originals/ holds downloaded files, pkg/ holds the raw contents extracted from the package, and work/ holds lab artifacts.

Unzip the downloaded ZIP and find the package:

unzip -l originals/FM1WxM073-980.zip
unzip originals/FM1WxM073-980.zip -d originals/update
find originals/update -name ALLUPD-NAVI.pkg -print

If several packages are listed, choose the one for the Mid head unit whose internal version matches the baseline above. Assign the path to a variable, for example:

export PACKAGE="$LAB/originals/update/your-actual-directory/ALLUPD-NAVI.pkg"
sha256sum "$PACKAGE" | tee logs/package.sha256

Only that path line needs to be adjusted to the ZIP’s actual directory.

3. Extracting the Kernel and Filesystem from the Update Package

3.1 List the Directory First, Then Unpack

ALLUPD-NAVI.pkg is a container with a big-endian index. It contains group names, group versions, file lengths and file offsets. We provide a parser that depends only on the Python standard library:

python3 scripts/unpack_pkg.py "$PACKAGE" > logs/package-index.json
python3 -m json.tool logs/package-index.json | less

After confirming the group name and version, extract:

python3 scripts/unpack_pkg.py "$PACKAGE" --extract pkg \
  > logs/package-extract.json
python3 scripts/verify_hashlistex.py pkg/*/HASHLISTEX.DAT

The second script reads HASHLISTEX.DAT from the package, recomputes each file’s CRC-32 and compares it with the record, so extraction errors or corrupted content show up before boot. After that, focus on these files first; other voice data and application resources can be brought in later as needed:

FileTypeUse in the lab
pkg/NAVIHVOS/bzImageLinux x86 boot kernelKernel input shared by standalone Linux and nested Linux
pkg/NAVILRFS/Linux_rfs.imgext4 filesystem imageLinux’s /
pkg/NAVIQRFS/QNX_rfs.imgQNX6 filesystem imageAuxiliary QNX files and user-space content
pkg/NAVIHVOS/hvImage_abl_emmc.imgext2 filesystem imageContains the QNX host boot files inside
pkg/NAVIHVOS/linux.config_*qvm text configurationShows how the original platform arranges the Linux guest
pkg/NAVIVR/Linux_vr.imgext4 filesystem imageVoice data; not connected in this minimal experiment

3.2 Identify the Image Layers

Run:

file pkg/NAVIHVOS/bzImage \
  pkg/NAVILRFS/Linux_rfs.img \
  pkg/NAVIQRFS/QNX_rfs.img \
  pkg/NAVIHVOS/hvImage_abl_emmc.img
fdisk -l pkg/NAVILRFS/Linux_rfs.img
dumpe2fs -h pkg/NAVILRFS/Linux_rfs.img

Although these files all carry a .img suffix, their contents sit at different layers. Linux_rfs.img starts at the beginning of an ext4 filesystem, so it can be QEMU’s first virtio disk directly: in the guest, the whole device /dev/vda carries the root filesystem, hence root=/dev/vda in the boot arguments. hvImage_abl_emmc.img, by contrast, is an ext2 filesystem on the outside; the boot file must first be extracted from it, and then its wrapper and ELF structure handled. Once the relationship among update container, whole disk, partition filesystem, boot image and directory archive is clear, it becomes easy to decide whether a file goes to QEMU, a mounting tool or an unpacking program.

3.3 Reading the Kernel and Distribution Information

An ext4 image can be read directly with debugfs:

debugfs -R 'cat /etc/os-release' pkg/NAVILRFS/Linux_rfs.img

This sample contains:

ID="mel-automotive-adit"
NAME="ADIT embedded Linux Platform - Genivi6  - MGC drop COS2"
VERSION_ID="2017.02"

file bzImage shows the kernel version 4.9.52-i915-gvt_qnx-gdecfd5b, and the qvm configuration provides the virtio devices, the root device and the original kernel command line. Together they explain how this kernel expects to be booted again: drivers for virtio-blk, virtio-net, ext4 and the serial console are all built into the kernel, so the root filesystem can be found and mounted directly during boot and this article can skip an extra initramfs. The same check applies to other firmware: first determine whether the drivers needed for boot are in the kernel or in modules; on a module-based system, initramfs loads them before the root is mounted.

4. Stage One: Booting the Factory Linux Kernel and Rootfs Directly

4.1 The First Boot Command

On the host:

qemu-system-x86_64 \
  -machine pc -accel tcg -cpu max -m 2048 -smp 2 \
  -kernel pkg/NAVIHVOS/bzImage \
  -drive file=pkg/NAVILRFS/Linux_rfs.img,format=raw,if=virtio,snapshot=on \
  -append 'root=/dev/vda rootfstype=ext4 rw rootwait console=ttyS0,115200 init=/bin/sh virtionet=a0:b0:c0:d0:e0:f1;eth_inter;10.0.2.15;255.255.255.0' \
  -netdev user,id=n0,hostfwd=tcp:127.0.0.1:17000-:7000 \
  -device virtio-net-pci,netdev=n0,mac=a0:b0:c0:d0:e0:f1 \
  -display none -serial stdio -monitor none -no-reboot

Expect Linux boot output first, then a shell. You may see:

/bin/sh: can't access tty; job control turned off

This is the current PID 1 shell’s message about terminal job control. The kernel has already mounted the root filesystem and executed /bin/sh; we can start typing commands and filling in the runtime environment. With the boot result in hand, look at the main arguments in the command:

ArgumentWhy it is needed
-kernel / -appendProvides the kernel and kernel command line directly, skipping boot media
if=virtio / root=/dev/vdaChooses a virtual disk interface and root device the kernel already supports
init=/bin/shRuns a shell after mounting root, so the environment can be filled in item by item
console=ttyS0,115200Connects the kernel console to the virtual serial port
snapshot=onPuts writes to this disk in a temporary layer, discarded on exit
-netdev userProvides QEMU user-mode networking, skipping bridge and TAP setup
hostfwd=...Forwards the host’s 127.0.0.1:17000 to TCP 7000 in the guest

With the root disk set to snapshot=on, writes from this run go into a temporary layer and QEMU can start again from the same baseline after exiting. A second disk with persistent writes is added later for logs and cores, giving system state and experiment output separate places to live. For how the kernel, root disk and command line work together, see the QEMU Direct Linux Boot documentation.

4.2 Retrospective: Why Did Adding a NIC Make the Kernel Panic?

The command above already includes the virtionet= argument determined through debugging. My early command provided only the root device, the serial port and init=/bin/sh, and let QEMU add a virtio NIC. To anyone used to ordinary Linux images, that configuration is natural: get into a shell, then configure the interface and address with ip. But this kernel panics during network initialization, before leaving a chance to configure anything by hand.

The most useful information here is the phase and call chain where the panic happens. The program has not reached user space yet; the failure-related calls on the serial port come down to:

do_setup_if → ic_setup_if_routes → mac_pton(NULL)

mac_pton converts a text MAC address into bytes, and here it receives a null pointer. Looking upward from it, the kernel is running some network auto-configuration logic. That clue points at how the original machine supplies its network configuration: in the package’s qvm configuration we found virtionet=, a vendor-added kernel command line interface that passes the MAC, interface name, IPv4 address and netmask together:

virtionet=<MAC>;<interface name>;<IPv4 address>;<netmask>

The original platform’s boot flow always passes this set of values, and the custom code carries that assumption over; when we took the kernel out on its own, this dependency was left behind. Restoring the argument, matching its MAC to the QEMU NIC and moving the address to the lab’s 10.0.2.0/24 subnet lets the kernel get past the original failure point and reach a shell. What fixed it here was the boot configuration; the factory kernel can keep being used.

This experience also changed how I read firmware configuration. Besides describing how devices are connected, the qvm configuration file records the input the vendor kernel expects to receive. Comparing its command line item by item against the minimal boot command is often an easier way to find clues than trying different virtual hardware first. When switching to a different kernel build later, I make the same comparison first and then decide which arguments to keep.

4.3 Complete the Minimal Boot Environment

The commands in the rest of this section run in the guest shell:

export PATH=/sbin:/usr/sbin:/bin:/usr/bin
mount -t proc proc /proc
mount -t sysfs sysfs /sys
mkdir -p /var/volatile/tmp /var/volatile/log /run /data/tel
ip link set lo up

These commands fill in the base environment that init and the boot scripts used to provide: /proc and /sys give tools process, device and network state, the volatile directories hold temporary files and logs, and loopback supports local communication. In this sample /tmp points to /var/volatile/tmp, so the target directory has to be created as well. If proc or sysfs is already mounted by some other init step, check the current state with mount and carry on.

init=/bin/sh puts us directly in charge of PID 1, which makes it convenient to start services by hand and observe the result of each step. The service dependencies, process reaping and shutdown management the original init handled have to be filled in as the experiment requires; when the work reaches systemd units, service restarts or D-Bus, restore the corresponding boot flow then. First confirm the kernel, root disk and file versions with the commands below:

uname -a
cat /proc/cmdline
cat /etc/os-release
mount
ip addr show
sha256sum /bin/carplayd /lib64/libcarplay-plugin-core.so

Depending on the build, the shared libraries may actually live in /lib64, /lib or /usr/lib64; ls and the program’s load results will confirm it. Save versions and hashes with the lab log, so the matching program and libraries can be found when analyzing a core later. Stage one is now complete: the factory kernel booted, the root filesystem is mounted, and the shell can read system information; attention can move to user-space services.

5. Stage Two: Getting a Factory Service to Actually Handle Requests

5.1 Start with a Single Service

I use carplayd from the firmware as the example. Here we send only a normal OPTIONS request, to check whether the service can handle network input.

In the guest:

export XDG_RUNTIME_DIR=/tmp
export LD_LIBRARY_PATH=/usr/lib64
/bin/carplayd >/tmp/carplayd.log 2>&1 &
SERVICE_PID=$!
echo "$SERVICE_PID"
sleep 2
tail -40 /tmp/carplayd.log

The program can simply be run directly at startup; the -v here prints the version and exits, and detailed runtime state is observed from logs and process information. The log will show errors about missing camera, media nodes and so on, which we can judge against the current goal: first check whether the process keeps running and whether the network listener is established, then follow the dependencies that actually block the service. Each environment adjustment then has a corresponding symptom and a way to verify it.

5.2 Retrospective: A CoreFoundation Crash That Ended at a Network Command

Once the kernel reaches a shell, failures move to user space. In one early run, carplayd exited during startup with the call stack landing in CoreFoundation’s CFRunLoopAddSource. On its own, that location looks like a bad event loop object or call argument; but we had just taken over the original init’s work, many conditions the boot scripts used to prepare were not restored yet, and a crash inside a library can also be the consequence of an earlier initialization failure.

Following the initialization process further back, the cause became concrete: CFRunLoopEnsureInitialized tries to bind a local 127.0.0.1 address, loopback not being up makes that step fail, CFRunLoopGetCurrent() then returns NULL, and later code still passes that result to CFRunLoopAddSource. Laying out the sequence behind this failure shows how the environment gap reached the final crash site:

loopback not yet up
  → CFRunLoop initialization fails to bind 127.0.0.1
  → CFRunLoopGetCurrent() returns NULL
  → later CFRunLoopAddSource uses the empty object and the process crashes

What is worth checking here is the guest’s own local networking. QEMU’s port forwarding and virtio NIC handle requests from the host, while the 127.0.0.1 used inside the program depends on the guest’s lo interface. Running ip addr show lo in the guest checks this condition; adding the earlier ip link set lo up and starting the service again makes the initialization crash disappear. Only then can the listener and normal requests be checked.

The fix was one command, but locating it required following the crash stack back to where the object was created. With init=/bin/sh we took over the original boot scripts’ responsibilities: temporary directories, proc/sysfs and loopback all have to be prepared item by item. They look like small details of environment setup, but the program treats them as given. Section 6 fills in ways to observe process state, system calls and cores, so a similar failure can be followed from “the process exited” down to the specific operation that failed.

5.3 Confirm the Listener First, Then Check Port Forwarding

Check in the guest:

pidof carplayd
grep -i ':1B58' /proc/net/tcp
iptables -L INPUT -n -v

1B58 is 7000 in hex and 0A in /proc/net/tcp means LISTEN; if ss -lntp is available, it gives more readable output directly. Once the listener exists, look at the firewall: this sample’s rules distinguish the wireless interface from the internal interface, and the lab NIC is called eth_inter, so a matching accept rule is needed for this entry point. Add this in the guest:

iptables -I INPUT 1 -i eth_inter -p tcp --dport 7000 -j ACCEPT

Back on the host, in another terminal:

cd "$LAB"
python3 scripts/probe_rtsp.py --wait 30

Expect a response beginning with:

RTSP/1.0 200 OK

The service needs initialization time between the process appearing and the listener being established; --wait 30 retries the normal request during that window until a valid response arrives. The request passes through the host loopback port, QEMU forwarding, the guest NIC, the firewall, the listening socket and the application’s handling logic, and the RTSP 200 response connects the whole path. It is also more useful than checking a bare TCP connect: QEMU can accept the connection first, whereas an application-layer response further confirms that the service in the guest has started processing input. Stage two ends here, with the factory service accepting ordinary requests from the host.

5.4 Retrospective: Standing In for the MFi Authentication Chip, Starting with the ABI

With the OPTIONS response received, the basic network path is up. Only when watching session initialization does another layer of dependency appear: the CarPlay service in the car accesses the MFi authentication chip through libipod-auth.so, asking it for a certificate and signature, with I²C/GPIO underneath. The QEMU virtual PC environment lacks this chip, so the authentication requests in early experiments returned 403 Forbidden. The addresses, interfaces and firewall worked out earlier are enough to get requests to the service; what is missing this time is the hardware interface the application calls.

This dependency has a clear boundary in user space: the dynamically exported functions MFiPlatform_Initialize, MFiPlatform_CopyCertificate, MFiPlatform_CreateSignature and so on. We can implement a shared library around these functions as an experimental stand-in for the hardware interface, a stub. Specifying it through LD_PRELOAD at startup makes the dynamic linker resolve the corresponding calls with its same-named functions first. The program still gets return values at the original call sites, and the environment we need to prepare shrinks to this set of interfaces:

Original call: CarPlay service → MFiPlatform_* → libipod-auth → I²C/GPIO → authentication chip
Lab call:      CarPlay service → same-named MFiPlatform_* stub → placeholder certificate and signature

The stub at the time returned the appropriate status for initialization and teardown, allocated buffers for the certificate and signature interfaces, and returned pointers and lengths through output parameters. The lab client skips validating the authenticity of the certificate and signature, so placeholder data lets us observe the service flow past the hardware interface; a normal CarPlay client also validates this data. When writing a stub, buffer ownership must match the original interface as well, because the caller still frees it after use.

The first version of the stub brought a new crash. The output content was in the buffer and the length appeared to be written back, but the later cryptographic processing read an unusually large length. Checking the original function’s disassembly showed the two easy-to-confuse facts: the length field read inside the function is 16 bits, but it zero-extends the value and then writes it to the caller-provided output address as 64 bits. The stub wrote only the low two bytes, the upper six bytes kept their old contents, and the caller reading the full 64 bits got a different number.

With 128 bytes as the example, on this little-endian x86-64 environment the memory left by the two versions looks like:

Writing only 16 bits: 80 00 ?? ?? ?? ?? ?? ??
Writing all 64 bits:  80 00 00 00 00 00 00 00

The fix changes the output length pointers in both stub function definitions to uint64_t *, so *out_len = ... in the implementation writes all eight bytes. Summarized as function declarations, the change is:

- int MFiPlatform_CopyCertificate(void **out_ptr, uint16_t *out_len);
+ int MFiPlatform_CopyCertificate(void **out_ptr, uint64_t *out_len);
- int MFiPlatform_CreateSignature(const void *data, uint64_t data_len, void **out_ptr, uint16_t *out_len);
+ int MFiPlatform_CreateSignature(const void *data, uint64_t data_len, void **out_ptr, uint64_t *out_len);

With the write width fixed, the huge length and out-of-bounds crash produced by the stub disappeared. This bug sits some distance from the final crash site: the interface returned six bytes short, and the code downstream kept using it as normal output. Checking a stub therefore means reading the caller and the original function together, confirming argument widths, pointer levels, memory ownership and return semantics. The conventions involved here are usually called the ABI (application binary interface); even with the same function names and seemingly correct return values, these conventions decide whether a stub works with the original program.

The tutorial’s normal OPTIONS request can already be done with the original program; this MFi retrospective corresponds to the work of bringing in a hardware dependency further along. Next we return to the service that already answers requests and give it stable serial, logging and crash-collection channels.

6. Giving the Debug Environment Observation and Output Channels

6.1 Detach the Serial Port from the Current Terminal

On the first boot, an interactive terminal lets us see output and try commands directly; once experiments repeat, it is more convenient to hand the serial port to a host program that sends commands, reads exit codes and saves the full log. First stop the service started earlier in the guest and sync the filesystem, then restart QEMU with a different serial connection:

kill "$SERVICE_PID"
sync

With the current -serial stdio command, Ctrl+C in the host terminal ends QEMU, and the root disk returns to its original state along with the temporary write layer. On restart, put the serial port on a TCP socket and the monitor on a Unix socket:

qemu-system-x86_64 \
  -machine pc -accel tcg -cpu max -m 2048 -smp 2 \
  -kernel pkg/NAVIHVOS/bzImage \
  -drive file=pkg/NAVILRFS/Linux_rfs.img,format=raw,if=virtio,snapshot=on \
  -append 'root=/dev/vda rootfstype=ext4 rw rootwait console=ttyS0,115200 init=/bin/sh virtionet=a0:b0:c0:d0:e0:f1;eth_inter;10.0.2.15;255.255.255.0' \
  -netdev user,id=n0,hostfwd=tcp:127.0.0.1:17000-:7000 \
  -device virtio-net-pci,netdev=n0,mac=a0:b0:c0:d0:e0:f1 \
  -chardev socket,id=console,host=127.0.0.1,port=17001,server=on,wait=off,logfile=logs/linux-serial.log \
  -serial chardev:console \
  -monitor unix:work/linux-monitor.sock,server=on,wait=off \
  -display none -no-reboot

No output in the QEMU foreground for now is normal: the serial port has moved to another connection. Open another host terminal:

cd "$LAB"
python3 scripts/serial_cmd.py 'echo GUEST_READY'
python3 scripts/serial_cmd.py 'cat /proc/version'
tail -f logs/linux-serial.log

serial_cmd.py appends a random completion marker and $? to each command, and returns only after receiving the exit code expanded from the marker. That distinguishes terminal echo from command completion: the echo keeps the text that was sent, while only shell execution produces a marker with a concrete exit code. This approach suits running shell commands that return, one at a time; a timeout tells us to look further in the serial log. For interactive input, switch to socat to take over the terminal:

socat STDIO,raw,echo=0,escape=0x1d TCP:127.0.0.1:17001

Press Ctrl+] to exit socat and hand the serial port back to serial_cmd.py. This connection lets interactive and scripted work take turns on the same entry point, and keeps one unified serial log during debugging.

6.2 The Monitor and the Serial Port Are Two Different Channels

Connect the monitor:

socat - UNIX-CONNECT:work/linux-monitor.sock

At its prompt, enter:

info status
info cpus
info registers

The two channels serve different purposes: the serial port reaches the guest’s shell, good for reading process, file and network state; the monitor reaches QEMU, good for looking at virtual CPUs and whole-machine state. Even when the guest’s shell stops responding, we can still read registers from the monitor, pause execution or end the VM with quit. The above uses the interactive HMP; automated control can use the structured QMP interface instead, and whichever is chosen takes commands in that interface’s format.

6.3 Add a Disk Dedicated to Saving Experiment Output

We want the root disk to return to its original state each time, while logs, tools and cores stay. The easiest way to understand is to add a second disk:

/dev/vda  temporary write layer for the original Linux rootfs
/dev/vdb  lab output disk, holding tools, logs and cores

End QEMU first. On the host, prepare a practice program with debug symbols and the output disk:

mkdir -p work/share-seed/bin work/share-seed/cores
gcc -static -g -O0 -o work/share-seed/bin/abort-demo examples/abort-demo.c
truncate -s 512M work/share.img
mkfs.ext4 -F -O ^metadata_csum_seed,^orphan_file \
  -d work/share-seed work/share.img

This makes an ext4 filesystem with Debian 12’s e2fsprogs and disables the newer metadata_csum_seed and orphan_file features so the output disk suits the guest’s old kernel. -d fills the filesystem directly from a directory’s contents, which makes preparation convenient; with another e2fsprogs version, check which feature names it supports first. If mounting reports a format or feature error, there is a clear direction to investigate.

The disk’s abort-demo is a small program written for this exercise: it keeps its debug symbols and deliberately raises SIGABRT, so the full core generation process can be observed. It is statically linked here, packing the runtime libraries the exercise needs into the program so it runs directly in the guest’s old user-space environment. Then add the output disk after the first -drive in the previous QEMU command:

  -drive file=work/share.img,format=raw,if=virtio \

This disk uses persistent writes, so the program and cores stay on it. After boot, repeat the initialization from Section 4.3, then mount it in the guest:

mkdir -p /mnt/share
mount -t ext4 /dev/vdb /mnt/share
ls /mnt/share/bin

6.4 Collect a Core Dump and Restore the Call Stack

In the guest:

mkdir -p /mnt/share/cores
ulimit -c unlimited
echo '/mnt/share/cores/core.%e.%p.%t' > /proc/sys/kernel/core_pattern
cp /mnt/share/bin/abort-demo /tmp/abort-demo
/tmp/abort-demo
ls -lh /mnt/share/cores
sync

Running it prints the program’s marker and the SIGABRT exit message and creates a core.abort-demo.* file. These steps lay out crash collection clearly: ulimit sets the core size limit for child processes started by the current shell, core_pattern specifies the output location and file name, and the program is copied to the guest’s /tmp to run. When debugging factory services later, start them from a configured shell as well so the service inherits the limits. After collection, sync and unmount the output disk in the guest:

sync
umount /mnt/share

Then quit in the QEMU monitor. Once the output disk is no longer in use by QEMU, read it on the host:

debugfs -R 'ls -l /cores' work/share.img
debugfs -R 'rdump /cores exports' work/share.img
find exports -type f -name 'core.abort-demo.*' -print

Pass the file just produced to GDB:

gdb work/share-seed/bin/abort-demo exports/cores/actual-core-file-name

In GDB:

bt
thread apply all bt
info registers
frame 2
info args
list

In this re-run, frame 2 of bt was finish_experiment(marker=2024). After finding that frame, info args reads the arguments and list shows the corresponding source; frame numbers can differ between builds, so pick by your own backtrace. At this point “run, collect the crash scene, export, analyze offline” forms a complete flow; switching to a factory program mainly adds the preparation of saving the matching executable, dynamic linker and shared libraries. The original ext4 image can be mounted read-only as GDB’s analysis environment:

mkdir -p work/linux-root
sudo mount -t ext4 -o ro,noload,loop \
  pkg/NAVILRFS/Linux_rfs.img work/linux-root

Point set sysroot at this directory in GDB, then load the program and core, and the debugger can interpret the crash scene with runtime libraries from the same build. The ext4 mount options here include noload, which keeps the read-only analysis at the image’s journal state at the time. The whole data flow follows the same order: the guest writes and syncs, the output disk is unmounted, QEMU is ended, and the host reads it; each step has a clear filesystem user.

6.5 Judge Service Health from Process State and Protocol Responses

In the end I usually look at several indicators together:

ObservationQuestion it answers
QEMU process presentIs the outer emulator still running
Guest serial command returns an exit codeCan guest user space still schedule and run commands
Target PID and restart recordsHas the service exited, has it been restarted
LISTEN socketHas the service established a listening port
Normal request and valid responseDoes the current application path really work
Thread state, logs and coreWhere a timeout or exit happened

For example, record in the guest:

pidof carplayd
ps -T -p "$(pidof carplayd)"
cat /proc/"$(pidof carplayd)"/status
ls -l /proc/"$(pidof carplayd)"/fd

Combining this information shows which layer a failure is in. The status and wchan of each thread under /proc/<pid>/task/ supplement ps output, showing whether a thread waits on a socket, waits on some other event, or the process has already exited. To find which files the program accesses and what connections it makes, stop the current instance and start it under strace:

strace -f -tt -s 128 -e trace=network,file \
  -o /tmp/carplayd.strace /bin/carplayd

-f follows child execution contexts, -tt adds timestamps, and the network and file calls let the system call trail show what the program is looking for and where it connects. For log entries that all look like a startup failure, the call records can further separate a missing device file, a shared library load failure and a connection timeout. For repeatable experiments, a simple supervisor can restart the service automatically and record each start time, PID and exit status; matching those records with request times identifies the crashes and recoveries the service went through.

6.6 Process Debugging and Kernel Debugging

Where GDB connects depends on the execution context to observe. To debug a user-space process, use GDB in the guest, or put in a gdbserver compatible with the target ABI and glibc and connect from the host; the debugger then presents threads and address space around a process. To debug kernel boot or driver initialization, connect to the gdbstub QEMU provides and observe execution from the virtual CPU’s point of view. Adding these two items to the QEMU boot arguments makes it pause and wait for GDB:

  -gdb tcp:127.0.0.1:1234 -S \

Connect from the host:

gdb
set architecture i386:x86-64
target remote 127.0.0.1:1234
info registers
info threads
continue

-S makes the CPU wait before starting execution, so continue is needed after connecting to advance boot. This entry point shows virtual CPU registers and execution state, and the threads in info threads mainly correspond to vCPUs; with guest-matched ELF and symbol information, kernel function names become locatable. The compressed bzImage used for boot handles the initial loading, while source-level debugging usually uses the corresponding vmlinux. My own work located problems mainly through serial output, process state and cores; the gdbstub is an entry point for going deeper into boot and driver issues, and its usage is covered in the QEMU GDB documentation.

7. The Second Route: First Get QNX Itself Running in QEMU

7.1 IFS, startup and the Disk rootfs

The Linux boot above can be understood as the kernel finding a block device, mounting the ext4 root, then executing init. QNX instead packs startup, the kernel, boot scripts and the first batch of drivers and programs into an IFS (Image Filesystem), loaded into memory early. The IFS establishes the base runtime environment, then starts other drivers and mounts later filesystems; the QNX6 rootfs in the update package is part of that on-disk content. Understanding the order between the two makes it easy to see where programs and data come from when reading boot scripts; the composition is described in QNX’s IFS documentation. First, on the host, extract the boot file from the outer ext2:

debugfs -R 'ls -l /' pkg/NAVIHVOS/hvImage_abl_emmc.img
debugfs -R 'dump /hvImage_abl_emmc work/hvImage.wrapped' \
  pkg/NAVIHVOS/hvImage_abl_emmc.img

The inner file in this sample has a 36-byte ipk. wrapper. Check the marks first, then write out the ELF:

python3 - <<'PY'
from pathlib import Path
data = Path('work/hvImage.wrapped').read_bytes()
assert data[:4] == b'ipk.', '不是本文识别过的外层包装'
assert data[0x24:0x28] == b'\x7fELF', '预期位置不是 ELF,请重新检查样本'
Path('work/hv-original.elf').write_bytes(data[0x24:])
PY
file work/hv-original.elf
readelf -h -l work/hv-original.elf

readelf shows an ELF32 boot wrapper with Multiboot information; startup then takes over to enter 64-bit QNX. The wrapper describes the entry point and load segments to the loader, while the boot code arranges the OS’s later execution mode. We first hand the extracted original ELF to QEMU and see how far it gets.

7.2 Confirm QNX’s Boot Progress from the CPU State

qemu-system-x86_64 \
  -machine pc -accel tcg -cpu max -m 8192 -smp 4 \
  -kernel work/hv-original.elf \
  -display none -serial stdio \
  -monitor unix:work/qnx-original-monitor.sock,server=on,wait=off \
  -no-reboot

This boot likely sits on a quiet terminal: the factory image expects its own storage and peripherals, and the serial port follows the car’s configuration too. Open another terminal and connect to the monitor to read CPU state; my early experiments showed execution had entered QNX’s high-address region and was then in HLT wait. That gives a useful progress point: the boot wrapper has run, and the next step is to connect QNX’s console and boot dependencies, then judge from user-space logs what it is waiting for. After ending this QEMU instance, we build a lab image with an interactive serial port.

7.3 Prepare the IFS Tools and Generic Drivers

This uses ifsdump from the open-source qnx_dumpers; the listing, extraction and expansion commands below are written for this tool. The QNX SDP also provides the official dumpifs; with another tool, the same operations can be done with its corresponding arguments. First install the dependencies, build the version used here and record the commit:

sudo apt-get install zlib1g-dev liblzo2-dev libucl-dev
git clone https://github.com/jtang613/qnx_dumpers.git tools/qnx_dumpers
git -C tools/qnx_dumpers checkout 68424a67efbcc0ff0f63e45223a530a0f3bde3fe
make -C tools/qnx_dumpers ifsdump
git -C tools/qnx_dumpers rev-parse HEAD > logs/qnx-dumpers-revision.txt

Get the uncompressed image and directory listing from the host file:

tools/qnx_dumpers/ifsdump work/hv-original.elf > logs/hv-ifs-list.txt
tools/qnx_dumpers/ifsdump -u work/hv-uncompressed.img \
  work/hv-original.elf

The generic 8250 serial driver and PCI module from the auxiliary FHI-sd.ifs are also needed. First mount the package’s QNX6 filesystem read-only:

sudo modprobe qnx6
mkdir -p work/qnx-root
sudo mount -t qnx6 -o ro,loop pkg/NAVIQRFS/QNX_rfs.img work/qnx-root
cp work/qnx-root/dos/FHI-sd.ifs work/FHI-sd.ifs
sudo umount work/qnx-root
tools/qnx_dumpers/ifsdump -x -d work/fhi work/FHI-sd.ifs

This step depends on the host kernel’s QNX6 filesystem module; the results of modprobe and mount help confirm the support. You can also extract the same FHI-sd.ifs in an existing QNX environment and come back to the host to expand it. Here it serves as a source of drivers, and its generic 8250 serial driver and PCI module happen to suit our virtual PC environment. Confirm the two extracted files:

file work/fhi/sbin/devc-ser8250 \
  work/fhi/lib/dll/pci/pci_hw-Intel_x86.so.2.0

7.4 Build a Host Image Dedicated to the Experiment

This lab image keeps the QNX kernel, qvm and the necessary drivers, brings up two legacy PC serial ports, and leaves the storage driver and Linux boot under our manual control. Graphics, audio and the car’s automatic mounting services are suspended for now, so each time a device is added or a program started, its effect can be observed directly. The accompanying script builds this environment from the files extracted above:

python3 scripts/prepare_qnx_host.py \
  --elf work/hv-original.elf \
  --uncompressed work/hv-uncompressed.img \
  --fhi-dir work/fhi \
  --kernel pkg/NAVIHVOS/bzImage \
  --config examples/linux-qvm.conf \
  --output work/hv-lab.elf

The script produces an ELF and a JSON change list recording replaced entries, lengths and file hashes, so the content used in this run can be checked later. When trying again, give the output a new name and keep the different configurations. The main adjustments are:

AdjustmentPurpose
Replace the original serial program content with generic devc-ser8250Use the legacy PC serial ports QEMU can provide
Use the auxiliary image’s generic Intel x86 PCI moduleReduce dependence on the APL-specific PCI module
Change the early script to establish a serial shellProvide an independent control channel for the lab
Suspend graphics, audio and factory auto-mount servicesFocus boot dependencies on the devices connected in this run
Put the Linux kernel and a trimmed qvm configuration into the lab IFSBoot the guest directly from a file in memory
Update sizes, compression marks, ELF load segments and imagefs checksumsMake the loader work with the modified image layout

The changes here are concentrated in driver selection, scripts and image layout; qvm stays original. An IFS organizes directory entries and file contents in one image, so editing has to take care of entries, offsets, sizes and checksums at the same time; the accompanying script therefore takes an approach specific to this sample: keep the original directory area, reuse the entries of components disabled in this run, append new files to the imagefs data area, and then update the related fields. The script checks the input structure and expected entries, and the change list lets us go back and verify each change. The result is a QNX host we have configured explicitly, convenient to observe and extend step by step.

8. QNX → Linux: Nested Virtualization in Practice

8.1 Switch to an Intel KVM Host That Meets the Requirements

Before the nested experiment, draw the layers the commands run in clearly. When QEMU starts directly from a physical PVE shell, PVE is L0, the QNX inside is L1, and the Linux started by qvm is L2; running the same command inside a Debian VM on the PVE adds another layer outside. SSH can reach any of these layers, so the deciding factor is the target machine’s runtime relationship. My available environment is a physical PVE on an Intel Xeon E5-2683 v4 with QEMU 11.0.3; record the following on that host first:

qemu-system-x86_64 --version
test -r /dev/kvm && test -w /dev/kvm && echo KVM_ACCESS_OK
cat /sys/module/kvm_intel/parameters/nested
cat /sys/module/kvm_intel/parameters/enable_apicv
cat /sys/module/kvm_intel/parameters/enable_shadow_vmcs

The last three items are Y in this environment and, together with the VMX capability visible to the CPU and KVM access permission, form the host conditions to compare when troubleshooting later. Boot with -enable-kvm -cpu host to expose the corresponding capabilities to QNX, and qvm creates the inner machine from them. For the specifics of nested, capability passing and the L0/L1/L2 relationship, see the Linux kernel documentation. Once the host is ready, copy the article’s accompanying directories, the original Linux rootfs and the generated hv-lab.elf over, keep the same $LAB layout, and verify the hashes after transfer.

8.2 Simplify Storage with a Virtual SD Card Holding Only the Linux rootfs

The car’s layout contains multiple filesystems and partitions, and restoring the whole layout brings extra dependencies. To verify only that qvm boots Linux, we use a smaller construction: one SD card whose byte zero is the start of the Linux ext4 rootfs, and QNX presents the whole card to Linux as a single virtio disk.

On the host:

qemu-img create -f qcow2 -F raw \
  -b "$LAB/pkg/NAVILRFS/Linux_rfs.img" work/linux-sd.qcow2
qemu-img resize work/linux-sd.qcow2 4G
qemu-img info --backing-chain work/linux-sd.qcow2

qcow2 uses the original ext4 image as a backing file and stores the changes this experiment makes, so the file is small when created and the underlying content still comes from the original image. Growing it to 4 GiB satisfies the QEMU SD card capacity requirement used here: once the virtual block device is larger, the original ext4 filesystem can still mount at its own size. When moving the lab directory, keep the qcow2 together with the original image and confirm the backing path resolves. In QNX this disk ends up as /dev/hda, qvm passes it to Linux, and the /dev/vda root disk appears later.

8.3 One QEMU, Two Serial Ports

Start on the Intel KVM host:

qemu-system-x86_64 \
  -machine pc -enable-kvm -cpu host -m 8192 -smp 4 \
  -kernel work/hv-lab.elf \
  -device sdhci-pci \
  -drive if=none,id=sd0,file=work/linux-sd.qcow2,format=qcow2,snapshot=on \
  -device sd-card,drive=sd0 \
  -chardev socket,id=qnxser,host=127.0.0.1,port=18001,server=on,wait=off,logfile=logs/qnx-host-serial.log \
  -serial chardev:qnxser \
  -chardev socket,id=linuxser,host=127.0.0.1,port=18002,server=on,wait=off,logfile=logs/qnx-linux-serial.log \
  -serial chardev:linuxser \
  -monitor unix:work/qnx-monitor.sock,server=on,wait=off \
  -display none -no-reboot

This command gives the outer QNX 4 vCPUs; the inner Linux’s CPU count is arranged separately by the qvm configuration later. The two serial ports also have owners: the first is for the QNX shell, the second carries the Linux console output, so the host can still be observed and operated while the inner system boots. Connect the first serial port and confirm QNX user space is available:

python3 scripts/serial_cmd.py --port 18001 'echo QNX_HOST_READY'

The minimal IFS keeps only a small set of commands, and the shell built-in echo is a good first check. Send the command after serial driver initialization completes; if it is sent too early and leading characters are lost during boot, send it again. Seeing the completion marker and exit code means the serial port, shell and command execution are connected; the second serial port starts producing Linux logs once qvm launches.

8.4 Make the Factory SD Driver Bind to the QEMU Controller

Run in the first QNX serial port:

devb-sdmmc blk cache=16m,maxio=1024,naming="a0#" \
  cam quiet,pnp disk maxio=1024 \
  sdio verbose=0 sdmmc verbose=0 &
waitfor /dev/hda
ls -l /dev/hda

With these arguments, the factory devb-sdmmc binds to QEMU’s sdhci-pci and creates /dev/hda for the root disk. Finding this command took some time, though. In early experiments the device node never appeared, and the driver kept reporting:

xpt_configure:  sdmmc SIM attach failure

This is an old driver for the Apollo Lake platform, and on a bind failure it is easy to suspect first that it does not support the virtual controller. Taking devb-virtio from the auxiliary QNX image and switching storage routes was also considered. But comparing the controller’s PCI identifiers against the boot script turned up a difference that could be tested first: the car’s command carries idx=0,vid=0x8086,did=0x5acc, while QEMU provides a different set of identifiers.

Controllervendor:device
Apollo Lake device filtered by the car’s script8086:5acc
QEMU’s sdhci-pci1b36:0007

These parameters restrict which controller the driver looks for. Keeping the car’s filter already tells the driver to accept only the car’s device; even if a compatible SDHCI controller exists on the PCI bus, it can be excluded during matching. After removing idx, vid and did and letting the driver scan on its own, the factory driver created the disk node. The comparison narrowed the problem to device filtering, and the original driver and QEMU SDHCI can keep being used.

One more small detail remains in the final command: sdio verbose=0. This argument parser requires the sdio section to carry at least one real option, otherwise the following sdmmc may be parsed as an option of sdio. The arguments in the original script look like ordinary strings, but they decide both device matching and the driver’s internal configuration sections; understanding them item by item is more reliable than copying the whole line.

Once the disk appears, confirm driver state with the device node and later reads. Many QNX drivers run as user-space processes and expose interfaces in the /dev namespace through resource managers; when a program establishes a background service, the shell job that started it may show Done. The observation order here is therefore: did the start command report an error, did /dev/hda appear, and can qvm read the inner root filesystem from it. The next step connects this disk to Linux.

8.5 Trim the qvm Configuration, Then Boot the Inner Linux

The accompanying examples/linux-qvm.conf has been placed in the host IFS as /scripts/linux.config; the core settings are:

cpu
sched 11r
runmask 2,3

load /scripts/bzImage
cmdline "root=/dev/vda rootfstype=ext4 rootwait rw init=/bin/sh console=ttyS0,115200n8"

vdev ser8250
    hostdev /dev/ser2
    loc io:0x3f8
    intr myioapic:4

vdev virtio-blk
    hostdev /dev/hda

The excerpt above covers the CPU, kernel, serial and disk settings; the run uses the complete accompanying configuration, which also arranges memory regions, ACPI, timers and interrupt controllers. One cpu block creates one vCPU; runmask 2,3 specifies which QNX CPUs that vCPU’s execution thread may run on. The outer environment provides 4 CPUs; the control shell is placed on the earlier CPUs and the later ones are left for the guest execution thread. The memory settings’ linuxram_low and linuxram_high are named regions established by this platform’s startup, which qvm obtains through QNX system information and which therefore correspond to this sample’s BSP.

Now the wiring between devices: hostdev /dev/ser2 connects Linux’s virtual UART to QNX’s second serial port, and hostdev /dev/hda makes the lab SD card the guest’s root disk. The inner machine consists of one disk, a serial port and basic timer/interrupt devices, enough to complete kernel and user-space boot. virtio-net can be added in a later networking experiment, bringing back the virtionet= argument and interface configuration from the first half.

In the first QNX serial port:

qvm @/scripts/linux.config &

Observe from another host terminal:

tail -f logs/qnx-linux-serial.log

Expect the second serial port to show, in order:

Linux version 4.9.52-i915-gvt_qnx-gdecfd5b ...
...
VFS: Mounted root (ext4 filesystem) ...
...
sh-3.2#

Verify with actual commands:

python3 scripts/serial_cmd.py --port 18002 'echo LINUX_GUEST_READY'
python3 scripts/serial_cmd.py --port 18002 'uname -a'

After mounting proc, the block devices and CPUs visible inside can also be read:

mount -t proc proc /proc
cat /proc/partitions
cat /proc/cpuinfo
cat /proc/cmdline

These outputs tie the nested boot together: the first serial port operates QNX, qvm creates the inner machine, the second serial port shows the Linux kernel booting and the root disk mounting, and the shell then runs commands and reads device and CPU information. Keeping both logs gives a boot baseline to compare item by item the next time the configuration changes.

8.6 Retrospective: qvm Passed Its Checks, So Why Was Linux Still Silent?

The steps above use a physical PVE and give the inner Linux one vCPU. This combination comes from two blockers during the build: one in qvm’s capability check, the other in Linux’s multicore initialization. The Linux hosting QEMU was itself a virtual machine at first; QNX could already reach a shell, and the storage driver and second serial port were ready, yet qvm reported this when creating the inner machine:

secondary procbased controls can not turn on 0x1

This error occurs before the Linux kernel starts producing output and comes from the capability check when qvm sets VMX control bits. QEMU being able to boot QNX shows the outer virtual machine is already running; QNX creating another machine with qvm depends on yet another layer of virtualization capability. Checking the KVM parameters on the Linux hosting QEMU showed nested=Y but enable_apicv=N and enable_shadow_vmcs=N. Seeing nested enabled alone is not enough to describe the whole set of runtime conditions qvm is about to use.

A diagnostic change was made along the way, loosening qvm’s check on APIC-related capabilities. The original error disappeared and the process kept running, but the inner serial port never produced Linux output. Looking further, the vCPU thread stayed in RUNNING while the printed VMCS fields were all zero. VMCS is the control structure Intel hardware virtualization uses, holding the guest’s controls and execution state; this output meant the VM still had to be checked for actually reaching a runnable state, and the process existing is only one observation point.

Another explanation for serial silence, easy to confuse with the first: Linux is already executing and we simply are not connected to its output. To separate the two, I ran a /dev/ser2 loopback test from the QNX side and received SER2_TEST externally. That verified the outer channel from the QNX serial driver to QEMU’s second serial port, so troubleshooting went back to qvm’s runtime state and host capabilities. Loosening the check showed more state but still gave no evidence that Linux was executing.

The setup then moved to a physical PVE, with the original qvm restored, and the conditions recorded on both sides compared:

ObservationEarly virtualized Linux environmentPhysical PVE used later
KVM nestedYY
KVM enable_apicvNY
KVM enable_shadow_vmcsNY
Original qvm’s capability checkErrors out and exitsPasses, and the inner layer starts producing Linux output

This move changed the CPU, host kernel and virtualization capability combination at the same time; the table records the differences that mattered during troubleshooting. On this PVE environment the original qvm passes the check and the second serial port finally shows Linux boot logs, so debugging moves into the guest kernel. The failed attempts are worth keeping: their most useful contribution is several distinct progress points—qvm passing its check, the vCPU thread existing, the serial channel working, and Linux starting to execute—each with its own way to observe it.

After Linux started producing output, another independent problem appeared. With two vCPUs configured for the inner machine, the serial port stopped where the second CPU is being started, and the last SMP-related lines were:

x86: Booting SMP configuration:
.... node  #0, CPUs:      #1

At this point the inner machine has reached Linux’s multicore initialization, a different place from the earlier qvm capability check. Removing the second cpu block and keeping one vCPU lets the same kernel and root disk continue to VFS: Mounted root and then a shell, while the outer QEMU still keeps 4 vCPUs. This comparison first yields a configuration that boots repeatably, leaving the cause of the multicore bring-up as a later question. The tutorial therefore starts with a single core and adds CPUs or other devices once the root disk, serial port and user space can be observed reliably.

8.7 Lessons from the Full Storage Layout

In experiments closer to the original car, I synthesized disks with multiple partitions and handled QNX device naming, the extended partition chain and QNX6 mounting item by item. This also turned up a bug in my own tool: the early disk generator wrote the next-EBR offset relative to the wrong base, so later logical partitions could not be enumerated. After correcting it to be relative to the start of the extended partition, partitions appeared one after another, consistent with the handling in Linux’s partition parsing code. On disk problems with an unfamiliar system, checking the image generation process and the filesystem parsing process together often yields very concrete clues.

Another piece of work was adapting to QNX6’s storage synchronization requirements, choosing mount options according to what the virtual controller provides and recording the corresponding sync semantics. The full boot flow brings these factors together, so this article first uses a single ext4 SD root disk to establish a qvm boot baseline. When later work reaches the factory mount scripts, partition tables, QNX6 filesystems and automatic mount steps can be added in turn, each layer compared against the environment currently known to work.

9. Turning the Experiment into Results That Can Be Compared and Explained

9.1 Locate Faults by Layer

With the observation channels above, “the program is not responding” can be unfolded layer by layer along the runtime relationships. First the whole machine, then guest user space and the target process, and finally application handling and the crash scene; each layer has a specific check:

  1. Is QEMU still executing?
  2. Can the guest shell run commands?
  3. Does the target program exist, and has the PID changed?
  4. Is the listener established, and do the interface and firewall match expectations?
  5. Did the normal request get a valid response?
  6. If it exited, what do the core and exit status say?

Troubleshooting this way, a port timeout becomes the starting point: confirm through the serial port that the guest can run commands, check the PID and listener state, then use the logs to judge which layer the request stopped at. The QNX dual serial setup divides the work similarly: when inner output stops, observation continues from the outer shell and qvm logs; when the outer serial connection has a problem, check QEMU’s connection and serial driver first. Each step uses the state already confirmed at the layer above, and the scope narrows gradually.

9.2 Record Experiment Configuration and Verification Results

To keep later experiments on the same line, I record environment adjustments together with what they make observable. The configurations in this article can be summarized in the table below; as the research goal expands, add the corresponding components around it:

LayerLab adjustmentWhat it makes observable
Linux bootLoad the kernel directly, replace the original init with a shellKernel boot and user-space services started by hand
Linux networkingLab addresses and specific port accept rulesThe request handling path from host to service
File writesOriginal rootfs with a temporary write layer, output disk saved separatelyComparing logs and crash scenes on the same baseline
Hardware dependenciesAdd devices or interface stubs step by step along the execution pathThe runtime conditions the current target function needs
QNX user spaceAdd serial ports, suspend several factory autostart servicesDriver initialization, device discovery and qvm boot
qvmSingle vCPU, single root disk, serial configurationInner Linux boot and basic operation

Experiment results can also accumulate progressively: “kernel boot”, “service response”, “function execution”, “complete boot flow”. That way each success is written down together with the configuration used and the output it produced, giving a clear basis for comparison when systemd, networking or more devices are added later.

9.3 Migrate from the Public Sample to a Car’s Build

Later in the research I also exported my own head unit’s kernel, file archive and raw block device image; they helped verify the boot content, inspect programs and preserve the filesystem layout. In a file-level export, hard links, symbolic links, owners and permissions are worth saving too, because they affect how the system behaves after it is rebuilt. Recording versions and hashes separately for the kernel, rootfs and applications makes specific differences easy to find when comparing builds.

Between the public update package and the build running in the car, shared library layout, directory paths and initialization conditions may differ. The lab built above is well suited to that migration: verify images and dependencies first, then compare behavior with the same boot checks and normal requests. Readers can complete the whole base flow with the 2024 public sample in this article, then replace inputs item by item and observe the results when another build is available.

9.4 Organize Logs and Crash Scenes

Each experiment run can have its own record directory holding the sample hash, QEMU and host versions, the full boot command, serial logs, service start commands and probe results, with modified configurations kept alongside. When a failure occurs, add the core, process and thread state, load mappings and the corresponding shared libraries. The registers, memory and call stack in a core belong to that build from that moment; archiving these files together is what lets GDB interpret the crash scene with the matching program.

A multi-layer system also needs a time base that is easy to align. The trimmed QNX environment’s clock may differ from the host’s, so record the host receive time, the relative time since kernel boot, and markers printed deliberately in each experiment. After collection, sync, unmount the output disk and end QEMU in that order, keeping the original baseline and clearly named write layers. The next run then starts from the same state, and new changes can be mapped to a specific configuration.

10. Further Reading

After Sections 4–6 we have a Linux lab environment that boots the factory kernel, runs services and analyzes cores; Sections 7–8 extend the observation scope to QNX and qvm. From here, devices can be filled in according to interest: for graphics and input there are practical cases such as “ROOT Tesla OS on QEMU Part 2: Debugging + fixing” and the tesla-qemu project; to start from a buildable automotive Linux, see AGL’s x86/QEMU build instructions. Comparing them with this article’s vendor images also helps show what the generic virtual hardware and the platform-specific parts each handle.

Main documents and tools used in this article:

Going from an update package to a debuggable system, the most helpful thing is to keep a visible execution path at all times: which kernel is loaded, which disk is mounted, who starts the program, which interfaces a request passes through, where the crash scene is saved. Each layer of environment added gets a repeatable check; when a problem appears, follow the path down layer by layer. This way, closed head unit software gradually becomes an experimental object we can observe, compare and run repeatedly.