.png)
1 month ago
7
QEMU’s Tiny Code Generator (TCG) provides the ability to emulate a number of CPU architectures on any supported host platform. Both System Emulation and User Mode Emulation are supported depending on the guest architecture.
Alpha |
Yes |
Yes |
Legacy 64 bit RISC ISA developed by DEC |
Arm (arm, aarch64) |
Yes |
Wide range of features, see A-profile CPU architecture support for details |
|
AVR |
No |
8 bit micro controller, often used in maker projects |
|
Hexagon |
No |
Yes |
Family of DSPs by Qualcomm |
PA-RISC (hppa) |
Yes |
Yes |
A legacy RISC system used in HP’s old minicomputers |
x86 (i386, x86_64) |
Yes |
The ubiquitous desktop PC CPU architecture, 32 and 64 bit. |
|
LoongArch |
Yes |
Yes |
A MIPS-like 64bit RISC architecture developed in China |
m68k |
Yes |
Motorola 68000 variants and ColdFire |
|
Microblaze |
Yes |
Yes |
RISC based soft-core by Xilinx |
MIPS (mips*) |
Yes |
Venerable RISC architecture originally out of Stanford University |
|
OpenRISC |
Yes |
Open source RISC architecture developed by the OpenRISC community |
|
Power (ppc, ppc64) |
Yes |
A general purpose RISC architecture now managed by IBM |
|
RISC-V |
Yes |
An open standard RISC ISA maintained by RISC-V International |
|
RX |
No |
A 32 bit micro controller developed by Renesas |
|
s390x |
Yes |
A 64 bit CPU found in IBM’s System Z mainframes |
|
sh4 |
Yes |
Yes |
A 32 bit RISC embedded CPU developed by Hitachi |
SPARC (sparc, sparc64) |
Yes |
A RISC ISA originally developed by Sun Microsystems |
|
Tricore |
Yes |
No |
A 32 bit RISC/uController/DSP developed by Infineon |
Xtensa |
Yes |
A configurable 32 bit soft core now owned by Cadence |
Semihosting
Semihosting is a feature defined by the owner of the architecture to allow programs to interact with a debugging host system. On real hardware this is usually provided by an In-circuit emulator (ICE) hooked directly to the board. QEMU’s implementation allows for semihosting calls to be passed to the host system or via the gdbstub.
Generally semihosting makes it easier to bring up low level code before a more fully functional operating system has been enabled. On QEMU it also allows for embedded micro-controller code which typically doesn’t have a full libc to be run as “bare-metal” code under QEMU’s user-mode emulation. It is also useful for writing test cases and indeed a number of compiler suites as well as QEMU itself use semihosting calls to exit test code while reporting the success state.
Semihosting is only available using TCG emulation. This is because the instructions to trigger a semihosting call are typically reserved causing most hypervisors to trap and fault on them.
Warning
Semihosting inherently bypasses any isolation there may be between the guest and the host. As a result a program using semihosting can happily trash your host system. Some semihosting calls (e.g. SYS_READC) can block execution indefinitely. You should only ever run trusted code with semihosting enabled.
Redirection
Semihosting calls can be re-directed to a (potentially remote) gdb during debugging via the gdbstub. Output to the semihosting console is configured as a chardev so can be redirected to a file, pipe or socket like any other chardev device.
Supported Targets
Most targets offer similar semihosting implementations with some minor changes to define the appropriate instruction to encode the semihosting call and which registers hold the parameters. They tend to presents a simple POSIX-like API which allows your program to read and write files, access the console and some other basic interactions.
For full details of the ABI for a particular target, and the set of calls it provides, you should consult the semihosting specification for that architecture.
Note
QEMU makes an implementation decision to implement all file access in O_BINARY mode. The user-visible effect of this is regardless of the text/binary mode the program sets QEMU will always select a binary mode ensuring no line-terminator conversion is performed on input or output. This is because gdb semihosting support doesn’t make the distinction between the modes and magically processing line endings can be confusing.
TCG Plugins
QEMU TCG plugins provide a way for users to run experiments taking advantage of the total system control emulation can have over a guest. It provides a mechanism for plugins to subscribe to events during translation and execution and optionally callback into the plugin during these events. TCG plugins are unable to change the system state only monitor it passively. However they can do this down to an individual instruction granularity including potentially subscribing to all load and store operations.
See the developer section of the manual for details about writing plugins.
Usage
Any QEMU binary with TCG support has plugins enabled by default. Earlier releases needed to be explicitly enabled with:
Once built a program can be run with multiple plugins loaded each with their own arguments:
Arguments are plugin specific and can be used to modify their behaviour. In this case the howvec plugin is being asked to use inline ops to count and break down the hint instructions by type.
Linux user-mode emulation also evaluates the environment variable QEMU_PLUGIN:
QEMU plugins avoid to write directly to stdin/stderr, and use the log provided by the API (see function qemu_plugin_outs). To show output, you may use this additional parameter:
Example Plugins
There are a number of plugins included with QEMU and you are encouraged to contribute your own plugins plugins upstream. There is a contrib/plugins directory where they can go. There are also some basic plugins that are used to test and exercise the API during the make check-tcg target in tests/tcg/plugins that are never the less useful for basic analysis.
Empty
tests/tcg/plugins/empty.c
Purely a test plugin for measuring the overhead of the plugins system itself. Does no instrumentation.
Basic Blocks
tests/tcg/plugins/bb.c
A very basic plugin which will measure execution in coarse terms as each basic block is executed. By default the results are shown once execution finishes:
Behaviour can be tweaked with the following arguments:
inline=true|false |
Use faster inline addition of a single counter. |
idle=true|false |
Dump the current execution stats whenever the guest vCPU idles |
Basic Block Vectors
contrib/plugins/bbv.c
The bbv plugin allows you to generate basic block vectors for use with the SimPoint analysis tool.
interval=N |
The interval to generate a basic block vector specified by the number of instructions (Default: N = 100000000) |
outfile=PATH |
The path to output files. It will be suffixed with .N.bb where N is a vCPU index. |
Example:
Instruction
tests/tcg/plugins/insn.c
This is a basic instruction level instrumentation which can count the number of instructions executed on each core/thread:
Behaviour can be tweaked with the following arguments:
inline=true|false |
Use faster inline addition of a single counter. |
sizes=true|false |
Give a summary of the instruction sizes for the execution |
match=<string> |
Only instrument instructions matching the string prefix |
The match option will show some basic stats including how many instructions have executed since the last execution. For example:
For more detailed execution tracing see the execlog plugin for other options.
Memory
tests/tcg/plugins/mem.c
Basic instruction level memory instrumentation:
Behaviour can be tweaked with the following arguments:
inline=true|false |
Use faster inline addition of a single counter |
callback=true|false |
Use callbacks on each memory instrumentation. |
hwaddr=true|false |
Count IO accesses (only for system emulation) |
System Calls
tests/tcg/plugins/syscall.c
A basic syscall tracing plugin. This only works for user-mode. By default it will give a summary of syscall stats at the end of the run:
Behaviour can be tweaked with the following arguments:
print=true|false |
Print the number of times each syscall is called |
log_writes=true|false |
Log the buffer of each write syscall in hexdump format |
Test inline operations
tests/plugins/inline.c
This plugin is used for testing all inline operations, conditional callbacks and scoreboard. It prints a per-cpu summary of all events.
Hot Blocks
contrib/plugins/hotblocks.c
The hotblocks plugin allows you to examine the where hot paths of execution are in your program. Once the program has finished you will get a sorted list of blocks reporting the starting PC, translation count, number of instructions and execution count. This will work best with linux-user execution as system emulation tends to generate re-translations as blocks from different programs get swapped in and out of system memory.
Example:
Hot Pages
contrib/plugins/hotpages.c
Similar to hotblocks but this time tracks memory accesses:
The hotpages plugin can be configured using the following arguments:
sortby=reads|writes|address |
Log the data sorted by either the number of reads, the number of writes, or memory address. (Default: entries are sorted by the sum of reads and writes) |
io=on |
Track IO addresses. Only relevant to full system emulation. (Default: off) |
pagesize=N |
The page size used. (Default: N = 4096) |
Instruction Distribution
contrib/plugins/howvec.c
This is an instruction classifier so can be used to count different types of instructions. It has a number of options to refine which get counted. You can give a value to the count argument for a class of instructions to break it down fully, so for example to see all the system registers accesses:
which will lead to a sorted list after the class breakdown:
To find the argument shorthand for the class you need to examine the source code of the plugin at the moment, specifically the *opt argument in the InsnClassExecCount tables.
Lockstep Execution
contrib/plugins/lockstep.c
This is a debugging tool for developers who want to find out when and where execution diverges after a subtle change to TCG code generation. It is not an exact science and results are likely to be mixed once asynchronous events are introduced. While the use of -icount can introduce determinism to the execution flow it doesn’t always follow the translation sequence will be exactly the same. Typically this is caused by a timer firing to service the GUI causing a block to end early. However in some cases it has proved to be useful in pointing people at roughly where execution diverges. The only argument you need for the plugin is a path for the socket the two instances will communicate over:
which will eventually report:
Hardware Profile
contrib/plugins/hwprofile.c
The hwprofile tool can only be used with system emulation and allows the user to see what hardware is accessed how often. It has a number of options:
track=[read|write] |
By default the plugin tracks both reads and writes. You can use this option to limit the tracking to just one class of accesses. |
source |
Will include a detailed break down of what the guest PC that made the access was. Not compatible with the pattern option. Example output: cirrus-low-memory @ 0xfffffd00000a0000
pc:fffffc0000005cdc, 1, 256
pc:fffffc0000005ce8, 1, 256
pc:fffffc0000005cec, 1, 256
|
pattern |
Instead break down the accesses based on the offset into the HW region. This can be useful for seeing the most used registers of a device. Example output: pci0-conf @ 0xfffffd01fe000000
off:00000004, 1, 1
off:00000010, 1, 3
off:00000014, 1, 3
off:00000018, 1, 2
off:0000001c, 1, 2
off:00000020, 1, 2
...
|
Execution Log
contrib/plugins/execlog.c
The execlog tool traces executed instructions with memory access. It can be used for debugging and security analysis purposes. Please be aware that this will generate a lot of output.
The plugin needs default argument:
which will output an execution trace following this structure:
Please note that you need to configure QEMU with Capstone support to get disassembly.
The output can be filtered to only track certain instructions or addresses using the ifilter or afilter options. You can stack the arguments if required:
This plugin can also dump registers when they change value. Specify the name of the registers with multiple reg options. You can also use glob style matching if you wish:
Be aware that each additional register to check will slow down execution quite considerably. You can optimise the number of register checks done by using the rdisas option. This will only instrument instructions that mention the registers in question in disassembly. This is not foolproof as some instructions implicitly change instructions. You can use the ifilter to catch these cases:
Cache Modelling
contrib/plugins/cache.c
Cache modelling plugin that measures the performance of a given L1 cache configuration, and optionally a unified L2 per-core cache when a given working set is run:
will report the following:
The plugin has a number of arguments, all of them are optional:
limit=N |
Print top N icache and dcache thrashing instructions along with their address, number of misses, and its disassembly. (default: 32) |
icachesize=N iblksize=B iassoc=A |
Instruction cache configuration arguments. They specify the cache size, block size, and associativity of the instruction cache, respectively. (default: N = 16384, B = 64, A = 8) |
dcachesize=N |
Data cache size (default: 16834) |
dblksize=B |
Data cache block size (default: 64) |
dassoc=A |
Data cache associativity (default: 8) |
evict=POLICY |
Sets the eviction policy to POLICY. Available policies are: lru, fifo, and rand. The plugin will use the specified policy for both instruction and data caches. (default: POLICY = lru) |
cores=N |
Sets the number of cores for which we maintain separate icache and dcache. (default: for linux-user, N = 1, for full system emulation: N = cores available to guest) |
l2=on |
Simulates a unified L2 cache (stores blocks for both instructions and data) using the default L2 configuration (cache size = 2MB, associativity = 16-way, block size = 64B). |
l2cachesize=N |
L2 cache size (default: 2097152 (2MB)), implies l2=on |
l2blksize=B |
L2 cache block size (default: 64), implies l2=on |
l2assoc=A |
L2 cache associativity (default: 16), implies l2=on |
Stop on Trigger
contrib/plugins/stoptrigger.c
The stoptrigger plugin allows to setup triggers to stop emulation. It can be used for research purposes to launch some code and precisely stop it and understand where its execution flow went.
Two types of triggers can be configured: a count of instructions to stop at, or an address to stop at. Multiple triggers can be set at once.
By default, QEMU will exit with return code 0. A custom return code can be configured for each trigger using :CODE syntax.
For example, to stop at the 20-th instruction with return code 41, at address 0xd4 with return code 0 or at address 0xd8 with return code 42:
The plugin will log the reason of exit, for example:
Limit instructions per second
This plugin can limit the number of Instructions Per Second that are executed:
ips=N |
Maximum number of instructions per cpu that can be executed in one second. The plugin will sleep when the given number of instructions is reached. |
ipq=N |
Instructions per quantum. How many instructions before we re-calculate time. The lower the number the more accurate time will be, but the less efficient the plugin. Defaults to ips/10 |
Uftrace
contrib/plugins/uftrace.c
This plugin generates a binary trace compatible with uftrace.
Plugin supports aarch64 and x64, and works in user and system mode, allowing to trace a system boot, which is not something possible usually.
In user mode, the memory mapping is directly copied from /proc/self/maps at the end of execution. Uftrace should be able to retrieve symbols by itself, without any additional step. In system mode, the default memory mapping is empty, and you can generate one (and associated symbols) using contrib/plugins/uftrace_symbols.py. Symbols must be present in ELF binaries.
It tracks the call stack (based on frame pointer analysis). Thus, your program and its dependencies must be compiled using -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer. In 2024, Ubuntu and Fedora enabled it by default again on x64. On aarch64, this is less of a problem, as they are usually part of the ABI, except for leaf functions. That’s true for user space applications, but not necessarily for bare metal code. You can read this section to easily build a system with frame pointers.
When tracing long scenarios (> 1 min), the generated trace can become very long, making it hard to extract data from it. In this case, a simple solution is to trace execution while generating a timestamped output log using qemu-system-aarch64 ... | ts "%s". Then, uftrace --time-range=start~end can be used to reduce trace for only this part of execution.
Performance wise, overhead compared to normal tcg execution is around x5-x15.
trace-privilege-level=[on|off] |
Generate separate traces for each privilege level (Exception Level + Security State on aarch64, Rings on x64). |
elf_file [elf_file …] |
path to an ELF file. Use /path/to/file:0xdeadbeef to add a mapping offset. |
–prefix-symbols |
prepend binary name to symbols |
Example user trace
As an example, we can trace qemu itself running git:
For convenience, you can download this trace qemu_aarch64_git_help.json.gz. Download it and open this trace on https://ui.perfetto.dev/. You can zoom in/out using W, A, S, D keys. Some sequences taken from this trace:
Loading program and its interpreter
open syscall
TB creation
It’s usually better to use uftrace record directly. However, tracing binaries through qemu-user can be convenient when you don’t want to recompile them (uftrace record requires instrumentation), as long as symbols are present.
Example system trace
A full trace example (chrome trace, from instructions below) generated from a system boot can be found here. Download it and open this trace on https://ui.perfetto.dev/. You can see code executed for all privilege levels, and zoom in/out using W, A, S, D keys. You can find below some sequences taken from this trace:
Two first stages of boot sequence in Arm Trusted Firmware (EL3 and S-EL1)
U-boot initialization (until code relocation, after which we can’t track it)
Stat and open syscalls in kernel
Timer interrupt
Poweroff sequence (from kernel back to firmware, NS-EL2 to EL3)
Build and run system example
Building a full system image with frame pointers is not trivial.
We provide a simple way to build an aarch64 system, combining Arm Trusted firmware, U-boot, Linux kernel and debian userland. It’s based on containers (podman only) and qemu-user-static (binfmt) to make sure it’s easily reproducible and does not depend on machine where you build it.
You can follow the exact same instructions for a x64 system, combining edk2, Linux, and Ubuntu, simply by switching to x86_64 branch.
To build the system:
To generate a uftrace for a system boot from that:
Uftrace allows to filter the trace, and dump flamegraphs, or a chrome trace. This last one is very interesting to see visually the boot process:
Long visual chrome traces can’t be easily opened, thus, it might be interesting to generate them around a particular point of execution:
Other emulation features
When running system emulation you can also enable deterministic execution which allows for repeatable record/replay debugging. See Record/Replay for more details.
