Probing BMC Firmware on Supermicro X11SSH

Trivia

During the last discussion I have successfully managed to gain access to BMC debug pins. This way we have access to stock firmware shell, which allows us to do some probing.

The issue and the reason

The whole reason for doing this is so we can successfully port OpenBMC to x11ssh platform. We already managed to build custom OpenBMC and run it on said platform, but "running" and being usable are two different things.

As for now we need to resolve two issues:

• [email protected] - failing due to missing GPIO definitions. We know this by attempting to manually run the binary.
  root@x11ssh:~# /usr/bin/power-control 0 <6> Start Chassis power control service for host : 0 <3> BiosMux is not a recognized power-control signal name <3> Host0: Error in Parsing... <6> SIO control GPIOs not defined, disable SIO support. <3> PowerOk name should be configured from json config file
• [email protected] - failing due to lack od defined KCS interfaces.
  root@x11ssh:~# systemctl start [email protected] Job for [email protected] failed because the control process exited with error code. See "systemctl status [email protected]" and "journalctl -xeu [email protected]" for details. root@x11ssh:~# systemctl status [email protected] x [email protected] - Phosphor IPMI KCS DBus Bridge Loaded: loaded (8;;file://x11ssh/usr/lib/systemd/system/[email protected]/usr/lib/systemd/system/[email protected];;; enabled; preset: 5:185mdisabled) Active: failed (Result: exit-code) since Thu 2024-12-19 22:03:08 UTC; 16s ago Invocation: 498e04833e5e4183bdecb22273fd34bd Process: 381 ExecStart=/usr/libexec/kcsbridged -c ipmi-kcs3 (code=exited, status=1/FAILURE) Main PID: 381 (code=exited, status=1/FAILURE) Dec 19 22:03:08 x11ssh systemd[1]: [email protected]: Scheduled restart job, restart coun...is at 2. Dec 19 22:03:08 x11ssh systemd[1]: 5:185m5:[email protected]: Start request repeated too quickly. Dec 19 22:03:08 x11ssh systemd[1]: 5:185m5:[email protected]: Failed with result 'exit-code'. Dec 19 22:03:08 x11ssh systemd[1]: Failed to start Phosphor IPMI KCS DBus Bridge. Hint: Some lines were ellipsized, use -l to show in full. root@x11ssh:~# /usr/libexec/kcsbridged -c ipmi-kcs3 FAILED: open `/dev/ipmi-kcs3`: No such file or directory [...] root@x11ssh:~# ls /dev/ | grep "ipmi\|kcs" root@x11ssh:~#

Both of these issues are related to incomplete device tree and missing defintions. Eg. for GPIO, we've got this and that GPIO defs, yet we seem to be missing POWER_OK definition. For the KCS configuration we're missing such KCS interfaces definitions.

What we were able to establish, the KCS addresses are to be found in the AST2400 documentation, they should be standardized between various Aspeed SoCs (I'll need to verify that). For GPIO, this is dependent how OEM "wired" the SoC, it is not standardized.

Resources and what do we know

Resources:
Stock firmware sources: https://www.supermicro.com/wdl/GPL/SMT/X10_GPL_Release_20150819.tar.gz
Note: Since this is Linux, they're obligated to supply sources. The issue is however that they use proprietary modules for kcs and the sources contain precompiled binaries.
AST2400 Docs: https://gitcode.com/Open-source-documentation-tutorial/69bbb
X11SSH gerber files: https://github.com/mithro/x11ssh-f-pcb
Stock BMC firmware (we did not use exactly this one): https://www.supermicro.com/en/support/resources/downloadcenter/firmware/SYS-5019S-M/BMC
Porting attempt by HardenedVault :https://github.com/hardenedvault/openbmc/tree/x11ssh-f, https://hardenedvault.net/blog/2024-11-17-openbmc-remote-deployment/, https://hardenedvault.net/blog/2024-03-15-openbmc-x11ssh-port/
Porting to u-bmc (Keno Fisher, osresearch): https://github.com/osresearch/u-bmc/tree/kf/x11

Additionally:

• Shell access to UART

What do we know:

• The stock BMC version is based on a patched, ancient Linux 2.6.28.9 kernel.
• The solution is from before device-tree era, and uses ATAGs. TL;DR the platform support is baked directly into kernel, hardware provides only basic stuff to identify itself. This means no device tree structure decompilation as it simply does not exist. We confirmed this by running firmware under qemu and gdb.
  (gdb) break *0x40008000 Breakpoint 1 at 0x40008000 (gdb) c Continuing. Breakpoint 1, 0x40008000 in ?? () (gdb) info registers r0 r0 0x0 0 (gdb) info registers r1 r1 0x385 901 (gdb) info registers r2 r2 0x40000100 1073742080 (gdb) info registers r3 r3 0x9007a 589946 (gdb) info registers r4 r4 0x0 0 (gdb) info registers r5 r5 0x4052fd0c 1079180556 (gdb) info registers r6 r6 0x404cffb8 1078788024 (gdb) info registers r7 r7 0x385 901
• Critical modules like KCS are proprietary and precompiled.
• I compiled HardenedVault's solution, as expected it suffers from same issues.
• The support for x11spi to OpenBMC has supposedly been added by guys from Supermicro, and hasn't been really maintained since.
• I attempted to compile mentioned u-bmc version, but it was also 5+ years (if I remember correctly) since it received any update, thus also can be categorized as unmaintained. I wasn't able to verify it works.
• Mentioned already "tested" the image under qemu, but due to missing devices it was hard to perform proper scoping, as many things failed.

Aprroach plan

I don't have any detailed approach plan. Let's just scope the system and see what we can learn. When Linux kernel 2.6.28 has been released in ~2008 I was 9, so I don't really know what to expect. This will be a learning experience to me as well.

Join the discussion

As in the previous discussion, the idea for the discussions is to perform semi-live "coding". This way it's easier to describe where are we at, and what are the current problems to the collaborators. For these sessions, I give myself the permission to use less formal language, so consider yourself warned 😆.

Feel free to join the discussion at any moment. Your feedback is much appreciated!

Let's start.