spispy: Open source flash emulation

This is the very rough transcript of my presentation at CCC Camp 2019, spispy: open source flash emulation for faster firmware development and early boot security research. The 30 minute video is also available if you prefer to watch. The source github.com/osresearch/spispy is available if you want to jump right to hacking on it. In this presentation I want to address three questions: what are SPI flashes, why do we want to emulate them, and how do we solve the tricky real-time difficulties in doing so.

What

The first question is "what is a spi flash?". These are the small chips on a computer mainboard, frequently called "boot ROMs" although that is a bit of misnomer since they are re-writable these days and not actual ROMs. These flash chips contain the "firmware" or "BIOS" for the machine that is executed when the system is first powered on, before any of the other devices are initialized.

The original BIOSes were proprietary to each machine and most were fairly simple since they had to fit in a few kilobytes of ROM chip -- they would locate the bootloader for the operating system on a floppy or harddisk boot block and jump into it. Most of the BIOSes remained resident to perform basic input and output functions,

This firmware used to be stored in actual ROMs, although in more modern systems it is in a non-volatile flash memory, typically in an 8 pin SOIC package. Pulling up the datasheet for this one reveals the expansion of the SPI acronym: Serial Peripheral Interface, a fairly generic term for the separate clock and data bus that it uses. Unlike the original 64 KiB BIOS chips, modern SPI flashes are typically 64 or 128 megabits, which is 16 or 32 MiB: not enormous, but plenty of space compared to 64 KiB.

These larger chips allow significantly more complexity in the firmware, and the proprietary Unified Extensible Firmware Interface (UEFI) firmware used by most commodity systems has added graphics, networking and more: it's a few million lines of code and an entire operating system. There are also open source firmware like coreboot and linuxboot. In addition to being able to read the source code and modify it to fit our needs, it is much simpler and reduced complexity compared tothe closed source vendor UEFI.

Why

The second question is "why would you need to emulate the SPI flash?".

Well, if you're developing open firmware like coreboot or linuxboot, or if you are doing security research into the closed UEFI firmware, you end up having to flash new firmware images into the SPI chips over and over and over. I have spent far too much of my life waiting for these chips to flash...

Slow erase speeds

You might think that the chips aren't very large only 16 MiB, remember? except that they are intended to be mostly read-only and are not optimized for frequent writes. The chips have to be written in 4KiB sectors, which have to be erased before they can be re-written, and this erase time in the worst case is 120 milliseconds. So multiply 120ms by 16 MiB divide by 4 KiB is around eight minutes for the worst case.

And this is every time you want to test a change or run an experiment. So let's say that you've just finished building coreboot and are ready to flash it.

First step is that you have to power down the machine under test -- the SPI bus doesn't like having multiple devices driving it. Then you have to carefully attach your flash programmer clip to the chip.

The you can run the flashrom or other tool to start the write. The first few megabytes go really fast, although that is only since the Management Engine section hasn't changed, so no erasing is necessary.. Once it hits the BIOS region, however, it has to start erasing blocks and slows down to 30-50 KiB/s.

Three minutes in, still slowly erasing and rewriting at 28 KiB/s. How long is this going to take? It. Is. So. Slow.

Finally the flash write finishes after five and a half minutes, at a blazing average of 49 KiB/s. But it isn't time to test yet!

Most of the time it is necessary to remove the programmer. Maybe you can keep it attached, although the additional capacitance and loading can cause problems on the bus, so frequently you have to reattach it each time. Finally you can power the system back on and see if you forgot something stupid and have to do it all over again! SPOILER ALERT: it won't work the first ten times and you will have to keep trying different things to get coreboot to work.

Seriously I have spent so much of my time flashing these chips on so many different platforms that my flickr feed is full of these things.

The better way

As they say in infomercials, there has to be a better way!

With spispy you attach the clip once and leave it attached to the SPI flash. Now when your coreboot or linuxboot build finishes, you can run the write command to upload the new image into the SDRAM of the spispy. Here is a video in real time flashing a new firmware with spispy

The new image is loaded in 12 seconds, with the speed limited only by the USB-CDC-ACM bandwidth. We could improve this since our RAM speed is hundreds of megabytes per second, although our bottleneck will be the USB 2.0 speeds.

Once the new image is resident, you can soft reboot into the new firmware without having to get out of your chair or go to the datacenter. This greatly reduces the cycle time to test out new firmware builds and experiment with debugging.

How an x86 really boots

There's another part to the "Why" question, which is understanding what happens in those first few milliseconds after you power on the computer. If you've just arrived at camp from the 1970s (or even just a few years ago), you might expect that modern x86 CPUs start the same way with a real mode reset vector at the top of the firmware. That's no longer the case and spispy gives us insight into is really happening during the system startup.

In monitor mode it prints the addresses that are being read from the flash so that we can analyze them. As a quick walk through what happens

• first thing we see the flash read is at offset 0x10 and 0x14, which is the Platform Controller Hub reading the Intel Flash Descriptor to verify that there is a valid flash attached.
• Then the ME starts up and reads its partition table from the flash and validates its signatures. Eventually it starts the x86.
• Which still has more to do before getting to the reset vector. Something in the x86, maybe microcode or an on-die ROM, reads the Firmware Interface Table and finds
• the microcode update entries. The correct microcode update is read and applied
• and then the bootguard authenticated code module is located and run (this platform didn't have bootguard).
• and finally, bootguard transfers control to the legacy reset vector,
• which jumps into the BIOS region. This is quite different than our time travelling visitors would have expected...

This log of flash accesses also reveals potential security vulnerabilities. If we plot the read address on the Y axis versus time on the X axis, we can see that some addresses are read during a linear scan and signature verification (called the "Time of Check"), and then some of those same addresses are re-read again from the flash (called "Time of Use"). If those re-reads are not correctly validated, a malicious flash chip could substitute new data and take control of the CPU even with otherwise strong cryptographic signatures.

Peter Bosch and I were able to leverage this TOCTOU attack against Intel's Bootguard and bypass their verified boot modes. We presented the results at HITB earlier this year in Amsterdam and this release of spispy was one of the things we promised, so i'm very pleased to be able to go into more details about how our system was able to emulate the SPI flash to discover this attack.

How

So that's the what and why, and since this is a very technical conference, let's get deep into the weeds of the how.

Credits

First I want to be sure to point out that this project is built on the wonderful work of many others. It would not be possible to have developed it as quickly without the excellent open source fpga toolchain of yosys, icestorm, project trellis and nextpnr. These tools have created an entire open source fpga ecosystem and are a delight to use compared to the vendor tools.

spispy also grew out of a collaboration with Alyssa @noopwafel Milburn and Peter Bosch. We were hacking on things at Revspace in Den Haag after 35c3 since Peter and I had both noticed the potential TOCTOU and met each other at Congress.

We built our Bootguard POC on a small iCE40up5k FPGA, supported by the open source icestorm tool chain. This was sufficient for the TOCTOU that we wanted to demonstrate -- we only needed a few kilobits of data that easily fit in the blockram of the iCE40. The bigger problem was deciding which TOCTOU to go after! However, for the full spispy we realized that we needed much more memory.

And here again we benefited from the open source ecosystem. EMARD at the Croatian hackerspace Radiona had designed the ULX3S, a wonderfully full featured board around the larger and faster ecp5 FPGA supported by project trellis. It had a 32 MiB SDRAM that can read/write at around 250 MiB/s, which is plenty to keep up with the SPI bus.

Unfortunately SDRAM is full of dark magic with complex state transitions [that look more like alchemic transmutation diagrams. There is a large amount of complexity in interfacing with it and keeping it working reliably. Again, open source to the rescue. Stefan Kristiansson wrote https://github.com/skristiansson/wb_sdram_ctrl a verilog SDRAM controller and released it under a permissive license, allowing us to very quickly integrate the SDRAM on the ULX3S with our proof of concept SPI TOCTOU code.

We're not the first people to try to emulate SPI flashes -- as with many projects scanlime had already done it. Her code was dealing with smaller and slower flashes for the Nintendo DS, although conceptionally it had many of the same pieces that we needed to understand the bus.

The SPI interface

So let's go back to the SPI flash chips and dive back into the datasheet to better understand what is happening on the SPI bus. As I mentioned earlier, most of them are in an 8 pin package and we can identify pin one from the dot on the chip. Most importantly we can locate the power and ground on opposite corners and be sure that we don't mix those up.

• The datasheet also contains timing diagrams for the signals.
• The first one to note is the Chip Select pin, which goes low when the platform controller hub is talking to the SPI flash. I've marked it here with an arrow going into the chip to indicate that it is generated from the PCH and received by the flash. We call this "negative logic" or "active low" since a 0 means selected and designate the signal with a ! or # mark in the name.
• The platform controller hub also generates the clock signal to the flash. On the falling edges of the clock the values on the data lines change and they must be stable on the rising edges. We call this "rising edge clocked", although some SPI buses are clocked on the falling edge.
• The PCH sends commands on the Serial In line to the flash. Typically there is one command byte followed by some number of argument bytes, although devices have to know how many bytes each command takes since there is no framing on the packet.
• After the command bytes, the flash responds on the Serial Out line and typically will continue to clock out bytes until the PCH raises the CS line, indicating that the transaction is over.

The datasheet lists the command bytes and the number of bytes that each command sends. The one that we're interested most in emulating is the normal read, 0x03. We currently don't support the other reads, although it would be useful to have them implemented since they would speed up the boot and allow us to support some other architectures.

Moving to the timing diagram for the 0x03 READ, we see that it sends the 24 address bits most significant bit first and the flash responds with the data on the clock immediately following the least significant address bit.

if (spi_done)


In our blockram based proof of concept we essentially had this logic. When the last bit of address arrived, look up the contents at that address and send it on the SO line. But when we put SDRAM in there, something went wrong...

Here the real flash is shown in blue and the emulated flash is shown in yellow. The correct output value didn't appear on the SO line until nearly all the way through the bit, delayed by around 50 ns. This totally missed the timing requirements and preventing the x86 from booting.

The reason is that SDRAM isn't "single cycle" like the block RAM. Random reads require multiple clock cycles to complete -- first the row and bank addresses have to be sent, which takes a clock cycle, and then there is a mandatory RCD (???) delay. Then the column address is sent, another clock cycle, and the CAS latency delays the data for another few cycles. With our 132 MHz SDRAM clock, this is a total of 5-7 clocks and 50ns of delay. Ironically faster memories sometimes have more latency for a random read.

Zooming in on the SPI read timing diagram, we see how tight the requirements are, and why that causes the problem -- with a 20 MHz SPI clock our emulator must have the data available in roughly half a clock, or about 25 ns. Doing a naive read from the SDRAM won't work if it takes 50ns or more to read a random byte from the DRAM! We need to get clever if we're going to meet timing requirements.

Since we're writing the fpga logic we can be very clever indeed. We don't have to limit ourselves to concepts like "byte boundaries" or limitations of existing SPI hardware, nor do we need to respect the abstraction that memory is a big array, so we can do things like initiate the row/bank activation when the emulator has received 14 bits of the address, overlapping tRCD with reception of the next 9 bits, which we send to the column address and overlap the CAS latency with receipt of the last address bit, which we use to select the upper or lower byte of the 16-bit read result.

This sort of hackery actually works! Again the real flash chip is in blue, our emulated flash is in yellow and you can see that our first output bit is only a few nanoseconds slower than the real flash chip, which is good enough to meet timing for a 20 MHz SPI bus. We can now supply the bits to the very first read request from the PCH with the response 0x5AA5 in msb order. Checking the hexdump of the flash chip at the read offset 0x10 in the flash in msb oder.

This is how the PCH knows that there is a valid flash attached, as Butterworth and Kovah discuss in their advanced x86 BIOS class. If this is not correctly read by the PCH, the system won't start up at all.

However, while the spispy can convince the PCH to start up, sometimes the ME fails its signature check or the Linuxboot kernel fails its integrity checks. This is due to another complexity of the SDRAM...

Going back to the SDRAM datasheet and that state transition diagram... The "auto refresh" cycle in the SDRAM controller takes over the SDRAM every 7.8us to activate a row and "recharge" the capacitors that are the memory cells in DRAM. While this is happening no reads are possible for around 60ns, which causes the flash emulator to miss its very tight timing requirements if a refresh happens during a read.

Yet again, open source and flexible hardware saves us! We can add our own refresh inhibit signal to the sdram controller and modify the logic to prevent refresh cycles when the SPI controller detects that a read is happening. The 7.8us is a rough requirement, so as long as we allow refreshes after the read transaction has completed, then SDRAM will continue to hold data correctly.

Another complication is how to prevent the real SPI flash from responding to the read requests. You might have noticed that most of these photos have a blue Pomona chip clip attached and that the flash chip is still on the mainboard - doesn't this mean that the !CS line is going to wakeup the real flash?

It turns out that on many mainboards the !CS pin on the SPI flash has a small series resistor between it and the PCH's !CS output. This means that we can create sort of an OR gate by driving the !CS line from the FPGA and overpowering the PCH without risk of damaging either.

Schematically it looks something like this. The PCH drives !CS low, which wakes up the real SPI flash chip and the spispy FPGA. The SPI flash starts to drive the SO line to send data back to the PCH, but if this is a request that the spispy wants to handle, the fpga drives its !CS gpio pin high. This puts the SPI flash back to sleep and allows the fpga to take over the SO line.

A complication is that the fpga now doesn't know when the PCH has finished the transaction, so we've added some logic that watches the CLK line for transitions. A few microseconds after no clock transitions, then FPGA assumes the transaction is over and tri-states the !CS gpio. This allows the PCH to re-assert !CS by driving it low later to start another transaction.

Status

Putting all of these hacks together works! We can boot the entire x86, including the PCH and ME, from the spispy and they interact with it as if it were the real flash.

We've tested it on numerous laptops, desktops and a few servers. There are some academic groups who have been using it to look into early-boot security and I'm using it to assist in bringup on the first modern x86 server to support coreboot (in conjuction with 9elements and mullvad vpn).

There are other SPI flashes that are of interest, such as the ARM based BMC on that same server. Unfortunately we don't support some of the commands that it uses, yet, but hopefully will be able to improve the speed of development for openbmc or u-bmc through fixing those feature.

Help wanted

And spispy is open source! So if you want to dive in and help out, there are lots of areas that we would appreciate help. One area that would make it even faster is if we move away from USB serial and implement USB ethernet or a custom endpoint. There are other buses in the system that are interesting to TOCTOU, so security researchers have at it!

You can checkout the source code and tools in our github tree, github.com/osresearch/spispy, chat with us on the open source firmware slack, or meet me here at camp (I'm hanging out at the hardware hacking area). So hopefully I've answered the what, why and how and I'd love to answer any additonal question that you might have!

Last update: December 22, 2020