UBOOT_00_01 - PCIe Passthru

Owned by Fei Wu
Last updated: Sept 26, 2023
2 min read

About

This is a project related to SE_01_005 - QEMU PCIe passthru on x86 hosts.


When enabling PCIe passthru on x86 hosts, the physical memory range of 3GB-4GB is reserved for this purpose.

If the RAM memory size is larger than 1GB, it's split into low and high memory, e.g, for a 3GB RAM space:

  • low memory - 2GB to 3GB
  • high memory - 4GB to 6GB


However, u-boot is not able to handle this situation and failed as follows:


The command line:

qemu-system-riscv64 -machine virt -nographic -m 3072 -smp 4 \
        -bios /usr/lib/riscv64-linux-gnu/opensbi/generic/fw_jump.elf \
        -kernel /usr/lib/u-boot/qemu-riscv64_smode/uboot.elf \
        -netdev user,id=vnet,hostfwd=:127.0.0.1:2223-:22 -device virtio-net-pci,netdev=vnet \
        -device vfio-pci,host=01:00.0 -device vfio-pci,host=01:00.1 \
        -drive file=ubuntu-22.04.1-preinstalled-server-riscv64+unmatched.img,if=virtio,format=raw

The results:

U-Boot 2022.01+dfsg-2ubuntu2.3 (Nov 24 2022 - 19:40:06 +0000)

CPU:   rv64imafdch_zicbom_zicboz_zicsr_zifencei_zihintpause_zawrs_zca_zcd_zba_zbb_zbc_zbs_sstc_svadu
Model: riscv-virtio,qemu
DRAM:  Unhandled exception: Store/AMO access fault
EPC: 0000000080201738 RA: 0000000080210f20 TVAL: 00000000ff739f90

Code: b823 06b2 bc23 06b2 b023 08b2 b423 08b2 (b823 08b2)


resetting ...
System reset not supported on this platform
### ERROR ### Please RESET the board ###

Root Cause

Looking at the faulting address TVAL: 00000000ff739f90, this is in the range reserved for PCIe passthru, it should be be accessed here.


In u-boot, it reads only one range to setup ram_base/size:

This is the output, see 'Ram top' which is limited to 4GB:

fdtdec_setup_mem_size_base: Initial DRAM size 140000000
initcall: 00000000802168ba
Monitor len: 000C7CD0
Ram size: 140000000
Ram top: FFFFFFFF




So u-boot should not use the 4G-1 as ram top without any checking, the solution could be doing the check first.

Testing

With the attached u-boot and patched qemu for PCIe passthru, here is the test result, and the OS will bootup successfully.


qemu-system-riscv64 -machine virt -nographic -m 3G -smp 4 \

  -bios /usr/lib/riscv64-linux-gnu/opensbi/generic/fw_jump.elf \

  -kernel /home/wufei/nvme/u-boot/u-boot.bin \

  -device vfio-pci,host=01:00.0 -device vfio-pci,host=01:00.1 \

  -drive file=ubuntu-22.04.1-preinstalled-server-riscv64+unmatched.img,if=virtio,format=raw


U-Boot 2023.10-rc1-00323-gb1a8ef746f-dirty (Aug 08 2023 - 16:59:13 +0800)


CPU:

rv64imafdch_zicbom_zicboz_zicsr_zifencei_zihintpause_zawrs_zfa_zca_zcd_zba_zbb_zbc_zbs_sstc_svadu

Model: riscv-virtio,qemu

DRAM:  1 GiB (effective 3 GiB)

Core:  22 devices, 10 uclasses, devicetree: board

Flash: 32 MiB

Loading Environment from nowhere... OK

In:    serial,usbkbd

Out:   serial,vidconsole

Err:   serial,vidconsole

No working controllers found

Net:   No ethernet found.

Working FDT set to bef21340

Hit any key to stop autoboot:  0

=> bdinfo

boot_params = 0x0000000000000000

DRAM bank   = 0x0000000000000000

-> start    = 0x0000000100000000

-> size     = 0x0000000080000000

DRAM bank   = 0x0000000000000001

-> start    = 0x0000000080000000

-> size     = 0x0000000040000000

flashstart  = 0x0000000020000000

flashsize   = 0x0000000002000000

flashoffset = 0x0000000000000000

baudrate    = 115200 bps

relocaddr   = 0x00000000bf743000

reloc off   = 0x000000003f543000

Build       = 64-bit

current eth = unknown

eth-1addr   = (not set)

IP addr     = <NULL>

fdt_blob    = 0x00000000bef21340

new_fdt     = 0x00000000bef21340

fdt_size    = 0x0000000000001a80

lmb_dump_all:

 memory.cnt = 0x2 / max = 0x10

 memory[0]      [0x80000000-0xbfffffff], 0x40000000 bytes flags: 0

 memory[1]      [0x100000000-0x17fffffff], 0x80000000 bytes flags: 0

 reserved.cnt = 0x4 / max = 0x10

 reserved[0]    [0x80000000-0x8007ffff], 0x00080000 bytes flags: 0

 reserved[1]    [0xbdf1d000-0xbfffffff], 0x020e3000 bytes flags: 0

 reserved[2]    [0xbef1fcc0-0xbfffffff], 0x010e0340 bytes flags: 0

 reserved[3]    [0x100000000-0x17fffffff], 0x80000000 bytes flags: 0



=> md 0x100000000 4

100000000: 00000000 00000000 00000000 00000000  ................

=> mw 0x100000000 0xabcd 4

=> md 0x100000000 4

100000000: 0000abcd 0000abcd 0000abcd 0000abcd  ................


Project Scope

Fix this problem on u-boot and upstream

Status

Dependency

None


Development

COMPLETED

URL: NA
Development TimelineQ3 2023
Upstreaming

ONGOING

URL:
Upstream version

Contacts
Fei Wu 

Fei Wu 


Updates