Mainboards

Most Desktop or Server mainboards work with OmniOS.
If there are problems, they are mostly related to unsupported or badly supported disk or nic controller

Form Factor
ITX: avoid for storage system as memory and slots are too limited and they are quite expensive
uATX: perfect for a most systems up to 32 GB RAM
ATX or EATX: for professional usage (dualcore, RAM up to TB range, up to 7 slots for pci-e cards)

Chipset
Intel makes it quite easy. If you want a solid and future proof system, you should look at ECC RAM
and vt-d if you think about virtualisation. This usually requires a server class chipset and a Xeon
as desktop chipsets lacks these features as well as enough slots to expand a system.

CPU
Storage performance is mostly not CPU but RAM limited, so a smaller CPU like the i3 is ok.
I3 and Celeron G4400 also offers ECC support so this is a good option for a smaller NAS.
I5/I7 adds vt-d but removes ECC capability, mostly not e real option
Xeon is the Winner if you look for ECC and VT-d and performance

RAM
Socket 1151 is limited to 64 GB RAM. Socket 2011 has much higher limits.
The rule is the more RAM the faster is your system.

Network adapter
use Intel all the way, avoid Realtek
If you intend to use 10G Ethernet, look for a board with 10G onboard, cheaper than an extra card
With a modern storage server use 10G example Intel X540 whenever possible (as fast as local storage)

SAS controller
Use LSI and HBA (raidless IT mode, avoid Raid 5/6 models)
Onboard LSI HBA is cheaper than extra cards

For a storage server with up to 6 slots, Intel 6G Sata in AHCI mode is as fast as SAS 6G.

Brand
I prefer SuperMicro at first place but Asrock has some very good offers as well.

napp-it 01.06.2025