• The article was written by Ford Morris for my Discord server.
  • With his permission I publish the article on my blog with some edits.
  • The original document is available here: PMEM Budget Build Info Doc.

So there’s a problem right now with DDR4 memory prices, and trying to spec a budget workstation build with LGA 2011 v3, LGA 2066, or LGA 3647 is really difficult right now. A single stick of 2666mHz ECC memory on ebay is ~$20-25 for 8gb, ~$40-45 for 16gb, ~$100 for 32gb, or ~$170 for a 64gb LRDIMM OR RDIMM (2133 mHz and 2400 mHz pricing is a little better though).

But there’s one secret way to get huge memory pools without breaking the bank, Intel Optane Persistent Memory DCPMM modules (Not to be confused with Intel Optane Memory, a feature of the Optane SSDs). These are DIMM modules that occupy ram slots, have huge capacities, much better speeds and durability than NAND Flash, and in memory mode, the system recognizes the Optane DIMMs as your actual memory, while treating your DRAM as an L4 cache.

Currently, the first gen DDR4-2666 Intel Optane memory modules, compatible with Xeon Scalable 2nd Generation, cost

So the problem now is getting a cheap platform to use Optane memory. To do this, you need both a Gold or Platinum Second Gen Xeon Scalable CPU (Xeon Gold 52 or 62, Xeon Platinum 82), and a motherboard that explicitly supports it.

CPU

The CPU is simple because Xeon Scalable Second Gen has some great budget options, despite not being as good of pricing as the first gen Xeon Gold 6138. Here is a list of the top candidates I found, in no particular order

Important:

  • Most motherboards require BIOS mod to work with CL CPUs.
  • Most motherboards require VRM mod to work with >205 W TDP CPUs.
  • Many motherboards require VRM mod to work with >165 W TDP CPUs.

Motherboard

Next is the motherboard platform, where we are looking for LGA 3647 motherboards that support it. Most of these were built for First Gen Xeon Scalable (Skylake) and need a bios update to support Second Gen Xeon Scalable (Cascade Lake).

The options we have are certain workstations like the Dell T7820, T7920, HP Z6 G4, Z8 G4, and Lenovo P920, along with some different Supermicro X11 motherboards I found that are made for standard ATX form factors. The other important information about these motherboards pertaining to Optane is the amount of DIMM slots.

Xeon Scalable 2nd Gen has 6 memory channels and up to 12 DIMM slots per socket. Intel generally wants one Optane DIMM and one DRAM DIMM per channel for optimal performance, but vendors can bend the rules a bit when using less than 12 DIMM Slots.

The rule of thumb is this: If you have 2 slots for some memory channels, place the Optane PMEM in the slot closer to the CPU and DRAM in the further slot for those channels and only DRAM for all other channels with just one slot. If your motherboard only has 6 memory slots and it supports PMEM, refer to the motherboard specific manual, or the manuals for the 6 slot workstations and motherboards I linked down below.

Generally, you will get:

  • 6 PMEM 6 DRAM with 12 slots.
  • 4 PMEM 6 DRAM with 10 slots.
  • 2 PMEM 6 DRAM with 8 slots.
  • 2 PMEM 4 DRAM with 6 slots.
  • 4 PMEM 2 DRAM with HP Z6 G4.
Here is an official install guide by Intel: [Configuring Intel® Optane™ DC Persistent Memory for Best Performance Intel Business](https://youtu.be/MtP_iUZfMWM?si=Vy70BzMJLfTLcUHO).

Potential builds

Here are some of the motherboards or workstations and details regarding building with them. I listed all the ones I know, including bad deals in case you can find a better price.

HP Z8 G4

  • Price: ~$400.
  • CPU: 2 sockets.
  • RAM: 12 slots per socket.
  • PSU: 1125W or 1450W.
  • Link: eBay.

This listing requires you to buy the 1st and 2nd slot HP CPU Coolers for $90 each or risk a $50 Dynatron aftermarket cooler for each. Yes the first and second slot have differently designed HP coolers.

When filling all memory slots, put PMEM in the white slots and DRAM in the black slots. For all possible configurations, refer to pages 21 and 22 of INTEL® OPTANE™ DC PERSISTENT MEMORY: CONFIGURATION AND SETUP ON HP Z6 G4 AND Z8 G4 WORKSTATIONS.

HP Z6 G4

  • Price: ~$370.
  • CPU: 1 socket + optional riser board with 2nd socket.
  • RAM: 6 slots per socket.
  • PSU: 1000W.
  • Link: eBay.

PMEM configuration is weird but good on this workstation, allows for 2 PMEM 4 DRAM or 4 PMEM 2 DRAM per CPU. Refer to page 20 for configurations INTEL® OPTANE™ DC PERSISTENT MEMORY: CONFIGURATION AND SETUP ON HP Z6 G4 AND Z8 G4 WORKSTATIONS.

Lenovo P920

  • Price: ~$600.
  • CPU: 2 sockets.
  • RAM: 8 slots per socket.
  • PSU: 1400W.
  • Link: eBay.

I only know this is supported because the P920 is listed under two Optane PMEM related vulnerability firmware fixes. I think you should put PMEM in the 2 black slots and DRAM in the 6 white slots for each socket, refer to X11DPH-T manual linked on next page for reference.

Here Lenovo lists P920 as receiving this PMEM related security patch. Here Lenovo lists P920 as receiving this PMEM related security patch.

You will have to also buy a CPU cooler for the second CPU if you want to add it, I didn’t investigate too hard into the P920 though.

Lenovo P720

Does not support PMEM (?), but regardless its a good deal for an LGA 3637 build.

Dell Precision T7920

  • Price: ~$650
  • CPU: 2 sockets.
  • RAM: 12 slots per socket.
  • PSU: 1100W or 1400W.
  • Link: eBay.

I know it supports Optane PMEM because it says so here Dell Precision 7920 Tower Owner’s Manual.

It seems that you would fill the 6 black slots with PMEM and 6 white slots with DRAM, judging from pictures, but I can’t find a guide. Refer to the HP Z8 G4 manual link for configurations.

Dell Precision T7820

  • Price: ~$380.
  • CPU: 1 socket + optional riser board with 2nd socket.
  • RAM: 6 slots per socket.
  • PSU: 950W.
  • Link: eBay, eBay.

I know it supports Optane PMEM because it says so here, Dell Precision 7820 Tower Owner’s Manual. No specific guide on memory, try referencing either the HP Z6 G4 or Supermicro 6 slot guides.

Supermicro X11SPM-F

  • Price: ~$200.
  • CPU: 1 socket.
  • RAM: 6 slots per socket.
  • Size: mATX.
  • Link: eBay.

I know the board supports Optane PMEM because it says so here: X11SPM-F.

Supported PMEM configuration (2 PMEM 4 DRAM per CPU) at page 15 here: MEMORY CONFIGURATION FOR X11 UP/DP/MP MOTHERBOARDS.

Supermicro X11DPH-T

  • Price: ~$279.
  • CPU: 2 sockets.
  • RAM: 8 slots per socket.
  • Size: E-ATX.
  • Link: eBay. (https://ebay.us/m/wVEkrR)

I know the board supports Optane PMEM because it says so here: X11DPH-T.

Supported PMEM configuration (2 PMEM 6 DRAM per CPU) at page 18 here: MEMORY CONFIGURATION FOR X11 UP/DP/MP MOTHERBOARDS.

Out of the deals I found, I think I would get a Xeon Gold 6240, HP Z6 G4, 2 x 128GB PMEM, and used 3090 or used quadros with NVLink for all my PyTorch shenanigans. The max memory per socket for Z6 G4 is apparently 384GB with normal DRAM, but considering they allow for a 4 PMEM 2 DRAM configuration, I am guessing PMEM doesn’t play by the same rules but I am respecting it anyways.

Final notes

Some things to remember if you are actually going through with a build like this:

  • Update the BIOS to support Cascade Lake CPUS and PMEM before installing.
  • Look for explicit PMEM support for the motherboard itself, not based on a product category or chipset; a lot of very similar Supermicro motherboards do not support PMEM.
  • Do not confuse Intel Optane Memory technology, the SSD feature, with PMEM.
  • Figure out how to set the PMEM to Memory Mode rather than Application Direct or Mixed Mode.
  • Application Direct mode is where the PMEM shows as a storage block device rather than memory, and an app specific API can be used to address information in it without having to copy to DRAM first.
  • Mixed mode is when some PMEM is in Memory Mode while some is in Application Direct Mode.
  • Look for max memory limits with each workstation or motherboard.
  • Get a T-30 Torx bit screwdriver for installing CPUs, potentially a torque screwdriver that helps you apply the exact correct amount of force.
  • Here is Intel’s guide and some Level 1 Techs discussion.
  • How to Install an Intel® Xeon® Processor into an LGA3647 Socket.
  • LGA 3647 torque specs.
  • Have fun!