🐏 1.5 TiB of DDR3 in a 120€ server

This story took place in October 2026, with a CPU upgrade in early November.

I bought an Intel R2312GZ4GC4 for 120€. That got me twelve drive bays, a motherboard, caddies and power supplies. The board is an Intel S2600GZ with two CPU sockets and 24 DDR3 slots.

The 120€ covered the server only. The RAM and the later CPUs were extra, so please don’t ask me where to buy 1.5 TiB of memory for that price :P

It came with a single Xeon E5-2620 v1 (six cores, twelve threads) and a very reasonable amount of RAM. I ordered two E5-2650L v2 CPUs and some used 64 GB ECC modules. The plan was 512 GB, which already sounded like a funny amount of memory to put in this thing.

Then more modules arrived, and the question became whether we could fill every slot.

We could: 24 × 64 GiB, 1536 GiB total, 768 GiB per socket, at 1066 MT/s with ECC enabled. Getting there took reverse engineering Intel’s memory initialization, writing several BIOS patches, filming POST LEDs with a Pixel 8, decrypting a BMC diagnostic archive, and physically reseating a CPU after I’d spent far too long blaming everything else.

For this experiment I had Codex working on the server and, later, on a separate backup machine. It did most of the log collection, reverse engineering, assembly and patch testing. I was the human arm: moving DIMMs, changing jumpers, carrying USB sticks between machines, and asking it to dig harder. A lot harder.

The hardware

The boot disk was a 2 TB drive from a charity shop. It turned out to have 96 reallocated sectors, 104 pending sectors, and real unreadable-sector errors.

That matters later. Debugging a machine that keeps losing power is annoying enough without keeping the only copy of the investigation on its dying boot drive, so I moved the working session, the firmware branches and the external recorder to another server. After that, a failed POST could take down the Intel machine without taking our notes with it.

The memory was Samsung M386B8G70DE0-YH93, 64 GiB DDR3L-1333 ECC LRDIMM. These are load-reduced DIMMs, with a buffer sitting between the controller and the DRAM. Physically they’re 8Rx4, but with the rank multiplication setup that ended up working, each module presents two logical ranks to the CPU.

At first some modules failed initialization, and Linux only got whatever the BIOS had managed to bring up. An empty-looking inventory entry didn’t necessarily mean the slot was empty, either. Intel distinguishes “Not Installed” from “Failed/Disabled”, and the event log can keep the initialization failure around even after the BIOS has disabled the module.

We started tracking serial numbers across boots and running stressapptest with kernel MCE and ECC monitoring alongside. One module, serial ending in 26D1, stayed on the suspect list because of an MCE corrected error.

Eventually I put four suspect modules in together (because I dropped the other 3) and named them the four horsemen of suspicion. Their half-hour test passed. We also caught a few mistakes in our own bookkeeping, like calling a module untested when it had actually been sitting in a later successful run. With this many identical sticks, I ended up putting tape on them in different patterns so I could tell how far each one had got through the selection process.

The machine had other productive uses in this period. I was cold, so I asked how to load the CPUs. A friend suggested converting all of Europe’s OpenStreetMap data to OsmAnd format, since the converter is supposedly terrible with memory. So we tried that too. One of the failures traced back to a signed 32-bit overflow in a hash table’s capacity, which gave us a ridiculously small table and a loop. Germany threw its own converter exception after hours of work.

Buying more RAM does not fix every program, apparently.

Ten sticks, then 1-5-4-2

Eventually, still on the original single v1 CPU, the machine could boot ten 64 GiB modules. That’s 640 GiB installed, more than the 384 GB per processor in the Sandy Bridge specification. Eleven or twelve modules got me this instead:

1-5-4-2: Power Unit Failure.

The eight POST LEDs were usually around B7 or B8 when it happened.

Rearranging the ten modules made a simple total-capacity limit look unlikely. One arrangement that worked:

A1 A2       B1 B2 B3    ||||   C1 C2 C3       D1 D2

That’s five DIMMs in the A/B pair and five in C/D. Filling a whole six-DIMM pair could fail even when a larger total spread across both pairs worked. I didn’t track serial numbers for the early swaps, so it wasn’t a clean experiment, but the machine was clearly doing something stranger than “the BIOS refuses more than 640 GB”.

There was also a legitimate support question. Intel’s platform tables list 32 GB quad-rank LRDIMMs for the original E5-2600 generation, and 64 GB 8Rx4 modules for E5-2600 v2. So at the start we were asking a v1 memory controller to initialize a topology Intel never listed for it.

Moving to the v2 CPUs was worth doing, and the final full-capacity setup used both E5-2650L v2s. It didn’t make the problem go away on its own, though.

Blaming the power supplies

I tried running the server from PSU2 alone, with PSU1 unplugged but still in its bay. Twelve sticks still didn’t work.

We also found out that two installed PSUs aren’t necessarily sharing the load. In the standby arrangement, one could be supplying roughly 156 W while the other supplied zero. Disabling cold redundancy got them sharing at about 84/80 W. Adding the troublesome DIMM still failed.

Intel’s debug bundle had more in it than the ordinary event log. The main archive opens with a publicly documented password. The second archive, the BMC-only one, uses a different password.

Both archives contained SDR_GP.dat, with the same CRC32 and the same compressed length. Rookie mistake, so we went for a bkcrack known-plaintext attack. The second archive was eventually decrypted and we verified its file CRCs.

There was even a Delta PSU black-box section. Several of its fields decoded into convincing PMBus LINEAR11 measurements: mains around 230 V, sensible current and power, plausible fan speeds. Nothing very interesting came out of it. Snapshots like that are a poor substitute for an oscilloscope, which I didn’t have.

The BIOS settings rabbit hole

Meanwhile, the BIOS settings rabbit hole kept getting deeper.

Forcing DDR3-800 didn’t help. Trying 1.5 V at that speed made even the ten-DIMM configuration less reliable, so we went back to 1.35 V. Phase shedding was already disabled. Intel’s Sandy Bridge erratum BT147, about refresh staggering and instantaneous power, was interesting enough to look into, but we never proved it had anything to do with our later Ivy Bridge setup.

We looked at Memory Power Limiting, Multi-Threaded MRC, rank multiplication, electrical throttling (electrical trolling), SCLTT/DCLTT/SOLTT and MEMHOT. These options aren’t exposed in the BIOS, so we changed them from the EFI shell with a tool of our own.

Our first attempt at that was an EFI utility. One run required me to type an argument full of punctuation into a terminal with about ten visible lines, and ended with:

invalidString(InteriorNul(30))

At that point I banned elaborate commands. I wanted to type palle.efi and have the machine do everything by itself, backups and logs included. That’s how fuckbios.nsh was born, a menu with 231 BIOS controls.

We did get twelve modules to POST, showing 768 GiB. Later we reached 832, 896 and 960 GiB with increasingly strange arrangements. It wasn’t a clean sequence where one option bought exactly one improvement. Several things had changed along the way, and the machine could behave differently from one boot to the next.

Channel E was the other recurring problem. It never worked, so we started calling it EX and decided to ignore it for a while. The reduced goal became 21 sticks, 1344 GiB, with E completely empty: get the working channels populated first and deal with the apparently dead one afterwards.

Reloading the sensor definitions brought back 19 missing CPU2 sensors. That let us see the machine properly, though it doesn’t prove the sensor definitions had anything to do with the boot failure. 2N command rate, lower open-loop thermal quotas, a different MEMHOT mode and a very relaxed tFAW all failed to unlock the next troublesome slot.

We tried skipping part of memory initialization by disabling ECC. The firmware answered with INVALID DIMM POPULATION, because the resulting settings conflicted with its directory configuration, and I had to use the defaults jumper to recover.

After that reset, the old fifteen-DIMM configuration booted again. So our magical BIOS settings had probably been useless all along.

Patching the firmware

By now the investigation had moved past settings and into patching the firmware.

Intel’s capsule is signed, and modifying it invalidates the signature. We didn’t forge a signature or patch out the verifier. We installed through the board’s own BIOS Recovery path, which doesn’t check signatures. We kept the original images, every experimental branch, the launchers, rollback files and logs, because I wanted to be able to fork backwards instead of losing a version that happened to work.

The EFI shell didn’t like my SSD and wanted physical USB media. I didn’t have a good USB drive, so I plugged in a half-broken stick that I knew was only reliable for its first 64 MB.

Later, another apparent read failure made me suspect the stick again. Nothing was broken that time. It was a Volume Full error, because we had run out of space on the 64 MB.

The first useful patch was called MRCSEQ.

MRC is Intel’s Memory Reference Code, which initializes the memory controller and the DRAM before an operating system can use them. Around checkpoints B9 and BA, the stock code can launch work on several channels at once. We changed its grouping loop so those operations start one channel at a time.

The tests, polling, retries and rank coverage all stayed. Only the way work gets launched changed. In one test fixture, six multichannel starts became eighteen single-channel starts. The initial change replaced a ten-byte instruction span, and nine bytes actually differed once the checksum adjustment was counted.

That got sixteen DIMMs, 1024 GiB, through initialization, and a stressapptest run with a 991 GiB payload passed afterwards.

It also created an entirely predictable new problem: doing everything slowly made POST take a long time.

The BIOS arms the FRB2 watchdog to hard reset the machine after 360 seconds. It’s there to reboot a server that has locked up. The watchdog did exactly what it was told and reset a machine that was still working, so we disabled it.

The next patch, SBRUGNOLERIA.nsh, did a similar job for two earlier training paths by invoking the original runner separately for each selected channel. One detail needed care: the Intel routine uses its incoming argument slots as scratch space, so every call needs a fresh copy of the twelve arguments. And no, disabling Multi-Threaded MRC in Setup doesn’t give you the same serialization.

A better debugger than LEDs

We were making progress, but the LEDs were still a terrible debugger.

I filmed them in slow motion with a Pixel 8, and we argued over whether the last fraction of a frame showed B8, B9, or a mix of LEDs switching at slightly different times. My trusty Cisco serial cable would have helped, but I’d lent it to a friend.

The board’s UART can be carried over BMC Serial over LAN. So we added NETMRC, and later more instrumentation, to send binary records out of the UART, and recorded them over SOL from the backup machine.

Each record was 48 bytes, with a marker and a CRC32 so the decoder could pick valid frames out of the normal console text. It carried the checkpoint and subcode, a timestamp counter, the call site, the socket or target context, a register and a value. We could mark a write before it happened, mark its return, and record the readback.

Seventeen modules booted with A2 populated. Then G2 worked too and we had eighteen, 1152 GiB. My guess is that the logger slowed the machine down enough to fix it. I think bringing up all that memory at once caused a sudden load step of around 200 W, and the resulting voltage sag tripped the power-good circuitry. A friend jokingly suggested recapping the board, but slowing initialization down seemed a lot less destructive.

Adding one more module exposed the next limit. With the extra stick on CPU2’s side, the trace stopped where that socket switches into NORMAL mode. Moving the stick to A3 moved the failure to CPU1.

At that point the controller has trained the memory and is about to enter its normal operating state. The register involved is MCMTR, and the transition we saw was:

MCMTR: 0x0000000c -> 0x0000010c

For two of the CPU1 failures we had the same 51 register values before the transition. The trace ended after the marker that precedes the write, with no return and no readback, and the BMC logged a power-good failure.

That narrowed things down a lot, though without an oscilloscope I can’t point at the exact CPU cycle.

MEOWNBIOSFINAL, which was of course nowhere near final, added a verified nominal 100 ms delay before CPU2 entered NORMAL. It still failed.

Then came MICETTA. The idea was to avoid enabling a fully populated pair all at once: hold one channel disabled, enter NORMAL, then enable that channel after a nominal 1 ms delay.

That booted nineteen DIMMs, 1216 GiB. POST took about sixteen minutes, during which I was reportedly standing by with a fire extinguisher and coloring JEDEC diagrams with a children’s marker. Apparently this is the one weird trick Intel engineers don’t want you to know: Intel patent US10762006 describes the power control unit making MCMTR.chn_disable read-only once memory training finishes, and raising an MCE if anything tries to change it. That’s the exact bit MICETTA was flipping. The patent was filed in 2017 and our trick did boot, so I can’t say this 2013 Ivy Bridge-EP works that way, but it might explain why the trick went sideways later.

CATERR

With more memory, the machine produced CATERR.

Intel’s CATASTROPHIC ERROR is a great thing to read after hours of patching the memory controller, and only one step less funny than ARM Mali’s KABOOM and EUREKA errors. It doesn’t automatically mean the CPUs are physically ruined. It means something serious got reported, and we had to work out what had happened just before.

We tried to pull machine-check information and added more serial probes. BMC PECI access could return MCA_ERROR, which didn’t tell us much.

The firmware names became CONFESSA, CAT, LENTE and SCAMBIA. CAT stood for “Catastrophic AMD64 Training”.

The traces showed pattern loading and its busy polls completing. The last access we saw was the channel’s final pattern-control store, with selector 0xc00445c. CATERR followed about twenty seconds later.

In one experiment we moved the temporarily held channel from C to D. C then completed, and D became the last write with no recorded return. A3 had changed too, so it wasn’t a perfect comparison, but it looked like our channel-disable trick was leaving the controller unhappy. Sadly, we had to drop this nefarious path.

RESPIRO

Ghidra made the next attempt less speculative. We compared its decompilation against the real x86 instructions and Intel’s controller register documentation. The relevant code programs the refresh timings during B5, before the later NORMAL transition.

Two fields were particularly interesting:

Refresh staggering was already enabled. The checkbox being on didn’t mean the spacing matched the refresh cycle time.

The Intel routine programs these two fields separately. The stagger value comes from DIMM-dependent tables and a frequency index, and repeating identical descriptors doesn’t necessarily raise that maximum as you add modules. To check that dependency, we replayed the original instructions in eighteen synthetic cases instead of translating the algorithm and trusting our translation.

The synthetic descriptors can’t tell us which exact indices the physical boot selected. The captures from real boots do tell us the values that ended up in the controller: 329 and 80.

At 1066 MT/s, the memory clock is about 533 MHz. Eighty clocks is about 150 ns and 329 clocks is about 617 ns, so another logical rank’s refresh can start before the previous rank’s tRFC window has elapsed.

A schematic of permitted refresh overlap at the minimum spacing. This is not a captured command trace or a measurement of current.

Intel documents this as spacing between ranks on one channel. It doesn’t sequence whole channels or sockets. Intel also recommends a stagger interval less than or equal to tRFC, so 80 wasn’t automatically an illegal value. Allowing refresh overlap can be intentional.

Our experiment was to pick the conservative end of that documented range: set the spacing equal to the live tRFC, which reduces the overlap allowed on each affected channel.

I asked where the 329 had been guessed from. It hadn’t been guessed:

TCRFTP = 0x48a49040
tRFC = (TCRFTP >> 15) & 0x3ff
     = 329

The final helper reads the value at boot. There’s no magic 329 hardcoded in it.

We called this version RESPIRO.nsh, Italian for “breath”. It seemed a reasonable thing to offer a motherboard we’d been asking to do everything at once.

It also removed the temporary channel-disable mechanism and went back to the original single, unmasked NORMAL transition. The earlier training and B9/BA serialization, the diagnostics and the CPU2 delay stayed in.

The refresh helper is guarded for the geometry we observed: fully populated pairs of channels, each channel holding three matching 64 GiB DIMMs with two logical ranks. It validates all selected channels before changing any of them, preserves unrelated register bits, and checks the readback. If a write fails, it tries to restore the old value and halts, so the machine never continues into NORMAL with an unverified partial change.

This last step was small: a 516-byte helper, and 502 bytes changed relative to its immediate parent. The complete working BIOS differs from official stock by 8086 bytes in 602 separate spans. Those are two different comparisons. The downloadable patcher reproduces the whole accumulated result.

The result was 21 modules, 1344 GiB. Both sockets completed B9 and BA, and the BIOS showed the expected capacity at 1066 MT/s. Good job, Codex.

My best guess: this board, these memory controllers and these dense LRDIMMs just don’t have much margin when everything initializes at once.

The dead channel

That left the small matter of an entire memory channel that still looked dead.

E1 showed up as Failed/Disabled, with the initialization-failed and disabled codes 852C and 854C. Swapping in another tested module didn’t make the problem follow the stick. It stayed with E.

I suspected I had never seated CPU2 properly in the first place. The very first install, before any of these experiments, had frozen POST at checkpoint BA. Reseating it got past that hang, only for EX to vanish entirely. Having accepted that I apparently can’t mount an Intel CPU in fewer than three tries, I opened the retention levers and went again.

With the levers open there was a little play in the socket, which is normal. I also asked about the small hole at the edge of the heat spreader. After all the firmware experiments, a perfectly ordinary package vent hole had started to look suspicious.

With the CPU remounted, the same RESPIRO firmware, the same settings and the same other 21 modules booted with E1 working. 22 sticks, 1408 GiB.

Then we added E2 and E3.

24 sticks. 1536 GiB. All eight channels, three DIMMs per channel.

Did it actually work?

Both controllers were in NORMAL, ECC was enabled, and all eight channels read back a tRFC of 329 and a refresh spacing of 329. Command rate was 1N and tFAW was 20, so the extreme timing experiments from earlier aren’t what got the final machine working.

We also read the primary firmware back and compared it with RESPIRO. The protected recovery and bootblock regions were unchanged. I trust that a lot more than an updater printing “success”.

The final capacity looked like this:

A big number in the BIOS wasn’t enough, of course, so I told it to stress the whole thing.

stressapptest got a 1492 GiB payload, roughly 746 GiB resident on each socket, with no swapping. We also watched for ECC errors, machine checks and throttling, instead of just trusting stressapptest’s own verdict.

The plan was two hours. Partway through I got impatient and wanted to change the acoustics and boot options, so we stopped early. The workload ran for 3284.38 seconds, about 54 minutes and 44 seconds, and finished with PASS: zero incidents, zero application errors, and no ECC or current-boot machine-check errors recorded during the run.

How long it takes to boot? 24 minutes and 41 seconds from a Linux reboot command to the memory/POST prompt.

I wanted a lot of RAM. I got a lot of time to think about that decision before each boot.

Side quests and new CPUs

There were a few side quests left. A Fujitsu/LSI SAS2008 card had an annoying legacy Option ROM. With EFI Optimized Boot, the machine reaches Linux and mpt3sas initializes the card. Then I wanted quieter fans, because after finishing the loudest experiment in the room I suddenly remembered that ears exist.

In November I swapped the low-power CPUs for two E5-2695 v2 chips. The first boot lost B1/B2/B3 on CPU1 and showed 1344 GiB. Reseating CPU1 brought the channel back, on the same firmware, and all 1536 GiB returned.

Apparently the CPU sockets wanted a second appearance in the story.

A later check from Linux saw both 2695 v2s, 24 cores and 48 threads, all the memory, working PCIe, and no active throttling or ECC errors at that moment. Package temperatures were 48/54°C under the light workload it was running.

The server today in Proxmox: 48 threads on two E5-2695 v2s, 1.48 TiB of RAM.

Downloads

If you have the same machine and want to inspect or reproduce the BIOS, I’m publishing the patcher and the technical notes here:

• BIOS patcher ZIP
• Technical notes and source ZIP
• SHA256 checksums

The patcher only needs Python’s standard library. You give it the original R02.06.0007 capsule. It checks the size and SHA256 before touching anything, verifies the original bytes of every replacement, and checks the hash of the complete output at the end.

The original capsule’s SHA256 is:

c009095f2e1df95c256f2cdda03dd24b75351b91897acdcb79f7c05b23ad4e2d

The reproduced RESPIRO capsule’s SHA256 is:

d4d3c964ba13946d9c5ac8b205008b40279511013b96f9a087dfcd94d0be37ab

Once you’ve extracted the patcher and obtained the original firmware, the command is:

python3 respiro.py R02.06.0007.cap RESPIRO.CAP

The tools, patch code and documentation we wrote are dedicated to the public domain under the Unlicense. Intel’s firmware and other people’s material keep their own rights.

Modifying the capsule breaks its signature, so the normal updaters will reject it. You have to use the board’s hardware Recovery mode, which skips signature verification (the guide in the archive explains how). I didn’t bundle our flashing scripts. They were hasty hacks for my specific test build, so ask your agent to write you one.

The final firmware still contains the diagnostics and the deliberately slow initialization that produced this result. FRB2 has to be disabled separately, and the 24-minute boot is part of the deal. There is no if CATERR then nop, and nothing removes power-good, ECC or thermal protection.

Legal stuff

On the legal side, in Italy: Article 64-ter of Law 22 April 1941, no. 633 covers acts necessary for the lawful acquirer to use software as intended, expressly including error correction, under the conditions it states. Its third paragraph also covers observing, studying and testing a program while doing things the user is entitled to do. Article 64-quater provides a narrower decompilation exception, for obtaining information indispensable to making an independently created program interoperate with other programs, with conditions on access, scope, use and disclosure.

These come from EU law, Directive 2009/24/EC, Articles 5 and 6. They’re not blanket permission to modify any BIOS or to redistribute Intel’s firmware.

The primary texts are on Normattiva, Article 64-ter, Normattiva, Article 64-quater, and EUR-Lex, Directive 2009/24/EC.

Finally, to the fullest extent permitted by law, I decline responsibility for damage, lost data, an unbootable motherboard, or any other consequence of using this experimental software. It comes as-is, with no warranty. Read the full notice before accepting it. What worked on my pile of hardware is evidence about my pile of hardware.

The technical notes link the Intel S2600GZ/GL platform specification.

For now, there are 24 sticks of DDR3 in the server, Linux can use them, and I have a collection of cool EFI scripts.

MIAO.