Reliability Features

Intel claims no less than 20 new RAS features for the new Xeon, most of them borrowed from the Itanium. Some of the RAS features are for the most paranoid of IT professionals. Let's face it, who has experienced a server crash that was caused by a bad CPU? For each CPU failure there must be a million failures caused by buggy software. So we are not too concerned if a competing CPU lacks "hot physical CPU board" swapping, and it is reasonable to think that most IT professionals—even those with mission critical applications—will agree. The most paranoid people usually have the highest budgets, as the mission critical applications they manage could cost them their job if they go down. Not to mention that the company they work for might lose millions of dollars. So those people tend to favor a very long list of reliability features.

All ironic remarks about paranoid people aside, most of these RAS features make a lot of sense even for the "down to earth" people, the rest of us. Memory does fail a lot more than CPUs. According to Google research, 8% of the DIMMs see one correctable error per year, and 0.22% have uncorrectable errors. These machines can have up to half a Terabyte (!) of RAM, and with 32 to 64 DIMMs an uncorrectable error is conceivable. So it is no surprise that most of the RAS features try to cope with failing DRAM chips. Also as the number of VMs that you consolidate on one machine increases, the risk of a bad VM bringing the complete host machine down increases.

The idea behind the Machine Check Architecture is that errors in memory and L3 cache are detected before they are actually "used" by the running software. A firmware based memory scrubber constantly checks ("patrols") for unrecoverable errors, errors that ECC cannot correct. Those errors will make the (ESX) hypervisor create a purple screen—which is in most cases much worse than the famous blue screen—to make sure your data does not get corrupted.

With MCA in hardware and support in both firmware and the hypervisor, data errors are transmitted to the hypervisor's error handler before they cause havoc. The memory location is placed in quarantine (poisoned data containment) and the CPU will not use that address again. The software handler can then retry to get the data, and as a result the hypervisor keeps running. This "recover" mechanism can of course only work if the error is created by the occasional glitch and not by bad hardware.

So the basic idea behind these increased reliability features is that the more memory you have, the higher the chances that an occasional glitch occurs and thus the more features like demand and patrol scrubbing and recovery from single DRAM device failure are handy. You will need something better than simple ECC. The same is true for QPI. As the number of Nehalem EX CPUs and the speed of QPI links increases, the chances for bad addresses or bad data increases as well.

Nehalem EX Overview The Uncore Power of the Nehalem EX
Comments Locked


View All Comments

  • klstay - Thursday, April 15, 2010 - link

    I agree. Being able to use all the DIMM slots in the R810 with only half the CPU sockets populated is a neat trick, and I do like having up to 16 drive bays in the R910, but overall the latest IBM 3850 is much more flexible than either of those systems. From a 2 socket 4 cores each system with 32GB RAM up to an 8 socket 8 cores each system with 3TB RAM. Barring some big surprises at HPs announcement in a couple of weeks IBM will be the one to beat in Nehalem EX for the foreseeable future.
  • Etern205 - Thursday, April 15, 2010 - link

    The AMD Opteron 6128 isn't $523.
    It's $299.99!

    (credited to: zpdixon @ DT for providing the link)
  • yuhong - Tuesday, June 15, 2010 - link

    "but when a dual-CPU configuration outperforms quad-CPU configurations of your top-of-the-line CPU, something is wrong. "
    Remember Xeon 7100 vs Xeon 5300?

Log in

Don't have an account? Sign up now