Rochester Memory Hardware Error Research Project

Our research focuses on the characteristics of memory hardware errors and their implications on software systems. A plethora of research works can be found on memory fault tolerance. Often times researchers use accelerated tests in their controlled environments to collect data. We have taken a different approach to conduct field tests that monitor computers in real time and record the actual errors happening on them.

Memory errors can be categorized as soft errors and hard errors. Soft errors are also known as transient errors, which refer to those errors that are caused by environmental factors only and would not have lasting effects on memory hardware. Hard errors (or permanent errors) include those that are due to real hardware defects. We discovered that in each of our measured environments, the soft error rates are orders of magnitude lower than previously reported on a per Mbit basis. In the server farm at Ask.com, we found quite a few hard errors. We made some error rate/pattern analysis in our USENIX ATC'07 and HotDep'07 papers. Our USENIX ATC'10 paper uses the collected data to predict the whole memory system failure rate/pattern given different ECC schemes and maintenance strategies.

Publications:

• Renzo Bustamante, Rouhuang Zheng, and Michael C. Huang, "Efficient Selective Multi-Channel Memory Protection". In SRC TECHCON, Austin, TX, September 2015. PDF version.
  
  
• Xin Li, Michael C. Huang, Kai Shen, and Lingkun Chu, "A Realistic Evaluation of Memory Hardware Errors and Software System Susceptibility". In Proc. of the USENIX Annual Technical Conference (USENIX ATC'10), Boston, MA, June 2010. PDF version.
  
  
• Xin Li, Kai Shen, Michael C. Huang, and Lingkun Chu, "A Memory Soft Error Measurement on Production Systems". In Proc. of the USENIX Annual Technical Conference (USENIX ATC'07), Pages 275-280, Santa Clara, CA, June 2007. PDF version.
  
  
• Xin Li, Michael C. Huang, Kai Shen, and Lingkun Chu, "An Empirical Study of Memory Hardware Errors in A Server Farm". In the Third Workshop on Hot Topics in System Dependability (HotDep'07), Edinburgh, UK, June 2007. PDF version.
  
  

Project Members:

Error Data Download:

• Machine ID: anonymized machine name;
• First seen: when the error was first observed;
• Last seen: when it was last observed;
• Times: how many times the error in that ECC word was reported during our whole monitoring period;
• Address: the system address of the error, a.k.a. physical memory address;
• Row/Column/Bank: row/column/bank address of the error, translated from the system address;
• Syndrome: error syndrome, which can be used to determine which bits are wrong within an ECC word.

Some necessary detailed facts you should notice:

• Our monitoring started on Nov 30, 2006.
• The memory hardware scrubs the memory continuously. It records the errors discovered in some special purpose registers. At any time, the registers can only hold the info about two errors, and before the info is cleared by a user-level program, further error info will be discarded. Therefore, in any period of time, there may be more errors present in the system than what the registers have recorded.
• We poll the memory controller regularly (typically 3 times a day) for error information. After we collect the info, we clear the registers so that the hardware can continue to work.

We are a participant of USENIX computer failure data repository. You can find more failure data in addition to ours following this link.

Support:

This project is also supported in part by the Semiconductor Research Corporation (SRC) under Contract No. 2013-HJ-2405.