123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449450451452453454455456457458459460461462463464465466467468469470471472473474475476477478479480481482483484485486487488489490491492493494495496497498499500501502503504505506507508509510511512513514515516517518519520521522523524525526527528529530531532533534535536537538539540541542543544545546547548549550551552553554555556557558559560561562563564565566567568569570571572573574575 |
- EDAC - Error Detection And Correction
- Written by Doug Thompson <norsk5@xmission.com>
- 7 Dec 2005
- EDAC was written by:
- Thayne Harbaugh,
- modified by Dave Peterson, Doug Thompson, et al,
- from the bluesmoke.sourceforge.net project.
- ============================================================================
- EDAC PURPOSE
- The 'edac' kernel module goal is to detect and report errors that occur
- within the computer system. In the initial release, memory Correctable Errors
- (CE) and Uncorrectable Errors (UE) are the primary errors being harvested.
- Detecting CE events, then harvesting those events and reporting them,
- CAN be a predictor of future UE events. With CE events, the system can
- continue to operate, but with less safety. Preventive maintenance and
- proactive part replacement of memory DIMMs exhibiting CEs can reduce
- the likelihood of the dreaded UE events and system 'panics'.
- In addition, PCI Bus Parity and SERR Errors are scanned for on PCI devices
- in order to determine if errors are occurring on data transfers.
- The presence of PCI Parity errors must be examined with a grain of salt.
- There are several add-in adapters that do NOT follow the PCI specification
- with regards to Parity generation and reporting. The specification says
- the vendor should tie the parity status bits to 0 if they do not intend
- to generate parity. Some vendors do not do this, and thus the parity bit
- can "float" giving false positives.
- [There are patches in the kernel queue which will allow for storage of
- quirks of PCI devices reporting false parity positives. The 2.6.18
- kernel should have those patches included. When that becomes available,
- then EDAC will be patched to utilize that information to "skip" such
- devices.]
- EDAC will have future error detectors that will be integrated with
- EDAC or added to it, in the following list:
- MCE Machine Check Exception
- MCA Machine Check Architecture
- NMI NMI notification of ECC errors
- MSRs Machine Specific Register error cases
- and other mechanisms.
- These errors are usually bus errors, ECC errors, thermal throttling
- and the like.
- ============================================================================
- EDAC VERSIONING
- EDAC is composed of a "core" module (edac_mc.ko) and several Memory
- Controller (MC) driver modules. On a given system, the CORE
- is loaded and one MC driver will be loaded. Both the CORE and
- the MC driver have individual versions that reflect current release
- level of their respective modules. Thus, to "report" on what version
- a system is running, one must report both the CORE's and the
- MC driver's versions.
- LOADING
- If 'edac' was statically linked with the kernel then no loading is
- necessary. If 'edac' was built as modules then simply modprobe the
- 'edac' pieces that you need. You should be able to modprobe
- hardware-specific modules and have the dependencies load the necessary core
- modules.
- Example:
- $> modprobe amd76x_edac
- loads both the amd76x_edac.ko memory controller module and the edac_mc.ko
- core module.
- ============================================================================
- EDAC sysfs INTERFACE
- EDAC presents a 'sysfs' interface for control, reporting and attribute
- reporting purposes.
- EDAC lives in the /sys/devices/system/edac directory. Within this directory
- there currently reside 2 'edac' components:
- mc memory controller(s) system
- pci PCI control and status system
- ============================================================================
- Memory Controller (mc) Model
- First a background on the memory controller's model abstracted in EDAC.
- Each 'mc' device controls a set of DIMM memory modules. These modules are
- laid out in a Chip-Select Row (csrowX) and Channel table (chX). There can
- be multiple csrows and multiple channels.
- Memory controllers allow for several csrows, with 8 csrows being a typical value.
- Yet, the actual number of csrows depends on the electrical "loading"
- of a given motherboard, memory controller and DIMM characteristics.
- Dual channels allows for 128 bit data transfers to the CPU from memory.
- Some newer chipsets allow for more than 2 channels, like Fully Buffered DIMMs
- (FB-DIMMs). The following example will assume 2 channels:
- Channel 0 Channel 1
- ===================================
- csrow0 | DIMM_A0 | DIMM_B0 |
- csrow1 | DIMM_A0 | DIMM_B0 |
- ===================================
- ===================================
- csrow2 | DIMM_A1 | DIMM_B1 |
- csrow3 | DIMM_A1 | DIMM_B1 |
- ===================================
- In the above example table there are 4 physical slots on the motherboard
- for memory DIMMs:
- DIMM_A0
- DIMM_B0
- DIMM_A1
- DIMM_B1
- Labels for these slots are usually silk screened on the motherboard. Slots
- labeled 'A' are channel 0 in this example. Slots labeled 'B'
- are channel 1. Notice that there are two csrows possible on a
- physical DIMM. These csrows are allocated their csrow assignment
- based on the slot into which the memory DIMM is placed. Thus, when 1 DIMM
- is placed in each Channel, the csrows cross both DIMMs.
- Memory DIMMs come single or dual "ranked". A rank is a populated csrow.
- Thus, 2 single ranked DIMMs, placed in slots DIMM_A0 and DIMM_B0 above
- will have 1 csrow, csrow0. csrow1 will be empty. On the other hand,
- when 2 dual ranked DIMMs are similarly placed, then both csrow0 and
- csrow1 will be populated. The pattern repeats itself for csrow2 and
- csrow3.
- The representation of the above is reflected in the directory tree
- in EDAC's sysfs interface. Starting in directory
- /sys/devices/system/edac/mc each memory controller will be represented
- by its own 'mcX' directory, where 'X" is the index of the MC.
- ..../edac/mc/
- |
- |->mc0
- |->mc1
- |->mc2
- ....
- Under each 'mcX' directory each 'csrowX' is again represented by a
- 'csrowX', where 'X" is the csrow index:
- .../mc/mc0/
- |
- |->csrow0
- |->csrow2
- |->csrow3
- ....
- Notice that there is no csrow1, which indicates that csrow0 is
- composed of a single ranked DIMMs. This should also apply in both
- Channels, in order to have dual-channel mode be operational. Since
- both csrow2 and csrow3 are populated, this indicates a dual ranked
- set of DIMMs for channels 0 and 1.
- Within each of the 'mc','mcX' and 'csrowX' directories are several
- EDAC control and attribute files.
- ============================================================================
- DIRECTORY 'mc'
- In directory 'mc' are EDAC system overall control and attribute files:
- Panic on UE control file:
- 'panic_on_ue'
- An uncorrectable error will cause a machine panic. This is usually
- desirable. It is a bad idea to continue when an uncorrectable error
- occurs - it is indeterminate what was uncorrected and the operating
- system context might be so mangled that continuing will lead to further
- corruption. If the kernel has MCE configured, then EDAC will never
- notice the UE.
- LOAD TIME: module/kernel parameter: panic_on_ue=[0|1]
- RUN TIME: echo "1" >/sys/devices/system/edac/mc/panic_on_ue
- Log UE control file:
- 'log_ue'
- Generate kernel messages describing uncorrectable errors. These errors
- are reported through the system message log system. UE statistics
- will be accumulated even when UE logging is disabled.
- LOAD TIME: module/kernel parameter: log_ue=[0|1]
- RUN TIME: echo "1" >/sys/devices/system/edac/mc/log_ue
- Log CE control file:
- 'log_ce'
- Generate kernel messages describing correctable errors. These
- errors are reported through the system message log system.
- CE statistics will be accumulated even when CE logging is disabled.
- LOAD TIME: module/kernel parameter: log_ce=[0|1]
- RUN TIME: echo "1" >/sys/devices/system/edac/mc/log_ce
- Polling period control file:
- 'poll_msec'
- The time period, in milliseconds, for polling for error information.
- Too small a value wastes resources. Too large a value might delay
- necessary handling of errors and might loose valuable information for
- locating the error. 1000 milliseconds (once each second) is the current
- default. Systems which require all the bandwidth they can get, may
- increase this.
- LOAD TIME: module/kernel parameter: poll_msec=[0|1]
- RUN TIME: echo "1000" >/sys/devices/system/edac/mc/poll_msec
- ============================================================================
- 'mcX' DIRECTORIES
- In 'mcX' directories are EDAC control and attribute files for
- this 'X" instance of the memory controllers:
- Counter reset control file:
- 'reset_counters'
- This write-only control file will zero all the statistical counters
- for UE and CE errors. Zeroing the counters will also reset the timer
- indicating how long since the last counter zero. This is useful
- for computing errors/time. Since the counters are always reset at
- driver initialization time, no module/kernel parameter is available.
- RUN TIME: echo "anything" >/sys/devices/system/edac/mc/mc0/counter_reset
- This resets the counters on memory controller 0
- Seconds since last counter reset control file:
- 'seconds_since_reset'
- This attribute file displays how many seconds have elapsed since the
- last counter reset. This can be used with the error counters to
- measure error rates.
- Memory Controller name attribute file:
- 'mc_name'
- This attribute file displays the type of memory controller
- that is being utilized.
- Total memory managed by this memory controller attribute file:
- 'size_mb'
- This attribute file displays, in count of megabytes, of memory
- that this instance of memory controller manages.
- Total Uncorrectable Errors count attribute file:
- 'ue_count'
- This attribute file displays the total count of uncorrectable
- errors that have occurred on this memory controller. If panic_on_ue
- is set this counter will not have a chance to increment,
- since EDAC will panic the system.
- Total UE count that had no information attribute fileY:
- 'ue_noinfo_count'
- This attribute file displays the number of UEs that
- have occurred have occurred with no informations as to which DIMM
- slot is having errors.
- Total Correctable Errors count attribute file:
- 'ce_count'
- This attribute file displays the total count of correctable
- errors that have occurred on this memory controller. This
- count is very important to examine. CEs provide early
- indications that a DIMM is beginning to fail. This count
- field should be monitored for non-zero values and report
- such information to the system administrator.
- Total Correctable Errors count attribute file:
- 'ce_noinfo_count'
- This attribute file displays the number of CEs that
- have occurred wherewith no informations as to which DIMM slot
- is having errors. Memory is handicapped, but operational,
- yet no information is available to indicate which slot
- the failing memory is in. This count field should be also
- be monitored for non-zero values.
- Device Symlink:
- 'device'
- Symlink to the memory controller device
- ============================================================================
- 'csrowX' DIRECTORIES
- In the 'csrowX' directories are EDAC control and attribute files for
- this 'X" instance of csrow:
- Total Uncorrectable Errors count attribute file:
- 'ue_count'
- This attribute file displays the total count of uncorrectable
- errors that have occurred on this csrow. If panic_on_ue is set
- this counter will not have a chance to increment, since EDAC
- will panic the system.
- Total Correctable Errors count attribute file:
- 'ce_count'
- This attribute file displays the total count of correctable
- errors that have occurred on this csrow. This
- count is very important to examine. CEs provide early
- indications that a DIMM is beginning to fail. This count
- field should be monitored for non-zero values and report
- such information to the system administrator.
- Total memory managed by this csrow attribute file:
- 'size_mb'
- This attribute file displays, in count of megabytes, of memory
- that this csrow contains.
- Memory Type attribute file:
- 'mem_type'
- This attribute file will display what type of memory is currently
- on this csrow. Normally, either buffered or unbuffered memory.
- Examples:
- Registered-DDR
- Unbuffered-DDR
- EDAC Mode of operation attribute file:
- 'edac_mode'
- This attribute file will display what type of Error detection
- and correction is being utilized.
- Device type attribute file:
- 'dev_type'
- This attribute file will display what type of DRAM device is
- being utilized on this DIMM.
- Examples:
- x1
- x2
- x4
- x8
- Channel 0 CE Count attribute file:
- 'ch0_ce_count'
- This attribute file will display the count of CEs on this
- DIMM located in channel 0.
- Channel 0 UE Count attribute file:
- 'ch0_ue_count'
- This attribute file will display the count of UEs on this
- DIMM located in channel 0.
- Channel 0 DIMM Label control file:
- 'ch0_dimm_label'
- This control file allows this DIMM to have a label assigned
- to it. With this label in the module, when errors occur
- the output can provide the DIMM label in the system log.
- This becomes vital for panic events to isolate the
- cause of the UE event.
- DIMM Labels must be assigned after booting, with information
- that correctly identifies the physical slot with its
- silk screen label. This information is currently very
- motherboard specific and determination of this information
- must occur in userland at this time.
- Channel 1 CE Count attribute file:
- 'ch1_ce_count'
- This attribute file will display the count of CEs on this
- DIMM located in channel 1.
- Channel 1 UE Count attribute file:
- 'ch1_ue_count'
- This attribute file will display the count of UEs on this
- DIMM located in channel 0.
- Channel 1 DIMM Label control file:
- 'ch1_dimm_label'
- This control file allows this DIMM to have a label assigned
- to it. With this label in the module, when errors occur
- the output can provide the DIMM label in the system log.
- This becomes vital for panic events to isolate the
- cause of the UE event.
- DIMM Labels must be assigned after booting, with information
- that correctly identifies the physical slot with its
- silk screen label. This information is currently very
- motherboard specific and determination of this information
- must occur in userland at this time.
- ============================================================================
- SYSTEM LOGGING
- If logging for UEs and CEs are enabled then system logs will have
- error notices indicating errors that have been detected:
- EDAC MC0: CE page 0x283, offset 0xce0, grain 8, syndrome 0x6ec3, row 0,
- channel 1 "DIMM_B1": amd76x_edac
- EDAC MC0: CE page 0x1e5, offset 0xfb0, grain 8, syndrome 0xb741, row 0,
- channel 1 "DIMM_B1": amd76x_edac
- The structure of the message is:
- the memory controller (MC0)
- Error type (CE)
- memory page (0x283)
- offset in the page (0xce0)
- the byte granularity (grain 8)
- or resolution of the error
- the error syndrome (0xb741)
- memory row (row 0)
- memory channel (channel 1)
- DIMM label, if set prior (DIMM B1
- and then an optional, driver-specific message that may
- have additional information.
- Both UEs and CEs with no info will lack all but memory controller,
- error type, a notice of "no info" and then an optional,
- driver-specific error message.
- ============================================================================
- PCI Bus Parity Detection
- On Header Type 00 devices the primary status is looked at
- for any parity error regardless of whether Parity is enabled on the
- device. (The spec indicates parity is generated in some cases).
- On Header Type 01 bridges, the secondary status register is also
- looked at to see if parity occurred on the bus on the other side of
- the bridge.
- SYSFS CONFIGURATION
- Under /sys/devices/system/edac/pci are control and attribute files as follows:
- Enable/Disable PCI Parity checking control file:
- 'check_pci_parity'
- This control file enables or disables the PCI Bus Parity scanning
- operation. Writing a 1 to this file enables the scanning. Writing
- a 0 to this file disables the scanning.
- Enable:
- echo "1" >/sys/devices/system/edac/pci/check_pci_parity
- Disable:
- echo "0" >/sys/devices/system/edac/pci/check_pci_parity
- Panic on PCI PARITY Error:
- 'panic_on_pci_parity'
- This control files enables or disables panicking when a parity
- error has been detected.
- module/kernel parameter: panic_on_pci_parity=[0|1]
- Enable:
- echo "1" >/sys/devices/system/edac/pci/panic_on_pci_parity
- Disable:
- echo "0" >/sys/devices/system/edac/pci/panic_on_pci_parity
- Parity Count:
- 'pci_parity_count'
- This attribute file will display the number of parity errors that
- have been detected.
- =======================================================================
|