Advanced Simulation Technology inc.
ASTi Application Note 65

ASTI's Proposed Remediation Program to Mitigate the Industry-wide "Capacitor Plague" (Electrolytic Capacitor Failure)

Attention all ASTi customers - Notice of Potential Equipment Failure
This notice is to advise all ASTi customers of an industry-wide issue that may affect the long-term reliability of ASTi Telestra platforms shipped between January 2004 and January 2006. Note that this problem does not appear to pertain to any DACS platforms. We must also emphasize that this alert is being given as a precautionary measure, at this time we have no data to indicate an unusual incidence of premature failure for these shipments.

Introduction

About twelve months ago, ASTi became aware of industry reports of what was reported to be an isolated problem due to the use of defective Taiwanese capacitors on a limited number of Intel motherboards used in one or two shipments of ASTi Telestra platforms. We promptly notified the affected customers and worked with them to mitigate the impact. One customer observed some fault indications from the problem and we worked with them to replace the defective components on all of the affected platforms.
It now appears that the issue may be far more pervasive and affects a wide range of electronic products over many years of production.

Why is ASTi the only one who is highlighting this as an issue?

Obviously, it is not just ASTi platforms that are affected but all electronic assemblies in the global marketplace that are vulnerable. The difference is that most corporations seem to be following a procedure of "don't ask, don't tell" hoping that their customer base will just regard the numerous product failures as isolated incidents, and will replace the failing units as part of business as usual without taking offense.
Since we expect most ASTi platforms to exhibit product lifetimes substantially greater than one or two years, and because ASTi appears to be one of the few companies that feels a sense of obligation to their customers, we are choosing to draw your attention to the impending problem.

What is Capacitor Plague?

Capacitor Plague is when a flawed electrolytic capacitor prematurely deteriorates and dries up. Once this happens the gas in the capacitor will expand with heat and build up a great pressure inside the capacitor. When the pressure builds up, the capacitor will expand (bulge), leak, burst or explode. During this time the capacitance will vary widely and may even short out. See Attachment B for some pictorial examples.
System errors or symptoms you may notice are:
  • Motherboard fails to POST
  • Memory Test Fails
  • System randomly and/or constantly reboots itself
  • Fails to fully boot (or even install) Operating System
  • System randomly and frequently freezes
  • Random & frequent 'Blue Screens of Death'
  • Hard freeze under heavy drive activity
  • CPU temps abnormally higher than usual under typical or less load
  • CPU & other system voltages are erratic or far out of tolerances
  • Resetting the system after a freeze and the system will not repost (You have to completely power down then power back up.)
Note: These symptoms are not exclusive to capacitor failure.

More Reading

Attachment A shows the notice discretely publicizing the issue on an obscure part of the Intel website at: http://www.intel.com/cd/channel/reseller/asmo-na/eng/tech_reference/box_desktop/int_inst_info/dsk_factory_notes/193414.htm

What do we know?

First of all, we believe the problem is restricted to the following:
  • ASTi Telestra systems shipped between 2004 and 2006
  • Telestra systems with an Intel motherboard model number D865GBF and D865GCLC
  • BIOS identifier BF86510A.86A.xxxx.Pxx
  • Capacitor manufacture and model number, Nichicon HN (M) Note that no other model capacitors under this manufacturer are affected.
  • Failures may be experienced after several years of powered operation, while visible distortion of the capacitors occurs after 6-12 months of operation.
If your system meets all of the above requirements, it may be at risk for capacitor failure at some time in the future. Based on all currently available information, all other ASTi equipment including DACS, RIUs, etc. do not use any at-risk capacitors and have not exhibited any signs of capacitor failure.
Unfortunately, despite our rigorous Configuration Management system at ASTi, the way that most electronic assembly vendors such as Intel have dealt with the problem (i.e., surreptitiously moving their product bill-of-materials onto new capacitors without changing revision levels of the assemblies), makes it impossible for us to track which of the Telestras shipped during the specified period of time are affected.
It is important to note that the failure of these systems is unlikely to be imminent. Evidence to date suggests that systems do not have premature failures until almost four years of powered operation.
Let us stress again that this problem is not an ASTi-unique problem nor is it restricted to Intel products, but may affect ALL electronic assemblies produced in this time period.
Many electronic assemblies such as Motherboards, Power Supplies, Video Cards, etc. are experiencing High Electrolytic Capacitor Failure (Capacitor Plague). The Macintosh G4s and Dell Computers' GX280 systems have been affected, and a class-action suit has been launched against ABIT Computers. This is an industry-wide problem and no company is immune. In fact, the use of these defective capacitors has been so pervasive that we suggest that you check all of the KVM switches, routers, and power supplies in your trainer and then if incipient capacitor breakdown is observed, contact the vendor to see if they also have instituted a remediation procedure.
The reason that you are hearing from us about this issue is that ASTi has concluded that it is important to make our customers aware of the potential for experiencing abnormally high in-service failure rates over the next few years, whereas it appears most other suppliers have taken a "wait-and-see" approach.

What can we do about it?

You will be pleased to know that even though many of the affected Telestra systems are past the one-year warranty, ASTi is extending the coverage to accommodate the remediation of these platforms.
We are doing this because we feel the most prudent course of action is to replace the capacitors on all of the suspect platforms before any of them begin to fail. We feel that at this early stage, such a procedure can be accomplished with an orderly process of cycling equipment through the remediation process in a way that minimizes trainer downtime.
The primary remediation process will be to replace the at-risk motherboard with one that has been modified by ASTi to replace all the suspect capacitors. In some special cases we may find it more cost-effective for you to recycle equipment back through our facility or even for an ASTi soldering expert to come to site to replace capacitors in situ with ASTi labor at no cost, travel time and expenses covered by a support contract.
To this end, we are creating a sufficient inventory of replacement motherboards and will begin a process with each of our customers to cycle through their inventory. We will advise every customer the serial numbers of those platforms which we consider to be at highest risk of containing these defective components and which should be first in the remediation cycle. Once any such platforms are identified, we will work with you to establish the most cost-effective way to accomplish the update. Telestras that are approaching the four-year threshold and most at-risk of total failure will be given priority.

Replacement Process

All actions described below will be provided at no cost to the customer but will require a formal purchase order to ensure that all replaced motherboards are returned to ASTi. (See Attachment C).
However, if on-site actions are required travel time and expenses will be chargeable.
All actions must be coordinated with ASTi Operations (susan.grunmeier@asti-usa.com) in advance to ensure availability of parts and a prompt turnaround.
  1. Our preferred option will be to send a batch of remediated motherboards to the customer for replacement, along with a step-by-step installation guide for the customer's on-site technical personnel to effect the replacement. As the faulty boards are returned to us, ASTi will rotate through all of your platforms. This service will be provided at no cost provided that removed motherboards are sent back to ASTi for us to repair and recycle. We will of course also render any reasonable telephone or email support that we can to resolve any problems that you might encounter.

  2. Where skilled on-site personnel are not available, you will be requested to rotate a hot spare unit through the inventory while returning platforms or motherboards to ASTi for a prompt turnaround.

  3. In other cases where a large number of platforms are involved at a single site, it may be more appropriate for an ASTi engineer to travel to site and rectify all the platforms in one session.
In all cases, we will work with you to ensure we reach a cost-effective solution.

Attachment A: Intel Website Posting

Electrolytic Capacitor Leak on some 875 & 865 based products
Products Affected
Intel(R) Desktop Boards D865GBF, D875BZL, D865GLC, D845EPI, D865PERC, D865GRH, D865PERL, D845GVSR, D865PESO, D845GLVA, D865GWV, and Intel(R) Entry Server Board S875WP1-E.
Issue Description
During inspection and routine cleaning, it was discovered that some capacitors had residue on their surface. The safety vent releases built up vapor, leaving a residue on the surface of the capacitor. These electrolytic capacitors are used for bulk decoupling of VCCP. No functional failures were observed. Intel believes that the capacitors will not short out and there are no safety concerns.
Corrective Action
The inspection frequency of the materials has been increased and automated to avoid this occurring again. All affected stock has been purged and restocked with new, improved parts. All future Intel desktop boards that utilize this capacitor will be manufactured using the new part.
Recommendation
Any boards that actually experience failure because of this issue may be returned under the standard RMA process.

Attachment B: Pictorial Examples

Photo of motherboard with faulty capacitors
Photo 1: Note the leaking capacitors at the rear left. Here the leakage is from the top of the capacitor.
Photo of motherboard with faulty capacitors
Photo 2: A closer view of the leaking capacitor. Note the leaking capacitors on the left side.
Photo of motherboard with faulty capacitors
Photo 3: A side view of the leaking capacitor. Note the leaking capacitors at the back.

Attachment C: Return Procedure

This process is unique to the motherboard replacement program (it does not follow our standard RMA process). A written notice will be sent to the customer purchasing representative.
  1. Include a brief summary statement describing the nature of the problem. Attach the "Notice of Problem" letter.

  2. List the effected Telestras including: Telestra Serial Number, and Intel motherboard PN xxx, and warranty expiration date.

  3. Provide contact information required to initiate the motherboard return and replacement procedure. ASTi will not accept motherboards which are returned without prior approval.
Provide the information in "a" through "c" above using our secure, online submission form. DO NOT use this form for standard RMA requests; it is only for use in the Capacitor Plague remediation program.
  1. To initiate the return and replacement procedure, a PO must be issued to ASTi in the amount of $1000 for each effected Telestra. However, the net price with return of original motherboard is $0. Do not list the value on the shipping documentation.

  2. The original motherboard must be returned to ASTi in good physical condition (with no damage due to board removal and return shipping and handling) to receive core charge credit. After you submit the online form, you will receive a confirmation email with shipping instructions.

  3. Deliverable items include: new or remanufactured Intel motherboard, which meets or exceeds the performance specifications of the original part and a revised ASTi software license file.
Note: This offer does not extend the period of the original product warranty.