Mastering the Art of Test and Debug

Introduction

The Art
Despite the necessity and the common practice of test and debug there seems to be no way to learn these skills. Test and debug has generally been relegated to the area of engineering 'art' somewhat similar to RF analog engineers who cut a wire, scratch a trace, or bend a short antennae with a deft pinch of the fingers. (Example of Eastern Airlines System Programmer job.) The art can then be seen as an overall personal approach to the subject.

The Philosophy
To begin to get a handle of what goes on during test and debug sessions we should start to look at an overall philosophy that good test and debug engineers, at least in part and mostly subconsciously, subscribe to.

The Mastery
The mastery of test and debug is then a combination of knowledge, personal discipline, and the internalization of philosophical concepts.

Types of Test & Debug Tasks

• Initial Prototype Test This is the first hardware opportunity to integrate hardware and software, correct gross design problems, and test out design alternatives. The test plan at this point should cover the major system specification areas.
• Pre-production Test & Debug During this phase the hardware and software should represent what will be in production. This is the time for a well constructed and thoroughly executed test plan. All areas of hardware and software integration must be tested. Problems that are found and fixed during this phase may necessitate repeating some or all of the integration test performed to date in order to cover possible problems introduced during the fix.
• Production Bugs Bug reports that are returned from the field must first be investigated to find the affected area. Typically bug reports will be so general that it may be impossible to duplicate. When enough information has been collected a method to reliably duplicate the bug must be designed. Very often the act of reliably duplicating the bug narrows the possibilities so much that the cause is readily apparent.
• ECO Test & Debug ECOs, whether from bug reports or design modifications, must be thoroughly tested to insure that the changes will not adversely affect other areas.

Test & Debug Philosophy

The Seven Principles

The 'bug' is your prey. You must learn its habits by careful observation, then hunt it down and capture it.

• Choose your battle ground. Define and control your own test and debug setting.
• Discard everything that is unnecessary. Simplify the system under test.
• Commit yourself to solving the problem. "No" is not the right answer. Leaving bugs in a product should be unthinkable for an engineer. Only management and marketing should make decisions to release products with known bugs.
• Relax and abandon yourself. Let go of all preferences. Your attachments to specific design implementations will hinder your progress.
• When all looks hopeless...take a coffee break. Take a break and let go of your personal interests. Turning your mind off for awhile will renew your energy.
• Compress time. Let your mind work outside of time. Discover ways to shorten the time between failures by modifying the test software or other system parameters.
• Never push yourself to the front. Don't jump to conclusions and then publicize them.

The Three Personal Attributes

The success of the debug process lies with you. Your temperament and impecability is all that stands between you and the task. These three personal attributes are the most important factors that you have direct and complete control over within yourself.

• Never take yourself seriously. Taking yourself seriously is the surest way to be misled. Your own ego will lead you to assumptions such as: "It can't be my fault so I'll look elsewhere." or "I know this must be the problem so there is no need to verify it." Have no points to defend. This will make you flexible and open minded.
• Have endless patience. Loosing your patience and often your temper cloud the sobriety you need to tackle the problem. Occasionally you will fall upon the problem quickly without effort but you should always be prepared to go about the methodical and sometimes time consuming way. Quitting, taking short cuts, or making unwarranted assumptions are the most common consequences of impatience.
• Have an endless capacity to improvise. This maneuver is one of the most important. You are setting a trap for the bug. If it eludes your trap you must improvise another more clever one. Very often you will be required to set these traps with inadequate tools. Do not, however, allow your improvisation to become the project !

Test & Debug Methods

• Test & Debug Methods
  • Initial Bring Up
    • Prototype Board Requirements
      • PLDs & FPLGs in sockets.
      • Firmware in sockets or emulated.
      • Processors in sockets.
      • Large memory banks in sockets.
      • Easy to access test points.
      • Commonly used test points.
        • Processor Status
        • Processor Memory Control Signals
        • Clocks
    • Power & Ground
    • Proper Supply Voltages
    • Clocks
    • Partitioning the Design
  • Assumptions
    • System Software
    • Firmware
    • Board Level Hardware
    • PLDs & FPLGs
    • ASICs
    • 3rd Party ICs
      • Beta Versions
      • Production
      • Shrinks of Ics
        • Linear
        • Redesign
    • Power Supplies
  • Simplifying the System
    • System Software
    • Firmware
    • External Inputs
    • PLDs & FPGAs for testing
  • Test Firmware/Software
  • Narrowing the Possibilities
• System Integration Considerations
  • Hardware
  • Firmware
  • Software
  • External I/O
• Tools
  Assemble all tools ahead of time. Nothing is more frustrating than having your concentration and thread of thought broken by having to chase down a crucial tool.
  • Schematics & all ECOs
  • Bare PCB
  • Software and Firmware Listings - Source and assembly with address map.
  • PLD and FPGA Listings
  • Logic Probe
  • DVM
  • Logic Analyzer
  • Digital Storage Scope
  • Inter-interment Triggering