The Difficult Process of Remediating Embedded Systems 

The following is a discussion of the intrinsic difficulties associated with remediating embedded systems. It is a first-cut discussion that has been commented on by Paul a Y2K technical advisor at an automated automobile factory in Australia. I have identified his major comments with his first name only, to emphasize that the comments represent his personal opinion and not necessarily the position of his employer. His minor comments appear in "( )" parentheses. Future revisions will explore and clarify his comments.

• Packaged in an enclosure that does not look like a computer. 
• Usually no keyboard or display, so special test equipment may be required (Available) to test it. 
• Usually involve "real-time" operations, which places constraints on operating systems used and application development, programmer and other skills required.
• Often use obscure operating systems, for which compilers may not be readily available. (example being 6805 Motorola) 
• Programs are often loaded into ROMs or PROMS and after software is remediated, the repair process will then require a repair action to replace that integrated circuit(s). 

• Few copies of each application, usually farmed out to 3^rd parties who may be difficult to find. Source code usually lost so there is nothing available to remediate. 

• Compliance can be claimed by vendor without regard to the fact that the item may be non-compliant in its system context. 

• Often interface to computers where a system interface non-compliance is often manifested. 
• Tendency for old systems to hang around on the factory floor much longer than PCs would persist in the office. (Meaning technology can be quite old.) 
• Also suffer from the tm_year problem that will cause the software to fail even if the application has no calendar function, does no date-based calculations and only does time stamping. Tm_year is a function that returns system date expressed as "years - 1900", so 1998 will return two-digit 98 and 2003 will return 103, which can cause overflows in the code, if the programmer did not write his code to accept the 3-digit value as well as the 2-digit. 
• Reporters, managers and other non-technical people confused by the misnomers "embedded chips" and "non-compliant chips". Rarely does the "embedded system" consist of one chip. Semiconductor manufacturers never ship non-compliant chips as they supply hardware and the problem is always in software.. Semiconductor manufacturer cannot fix the problem. The "embedded system" manufacturer or his customer must remediate the software and reload new software, by whatever means required by that "embedded system". . The software can be loaded in many ways, ROMs being one of these many ways. If the ROM happens to be electrically re-programmable, such as an EEPROM, it can be re-programmed by perhaps carrying the circuit board to a suitable test set, otherwise a PROM will have to be replaced as a physical repair action. 

• An IT person would be totally lost if he had to remediate embedded systems. He would have to learn how to work with and understand a physical system, learn the art of real-time programming, learn new languages and compilers, learn "embedded systems" testing techniques and be willing to get his hands dirty on the shop floor. They are usually programmed by engineers, who have a good understanding of what the real-time control system does. 

• One more thing. It may take a team of disciplines to work any given "embedded systems" problem. This may consist of one or more engineers, test technicians, programmers, the vendor of the "embedded system" hardware, the semiconductor manufacturer that supplied the ROMs, repair technicians etc. Also the special test equipment, test procedures and test equipment may be lost and have to be re-created. 

• Also, every system will be different and involve different considerations, which may involve all of the categories of problems mentioned above or some subset of them. The people that designed the system may be long gone and the easiest solution by far will be to replace it with a new system that provides the needed functionality. This is a long process because you would essentially be designing a new system. 

So there are many, many, many factors that make the embedded systems problem very costly and difficult to deal with. Many of us concerned with "embedded systems" emphasize that software loaded into some kind of ROM (read only memory) is the source of error, not the hardware.  Such distinction may prevent adoption of the usually false hope that the simplistic process of "searching the factory floor for non-compliant chips and ordering a replacement part" will solve the majority of "embedded systems" problems. Although this may, on occasion, solve a problem, usually the remediation process is much more complex. Even if the "embedded system" software appears in a mask programmable ROM from the semiconductor factory, one can’t order a replacement part. It usually must be created.

http://www.techstocks.com/~wsapi/investor/Subject-18990