Personal tools


Remediating - bug


The Difficult Process of Remediating Embedded Systems 

The following is a discussion of the intrinsic difficulties associated with remediating embedded systems. It is a first-cut discussion that has been commented on by Paul a Y2K technical advisor at an automated automobile factory in Australia. I have identified his major comments with his first name only, to emphasize that the comments represent his personal opinion and not necessarily the position of his employer. His minor comments appear in "( )" parentheses. Future revisions will explore and clarify his comments.

"Embedded systems" have many attributes different from what we normally call computers including, but not limited to:

  • Packaged in an enclosure that does not look like a computer. 
  • Usually no keyboard or display, so special test equipment may be required (Available) to test it. 
  • Usually involve "real-time" operations, which places constraints on operating systems used and application development, programmer and other skills required.
  • Often use obscure operating systems, for which compilers may not be readily available. (example being 6805 Motorola) 
  • Programs are often loaded into ROMs or PROMS and after software is remediated, the repair process will then require a repair action to replace that integrated circuit(s). 

Paul's comment: This is just about impossible with the short amount of time and resources available

  • Few copies of each application, usually farmed out to 3rd parties who may be difficult to find. Source code usually lost so there is nothing available to remediate. 

Paul's comment: Yes I agree.

  • Compliance can be claimed by vendor without regard to the fact that the item may be non-compliant in its system context. 

Paul's comment: Yes there must be a worldwide or countrywide standard that each vendor must adhere to

  • Often interface to computers where a system interface non-compliance is often manifested. 
  • Tendency for old systems to hang around on the factory floor much longer than PCs would persist in the office. (Meaning technology can be quite old.) 
  • Also suffer from the tm_year problem that will cause the software to fail even if the application has no calendar function, does no date-based calculations and only does time stamping. Tm_year is a function that returns system date expressed as "years - 1900", so 1998 will return two-digit 98 and 2003 will return 103, which can cause overflows in the code, if the programmer did not write his code to accept the 3-digit value as well as the 2-digit. 
  • Reporters, managers and other non-technical people confused by the misnomers "embedded chips" and "non-compliant chips". Rarely does the "embedded system" consist of one chip. Semiconductor manufacturers never ship non-compliant chips as they supply hardware and the problem is always in software.. Semiconductor manufacturer cannot fix the problem. The "embedded system" manufacturer or his customer must remediate the software and reload new software, by whatever means required by that "embedded system". . The software can be loaded in many ways, ROMs being one of these many ways. If the ROM happens to be electrically re-programmable, such as an EEPROM, it can be re-programmed by perhaps carrying the circuit board to a suitable test set, otherwise a PROM will have to be replaced as a physical repair action. 

Paul's comment: This is the big one that I'm facing. No code, no programmer with the computer language specifics, no way to access the code in the EPROM and if I do manage to access it, then, what language was it written in? If I reassemble it in memory what operating system will I need ,what debugger do I have to get ,how do I know if the "opcodes" have been recreated correctly to produce the correct code. All "rtc’s", if not a 4 digit chip roll over to 00, so my service report my not be forthcoming from the system because its out of date by 100 years.

  • An IT person would be totally lost if he had to remediate embedded systems. He would have to learn how to work with and understand a physical system, learn the art of real-time programming, learn new languages and compilers, learn "embedded systems" testing techniques and be willing to get his hands dirty on the shop floor. They are usually programmed by engineers, who have a good understanding of what the real-time control system does. 

Paul's comment: Sing it brother !!!!!!

  • One more thing. It may take a team of disciplines to work any given "embedded systems" problem. This may consist of one or more engineers, test technicians, programmers, the vendor of the "embedded system" hardware, the semiconductor manufacturer that supplied the ROMs, repair technicians etc. Also the special test equipment, test procedures and test equipment may be lost and have to be re-created. 

Paul's comment: yes this is the major problem and I don't think that many people have there heads around it yet ,when they do there will be a lot of problems world wide as people are promised the world and then don't deliver.

  • Also, every system will be different and involve different considerations, which may involve all of the categories of problems mentioned above or some subset of them. The people that designed the system may be long gone and the easiest solution by far will be to replace it with a new system that provides the needed functionality. This is a long process because you would essentially be designing a new system. 

Paul's comment: Guess what time it takes to develop a new system (as you know), so this is just about out of the question.

So there are many, many, many factors that make the embedded systems problem very costly and difficult to deal with. Many of us concerned with "embedded systems" emphasize that software loaded into some kind of ROM (read only memory) is the source of error, not the hardware.  Such distinction may prevent adoption of the usually false hope that the simplistic process of "searching the factory floor for non-compliant chips and ordering a replacement part" will solve the majority of "embedded systems" problems. Although this may, on occasion, solve a problem, usually the remediation process is much more complex. Even if the "embedded system" software appears in a mask programmable ROM from the semiconductor factory, one can’t order a replacement part. It usually must be created.







Archived CPSR Information
Created before October 2004

Sign up for CPSR announcements emails


International Chapters -

> Canada
> Japan
> Peru
> Spain

USA Chapters -

> Chicago, IL
> Pittsburgh, PA
> San Francisco Bay Area
> Seattle, WA
Why did you join CPSR?

I want to be part of the solution. I think CPSR is working to guide society in the proper creation and use of technology.