Notes for OOPSLA 2006 Workshop

Escaped from the Lab: Software Practices in Large Organisations

1. Experiences

I have some interesting experiences over the years working in the "Escaped from the Lab" environment. The experiences have been from a combination of sources: promoting new tools, working as a mercenary analysis or developer, and doing reengineering work on an existing project.

I've been working for 25 years in various jobs in AT&T and Lucent Technologies -- everything from factory automation to training to language technology.

My experiences:

• C++ -- the ultimate escaped-from-the-lab programming language
  • In its early history, it was a programming language and coding environment that you wouldn't wish on your enemies, no less your friends.
  • Cryptic error messages from cfront; sometimes you would need to look at the generated C code to figure out what was wrong.
  • But, it was easy for experienced C folks to dabble -- interoperability with existing C code was a primary design consideration.
  • C++ recognized the fact that most software development follows the technique of "modify something that already exists".
  • Good libraries and useful code examples can make up for a mediocre compiler. Therefore, a lot of the early consulting work involved creating useful and efficient library classes.
• software development tools
  • My experience with many tools -- escaped-from-the-lab tools exacerbate the "fool with a tool" problem.
    • For example: A test generation tool will help you write a lot of tests, but it makes you lazy about evaluating whether you are missing some important test scenarios.
  • "Research tools" almost always have awful human factors (lack of informative user messages, "undo" functionality, or a simple usage model).
  • Tools integration can be a nightmare.
    • Example: Research folks don't understand the "configuration management" of most software professionals, so a cool research code transformation tool might completely reformat a piece of existing code -- when you check it back into CVS, SourceSafe, or ClearCase, it appears that every line has changed.
• discovery cost issues
  • The non-research recipients of escaped-from-the-lab software are busy people -- no time to read the documentation (if it exists), not enough time to understand the limitations of the product.
  • Dealing with external mandates: What does a busy person do when they are told by managers that they have to use product A from research? They figure out a way to use one or two simple scenarios -- using only a small part of the product's real functionality.
  • I overheard someone at OOPSLA say "I never use a tool that doesn't have an O'Reilly book".
• exploring in your off hours...
  • AT&T had a great internal resource -- Exptools (experimental tools) -- a bunch of volunteers who wrote and ported tools to a number of environments, and a simple distribution mechanism to update the tools on a large number of far-flung server computers.
  • This gave a lot of people a chance to try out tools and expand their comfort zone -- especially useful were browser tools (cscope), editors, languages (C++, perl, python), and documentation tools (doxygen). The fact that the tools were pre-built integrated, and had locally-written documentation helped lower the obstacles to getting started.

Discovery costs issues and the ability to extract useful things from legacy code are important considerations in product development based on escaped-from-the-lab systems.

• Some developers have a low tolerance for novelty. They want their development tasks to be as straightforward as possible.
• In some cases, an escaped-from-the-lab system is an application or library that must be modified or extended, and the code is written in an unfamiliar paradigm.
  • There will be some extra discomfort -- either learning to read the existing code and to modify it without changing the overall paradigm, or converting the existing code's structure (imperfectly) to a familiar form.
• A less extreme case -- a new development project needs to "borrow" some pieces of an escaped-from-the-lab system. (The new system might need to reuse some algorithms or extract some key data structures and functions.)
  • This should be easier -- but there can still be problems understanding the "legacy environment". For example, the developers of the legacy system might make certain assumptions about the scenarios in which their code would be used. It isn't always an easy task to verify that the code is being reused correctly.

2. Ideas and conclusions

• Researchers aren't necessarily good software architects. Research tools are often "under-architected" -- they might demonstrate a neat idea, but the developer might not be interested in the "-ilities": interoperability, understandability, extensibility, performance, robustness.
• Researchers are reluctant to give up control. They want to be able to make direct updates to the software package that is their "research vehicle" -- to perfect their ideas and extend their "prototypes" without having to wade through any of the extra code bloat that may have been added to get to production-quality software.
• Researchers need to be aware of when their "prototypes" are moving into a second phase -- when it makes sense to think more about things like performance, reliability, and security.
• Product groups may need to assign some fraction of staff to working ahead a couple of releases -- doing the "discovery" work that may make it possible to integrate in some new research software. If everyone on the development team is doing "heads down development" on the current release, the product will never make any big leaps forward -- it will be in a slow incremental mode forever.
• Software products that are escaped-from-the-lab may need a hybrid architecture -- a stable core plus experimental plug-ins. This maximizes the opportunity for continued innovation but still allows for parts to be "product quality".