The Importance of Testing Technology for the NGI. Doug Montgomery Manager, Internetworking Technologies Group National Institute of Standards and Technology Building 820, Room 445 820 West Diamond Avenue Gaithersburg, MD 20899 USA Email: dougm@nist.gov Voice: +1-301-975-3630 Fax: +1-301-590-0932 The Importance of Testing Technology for the NGI. I. Requirements for testing The Next Generation Internet represents not just a simple "speed up" of existing networks, but rather an enormous increase in their diversity and protocol complexity. Over this increasingly complicated network infrastructure will run an equally diverse mix of traffic, with many new and sophisticated demands for network resources and large-scale coordination. The explosive growth in system complexity represented by this combination of network, protocol, and application-level demands, will multiply the opportunities for network inefficiency, error, and outright failure. The old IETF credo of "rough consensus and running code" being the only quality assurances needed to guide the development of new Internet technology will not scale well in the future. The complexities of and interdependencies among envisioned NGI control systems and services (e.g., signaling, routing, naming, security) will defy analysis by simple means. Likewise, the cost to build testbeds of the extent necessary to test critical design and deployment issues will be prohibitive in the early stages of technology development. Ensuring the timely realization of the NGI requires a new focus on techniques and tools to test and evaluate new networking technology at all stages of its development and deployment. Test and measurement technology should become a common part of the design and specification process, and should be integrated into, and evolve with, the implementation and deployment of the network itself. I. Challenges for testing We detail here some of the principal challenges for testing in the Next Generation Internet. Diversity of networks -- The old metaphor of the "information superhighway" may give an impression of some rough uniformity of traffic "speed", but precisely the opposite is the case. The course of a single packet may pass over mobile links, ISDN lines, local area nets, satellite links, and gigabit pipes . Diversity of resource demands -- New protocols and applications will make much more sophisticated demands on network resources. Applications such as interactive audio and video may have hard real-time (latency) constraints, and large, but widely varying demands on network bandwidth. And unlike in the past, where available network resources were more or less uniformly handed out to all traffic, new quality of service guarantees make for strong distinctions between the types of service requested (and paid for!) by different clients. Scale of coordination -- Unlike the original Internet, where most communication involved simply the establishment and maintenance of a connection between two fixed endpoints, new multicast protocols allow for the creation of "collaborations" with hundreds or even thousands of geographically-dispersed intercommunicating clients. Coherently managing such collaborations typically involves firm restrictions on allowable latency, and sophisticated mechanisms for coordination and floor control. Adaptive protocols and networks -- To deal with these new requirements, network researchers are increasingly turning to technologies such as adaptive protocols and active networks. While these likely represent the best approach for making intelligent use of available network resources, they themselves present new testing challenges. Adaptive protocols can be notoriously difficult to test: their "correct" behavior cannot be defined statically or often even in any simple deterministic fashion; and adaptive protocols at different levels or on different systems often interact poorly with each other in ways not detectable with only stand-alone testing. Especially with active networks, testing-relevant events may occur anywhere en route in response to local transient conditions -- hence testing hooks cannot simply be at connection endpoints, but must be present throughout the net. III. Approach to testing With the vast diversity in network capabilities and protocol and application demands, and the range of strategies available to adaptive protocols and active networks, exhaustive case testing to prove protocol "correctness" is no longer feasible, or even definable. Instead, a strategy which provides ongoing testing and monitoring support for protocols at all stages of development is required. This is provided through an environment which offers a continuum of test support through simulation, emulation, and live testing. Simulation -- is the usual strategy for initial testing and comparison of new network protocols. Intelligent use of simulation testing is especially necessary for assessing new adaptive protocols, to have some hope of proving them under a statistically significant range of possible network conditions. Simulation is useful as part of a larger test environment as well, to generate test loads and conditions to be used elsewhere in the test. Network simulation systems must grow in scope and capabilities to keep pace with (or preferably keep ahead of) the analogous growth in the real networks they attempt to simulate. Such sophisticated network simulations need to be available as a common resource for developers of new protocols and network-aware applications, to help accelerate their testing and refinement. Emulation -- provides the bridge between simulation and live testing. A network emulation environment provides the means to test real code in place, subjected to a variety of simulated and live loads and network conditions. Network emulation is particularly valuable in allowing the reproduction of the vast diversity of the real network infrastructure in a relatively small, controlled test environment. Combined with simulation systems providing a rough picture of other network traffic, an emulation system can be used to provide a manageable approximation to a large-scale test scenarios within a typical laboratory setting. Live testing and monitoring -- is of course always needed at the lab scale, but especially with new networks and protocols, must be provided for as a continuing part of the real network. A weakness of the original Internet was the relative lack of provisions for ongoing testing and monitoring. This was less of a problem when the Internet community was small and defects could fairly readily be identified and isolated. Now, however, with the explosive growth of the network, and the increase in the sophistication of the demands made on it, this situation is no longer acceptable. Comprehensive means for the ongoing monitoring of network conditions and in-field assessment of the behavior of new protocols and applications are required; these must be integrated into all new networks, and develop as the network and protocol and application demands on them develop.