Distributed Application Experience using the NSF vBNS Von Welch Senior Network Engineer National Center for Supercomputing Applications vwelch@ncsa.uiuc.edu (217) 244-3095 FAX: (217) 244-1987 605 E. Springfield Champaign IL 61820 In February of 1997 NCSA, at the University of Illinois, and WES, in Vicksburg Mississippi, ran a demonstration of a distributed application that connected virtual Chesapeake Bays running in CAVES (a 3- dimensional virtual environment - see http://www.evl.uic.edu/EVL/VR/CAVE.overview.html) at WES and NCSA. This application is part of the DOD modernization program and is composed of a database of environmental information about the bay and allows scientists at NCSA and WES to explore the bay together in real time. Although in this case there were only two users of the application, the application uses multicast and can handle an arbitrary number of users. NCSA has a direct connection to the National Science Foundation vBNS and it was used to carry traffic to iDREN where WES is connected. The two networks interconnect at multiple locations which is discussed later in this paper. The vBNS is a lightly-loaded OC-3 (155 mbit/s) ATM network. Performance tests run daily over the vBNS between FDDI-attached hosts at the NSF Supercomputing Centers show performance typically in the 60- 90 mbit/s range (depending on the two sites) - see http://www.vbns.net/reports/perf/daily. iDREN on the other hand is a heavily loaded DS3 ATM network that serves the Department of Defense sites for many of their wide area networking needs. The hosts at both WES and NCSA were SGI Onyxes. At NCSA the Onyx was connected to an Ethernet subnet connected to a router on NCSA's ATM OC-3 backbone, which is dual homed to the vBNS and NCSA's separate commodity Internet connection[RLB1]. The Onyx at WES was directly attached to their FDDI backbone, which connects to iDREN. The initial step of establishment of IP routing between the two sites over the vBNS involved approvals from NSF and iDREN policy makers. The vBNS AUP specifies that only meritorious traffic between approved sites and their collaborators can be carried - see http://www.nlanr.net/VBNS/vbns_aup.html. Initiating vBNS traffic between IDREN and NCSA required a route peering arrangement between the two networks and route acceptance at both sites (NCSA and CEWES). This task was accomplished by working with both the MCI's vBNS engineering staff and IDREN routing authorities. Initial configuration of the link between the sites indicated route asymmetry, which was traced to a typo made in one of NCSA's vBNS routing configuration files. Due to the added complexity of multi-peered networks, these sort of minor issues are not uncommon. Once connectivity via the vBNS and iDREN was established between the two sites, performance testing began. Ttcp (a program for testing host-to- host network performance) and ping between NCSA and WES recorded poor performance in the range of 1-3 mbit/s, with 5-40% packet loss. The initial investigations by MCI uncovered a packet loss problem on the vBNS related to the ongoing upgrade of the vBNS backbone to OC-12, and was quickly resolved. Unfortunately this had no significant effect on the NCSA/WES performance. Further investigation using traceroute to document each individual hop followed by pinging each hop on the path between NCSA and WES revealed a concentration of packet loss at MAE- East where the vBNS and IDREN met and exchanged traffic. Little could be done to resolve the well documented shortcomings at MAE- East. Given this we convinced the operators of the vBNS and IDREN to alter routing configuration so that traffic was exchanged at the Sprint NAP in New Jersey. That process took nearly a week. but fortunately improved end-to-end performance greatly. ttcp tests indicated throughput near Ethernet speeds and ping revealed packet loss less than 5%. Testing the application at this point revealed that it was still seeing very poor network performance (packet loss around 50%). Investigation showed that while normal IP traffic between NCSA and WES was following the vBNS/iDREN path, the multicast traffic used by the application was traversing the Internet following the multicast backbone (Mbone). We rerouted the multicast traffic over the vBNS/iDREN by creating a multicast tunnel from the Ethernet subnet at NCSA and the FDDI subnet at WES. Multicast traffic between the two subnets was now following the same route as normal IP traffic. This improved application performance a little but was still far from the level seen by ttcp and ping. Further investigation showed the multicast tunnel was carrying much more traffic than could be explained by the application. This turned out to be the tunnel end-point at NCSA advertising it's connection to the Mbone to WES and despite our efforts at setting a high metric on the tunnel to discourage traffic unrelated to the application, it continued. We solved this problem by disconnecting the Ethernet subnet at NCSA from the Mbone forcing the unrelated multicast traffic from WES to an alternate route to the Mbone. With another problem solved we again tested the application. We saw that although the application was seeing low loss (~5%) from NCSA to WES, in the return direction loss rates were still high (~50%). We made special arrangements with WES to gain root access on a machine on their FDDI subnet so that we could sniff the application's network traffic on both their subnet and ours. Doing this showed us that the application was sending out very bursty traffic. Working with the application developer we experimented with putting short delays between packets. This reduced the loss in both directions to levels comparable to what we were seeing with ping (<5%). We believe the application was sending out bursts of packets, especially on the FDDI- attached Onyx at WES, that were overflowing the hosts that were acting as ends of the multicast tunnel, causing them to drop packets. At this point the application was performing well except for a portion that was designed to utilize UDP and made the assumption that the network would be loss free. This portion of the application worked well when tested on the NCSA LAN, but was obviously now failing over the wide-area. Since it was impossible to reduce the packet loss to zero over the wide area, the application developer was forced to quickly modify the application to use TCP for the this portion until loss tolerance could be designed into the application at a later time. Lessons Learned * Having to obtain root access on someone else's network in order to analyze a applications network traffic is undesirable for security reasons and often will not be possible. Tools that enabled with this sort of analysis without giving away all the privileges of root are needed. * Multicast traffic adds a whole level of complexity to normal network traffic since it has it's own routing through the network. It needs it's own set of performance and diagnostic tools. * Application design needs early input from network engineers to ensure that the different network environments the application will be running in are taken into account. Network engineers should also be prepared to delve into the application to debug performance problem. * Having a common set of network performance tools on all hosts involves saves time and effort of installation, which often requires root access. * Having daily performance statistics for the vBNS was very useful since we knew what kind of performance to expect from it. Since it is a known quantity it allowed us to ignore it and narrow the list of potential bottlenecks. * A good collaboration environment for interaction between the network engineers and the application developers is a big plus. Regular synchronous and asynchronous communication is often needed among multiple people. Email can be too slow and setting up telephone conference calls too cumbersome. Suggestions include MUDs (http://www.clock.org/muds/) or multicast conferencing tools.