The FBI Stumbles Developing a Virtual Case File System
In March 2005, the FBI shelved a $170 million software development project designed to improve the efficiency and effectiveness of its investigations.The software, called Virtual Case File (VCF), was to serve three functions: manage investigative records, share and analyze information, and provide electronic approval for the flow of paperwork. While the FBI already had taken action to replace its cumbersome and obsolete Automated Case Support (ACS) system with VCF by 2000, the September 11 terrorist attacks exposed the FBI’s inability to “connect the dots” and focused public attention on the urgent need to overhaul its antiquated information sharing technology.
Four years in the making, VCF was one-third of a larger IT modernization initiative known as Trilogy. Its other two components consisted of a hardware and software upgrade and the construction of secure LAN and WAN networks. In May 2001, the FBI outsourced these two components of Trilogy to DynCorp, which finally completed the project in April 2004, 22 months late and $138 million over budget. As a result, the FBI did not have the technological infrastructure in place to deploy VCF until 2004.
The Department of Justice had deemed the project too large to be handled by one contractor, so in June 2001, the FBI outsourced the development of VCF to Science Applications International Corporation (SAIC), a San Diego-based science and technology firm that primarily services U.S.government agencies.In December 2003, after scrambling to meet its deadline for the project, SAIC delivered 700,000 lines of unworkable code. The FBI rejected it upon delivery.
Over the next few months, the FBI assembled a list of nearly 400 problems. When SAIC announced that it would fix the problems for an additional cost of $56 million, the FBI refused the offer. Instead, in June 2004, the agency began testing a scaled-down version of the product in its New Orleans field office. The testing checked the VCF’s ability to provide electronic approval for documents uploaded into the old ACS system. The FBI also hired the Aerospace Corporation for an additional $2 million to review the project. By early 2005, the agency realized that nothing was salvageable and accepted Aerospace’s recommendations to scrap the whole project.
“There is a long history of failures in large software projects, especially when you’re converting an existing system,” said Steve Bellovin, Professor of Computer Science at Columbia University. “This is almost a textbook example of how to not do it.”22
In answering the question “Where do we go from here?” in his testimony before Congress, FBI Director Robert Mueller III stated that “we will take with us a number of valuable ‘lessons learned.’”23 The Aerospace report, as well as several earlier audits and studies, pointed out exactly what these lessons should be and where the FBI and SAIC went wrong.
Ever-Expanding Scope, Time Crunch, and Spiraling Costs
In June 2001, when the FBI awarded the contract to SAIC, the purpose of the project was to update and consolidate the ACS and four other important FBI investigative applications. These updates were intended to allow agents to access these five integrated applications through the Internet. Following September 11, Mueller recognized that the original scope of the project would not allow the FBI to fulfill new objectives under its counterterrorism mandate.The FBI, an organization that collected information after a crime, now had to catch terrorists before the crime. In December 2001, Mueller asked SAIC to abandon the development of the Web interface and begin work on a new case management system from scratch. The FBI decided not to incorporate off-theshelf products, which not only lengthened production time but magnified the number of bugs in the final product
In addition, following September 11, the FBI was under the gun to produce the new case management system as quickly as possible. The FBI abandoned the project’s three-year time frame and moved up final deadlines. The deployment of VCF was divided into two phases, with deadlines set for December 2003 and June 2004. With six months of development time already lost to the Web interface, SAIC had only 22 months to develop a much more extensive product.
The target dates of the Trilogy project’s other two components were pushed up as well. The overall costs of the project spiraled from $379.8 million to $581.1 million.
No Blueprint, Changing Specifications, and Micromanagement
In March 2002, Sherry Higgins, FBI’s new director of the newly created Office of Program Management, asked Special Agent Larry Depew to become the VCF project manager. Depew was a technophile who had programmed his own case management database to handle an investigation in the early 1990s.Yet, he had no experience in IT project management. He had been one of a team of seven agents to evaluate SAIC’s original Web interface, and had recommended sacking it.
Depew organized a team of FBI agents to work with SAIC engineers and specify requirements for the VCF. For six months, Depew’s team of FBI subject experts met with SAIC engineers to define and redefine user needs through a software development process called Joint Application Development (JAD). But the JAD approach went horribly awry as FBI agents overstepped their boundaries, dictating details that should have been decided by experienced engineers. For example, agents went as far as proposing a design for a portion of the interface. Depew, who led the sessions, would decide what was inside or outside the scope of the project. The result was an unwieldy, 800-page system requirement document.
Unfortunately, the FBI’s attempts to define and redefine system requirements did not stop there. As SAIC delivered parts of the product, FBI’s team of agents demanded more design changes in a “we-will-know-it-when-we-see-it” approach. SAIC would then retroactively make these changes to related parts of the software. Inevitably, this process led to inconsistent implementation of the altered system requirements.
“This cycle was repeated over and over again and prevented SAIC from defining system acceptance criteria and suitable test standards,” said SAIC Executive Vice President and General Manager Arnold Punaro.24
Due to the time pressure created by the September 11 attacks, SAIC agreed to go forward without an enterprise architecture—a comprehensive blueprint that describes how the current and future structure of an organization, its IT systems, and its processes align with strategic goals. The decision turned the project into a high-risk venture.
“Here is where SAIC made honest mistakes,” admitted Punaro. “We should have made known that this approach was too ambitious.”25
The defining and changing of systems requirements was not the only area in which SAIC failed to insist that the FBI follow sound software engineering practices. SAIC let the FBI’s inexperienced IT team make several risky decisions, the worst of which was the intended “flash cutover.” Rather than implement a phased migration in which parts of the new system could be tested and repaired until the whole system was deemed fully functional, the FBI planned to switch from their old ACS system to VCF overnight, with no overlapping period.
University of Pennsylvania Computer Science Professor Matt Blaze was on the National Research Council (NRC) committee that reviewed the VCF project in 2004. He said the NRC committee was horrified when it learned of the planned flash cutover. “I remember thinking,” said Blaze, “that that would be a very good day for a crime spree.”26
Former SAIC vice president David Kay said, “SAIC was at fault because of the usual contractor reluctance to tell the customer, ‘You’re screwed up. You don’t know what you’re doing.’”27
Yet, SAIC’s mistakes may have gone beyond this problem. In response to the tighter deadlines and the demanded changes, SAIC brought more employees into the project, including new hires, and adopted a risky parallel development approach. SAIC personnel working on VCF increased to 250, a possible violation of what is known as Brook’s Law. This popular axiom states that “Adding manpower to a late software project makes it later.” The reasoning behind the law is that the time spent communicating to new, inexperienced team members outweighs the time it would have taken to complete the tasks at hand. In addition, to decrease development time, SAIC divided its staff into eight development teams working in parallel on different parts of the software. These parts would be integrated in the final phase of development. As later project reviews noted, this approach resulted in a failure to apply coding standards uniformly across the product.
When the original contract was signed, the Trilogy project was a relatively minor endeavor for the SAIC. Established in 1969, SAIC today is one of the largest science and technology government contractors. For the FBI alone, SAIC had developed CODIS, a national DNA database used by law enforcement agencies around the world, and NICS, the national criminal background check system. The IT revolution of the 1990s and the September 11 attacks created a boom in the industry. SAIC’s business, income, and staff expanded rapidly just before and after SAIC was awarded the contract, as shown in the following table.
FBI’s CIO Change-Over and Decentralized IT Structure
In reviewing the FBI’s failure with VCF, FBI Director Robert Mueller admitted, “We lacked skill sets in our personnel such as qualified software engineering, program management, and contract management. We also experienced a high turnover in Trilogy program managers and chief information officers.”28 During the three years of Trilogy’s development, the FBI had five different CIOs, as shown in the following table. The high turnover rate created an additional obstacle to defining system requirements and setting goals.
In addition, the FBI’s IT management was so decentralized that when Zalmai Azmi arrived, the FBI did not have a centralized budget for IT management. As CIO, Azmi managed a whopping budget of $5,800. Azmi instituted measures to centralize IT decision making while finally getting the organization to produce an enterprise architecture. Azmi has also focused on boosting the skill level of IT personnel.
In March 2006, after wasting over $100 million and five years, the FBI awarded Lockheed Martin a $305 million contract to create a new case management system. The project is under intense scrutiny as Congress, IT professionals, and the media wonder whether Azmi’s changes will be enough to ensure success. Some lessons have been learned. The project is to be deployed in four phases over six years, and the system will rely in part on commercial, off-the-shelf software. Yet the question remains: what other lesson does the FBI need to learn before it can obtain the IT system needed to “connect the dots” and possibly prevent another September 11?