APPVALDOCS: Application Validation Documents

By Thomas Grzybowski
August 16, 2004


The "appvaldocs" project [1] plans to develop model FDA-compliant system validation documentation. Our motivation is primarily to help lower the costs of drug development through the creation of "Open Source" validation materials which are intended to assist in compliance with relevant FDA regulations.

FDA-compliant validation of software is a complex subject, with legal ramifications. We touch on a number of the impediments to the introduction and validation of Open Source Software in the Bio-Pharma industries.


It has been reported many times in many venues that dollar costs associated with pharmaceutical drug development are extremely high, on the order of tens of millions of dollars per New Drug Application (NDA). These expenditures by Biotech and Pharmaceutical companies (Bio-Pharma) are one element in the high cost of drugs to the consumer.

Implementation and maintenance of software systems used for drug research is a major expense - a small number of preferred vendors such as Oracle Corporation and SAS Institute deliver specialized applications to this market segment at high prices. Software maintainance of a typical clinical-trial database can easily cost hundreds of thousands of dollars per year. Similarly, maintenance costs for a statistical analysis technology stack can also run to hundreds of thousands of dollars per year. And initial purchase of software licenses are roughly twice as expensive as maintenance. The expense of validating a new system can easily equal the cost of the new software. Despite the high costs of software systems used in Bio-Pharma clinical research and development, it can be observed that there is little to no movement toward use of Open Source or Free software applications in this domain. The following cultural and industry-environmental factors impede the introduction of new technologies such as Open Source software [2]:

Associated with the large corporations in Bio-Pharma is a legion of service companies - consulting companies and contract research organizations. These smaller institutions serve to mitigate some of the inefficiencies described above. These service organizations are somewhat more likely to be receptive to Open Source software.

It must be noted that software and process innovation does take place in the Bio-Pharmaceutical Industry. One of the most important and positive developments in the Bio-Pharma industry in recent years is the development of open industry-standard data formats for clinical and laboratory data by the Clinical Data Interchange Standards Consortium (CDISC), and the recent acceptance of the CDISC Study Data Tabulation Model by the FDA as the standard format for sponsors of clinical trials [3].

The existence of Open Source software, together with open data standards immediately suggests the implementation of open systems in the Bio-Pharma industry. There is tremendous room for cost savings. However, in addition to the cultural factors outlined above, another serious impediment to implementation of new systems is FDA-required systems validation. It is a problem that the costs for validation of Open Source software are likely to be similar to the costs for validation of proprietary software, if not higher. Since a large fraction of the cost of validation is the preparation of adequate documentation, this is the area that the appvaldocs project aims to address. There are some additional Open Source validation issues that the project addresses less directly.

FDA regulations covering the implementation of computerized systems used in clinical trials are largely specified within the Code of Federal Regulations (CFR), and these are largely based upon Title 9, Chapter 21 of the United States Code (USC). The CFR regulations, together with the basic laws located in the United States Code are commonly termed "predicate rules". A number of "guidance documents" are also published by the FDA which elaborate upon the predicate rules.

The guidance document that is most relevant to this article is "Guidance for Industry. Computerized Systems used in Clinical Trials (April 1999)" [4], where the systems in question involve data-entry, storage or processing of clinical-trial related data. Here we find specific validation issues needed to be resolved in the implementation of Open Source Software. For instance, the above-referenced guidance document states:

"For software purchased off-the-shelf, most of the validation should have been done by the company that wrote the software. The sponsor or contract research organization should have documentation ... of this design level validation by the vendor."

It is guidelines such as these which make the adoption of Open Source software for use in the conduct or analysis of clinical trials problematic. Hopefully, more guidance from the FDA on the use of Open Source software will be forthcoming. However, currently at the present time it is common for "off the shelf" general-purpose proprietary applications (such as Excel) to be used which do not conform to the above recommendation. That circumstance is also covered in the same guidance:

"In the special case of database and spreadsheet software that is (1) purchased off-the-shelf, (2) designed for and widely used for general purposes, (3) unmodified, and (4) not being used for the direct entry of data, the sponsor or contract research organization may not not have documentation of design level validation. However, the sponsor or contract research organization should have itself performed functional testing...".

The appvaldocs project documents may provide some assistance for testing this kind of general-purpose Open Source software. Other sources to look for assistance would be the Open Source Development Laboratory [5] and certain of the software development projects themselves.


[1] Application Validation Documentation project page []

[2] Current State of the Biopharmaceutical Industry - Kenneth Getz, CenterWatch []


[4] "Guidance for Industry. Computerized Systems used in Clinical Trials (April 1999)" []

[5] OSDL-Lab Projects []