Ada Conformity Assessment Test Suite

The Ada Conformity Assessment Test Suite (ACATS) is the test suite used for Ada processor conformity testing. A prior test suite was known as the Ada Compiler Validation Capability (ACVC).

ACVC era

Example of Ada validation certificate awarded to vendor after passing the ACVC tests, 1986

The Ada Compiler Validation Capability test suite, commonly referred to as the ACVC tests,[1] was the original test suite developed for the Ada language. The ACVC system was organized under the aegis of the Ada Joint Program Office.[2]

The tests were developed by the American company SofTech, beginning around 1980.[1] The test suites were modeled on a VAX/VMS system,[3] which was the dominant host platform for such defense-related applications at the time. Some of the tests were composed using orthogonal Latin squares as an approach towards get the most effective coverage of language feature combinations without employing an exhaustive enumeration of them.[1]

The individual test files were based on the section of the Ada reference manual they pointed to, for instance C45210A.ADA.[1] The suite included both positive tests and negative tests.[4] There was an organization set up to review queries vendors raised as to whether a certain aspect of one or more tests was an accurate reflection of the language standard.[2]

The year 1985 saw the issuing of the first Ada validation certificates.[2] At the height of the language's use, which corresponded to the years 1985 through 1993, there were five Ada Validation Facilities around the world that could process vendor ACVC submissions: the Language Control Facility at Wright Patterson Air Force Base (United States), the National Bureau of Standards (United States, soon renamed to the National Institute of Standards), the National Computing Centre (United Kingdom), AFNOR (France), and IABG (Germany).[2] However once Ada use for defense or similar applications began falling, the number of validation contracts fell as well, and several of these facilities became inoperative or transferred their responsibilities.[4]

The Ada Compiler Validation Capability came to an end with the closure of the Ada Joint Program Office in 1998.[2] Ada compiler vendors still wanted a validation mechanism, however, so a new validation system was devised to replace it, the Ada Conformity Assessment Test Suite.[2]

ACATS era

The preface to the test report includes the following:

Conformity assessment does not ensure that a processor has no nonconformities to the Ada standard other than those, if any, documented in this report. The compiler vendor declares that the tested processor contains no deliberate deviation from the Ada standard; a copy of this Declaration of Conformity is presented immediately after the certificate.

The second paragraph of the background of the current ACAA procedures says:

It is important to note the scope and intent of conformity assessment. The purpose of conformity assessment is to ensure that Ada processors achieve a high degree of conformity with the Ada standard (Ada95 as corrected by [TC1]). Characteristics such as performance and suitability for a particular application are not specified by the standard, and thus are outside the scope of Ada conformity assessment. Moreover, the ACATS is a set of test programs intended to check broadly for correct implementation; it is not possible to exhaustively test for conformity. Thus, conformity is checked only to the extent of these tests; processors that are certified as conforming may fail to conform to the standard in ways peculiar to each, under particular circumstances.

The ACATS consists of 1821 tests with 255,838 lines of code occupying 30 MB of storage. The test suite is publicly available, for example as a part of the source distribution of the GNU Compiler Collection, which also contains the Ada compiler GNAT.

See also

Overview document for the final release of the original ACVC tests, 1989

References

  1. ^ a b c d Mandl, Robert (October 1985). "Orthogonal Latin squares: An application of experiment design to compiler testing". Communications of the ACM. 28 (10): 1054–1058. doi:10.1145/4372.4375.
  2. ^ a b c d e f Brashear, Phil (March 1999). "Ada validation := Ada conformity assessment". Ada Letters. XIX (1): 48–51. doi:10.1145/311457.311470.
  3. ^ "Ada Compiler Validation Capability (ACVC) Version 1.11 (Field-Test Release)". Defense Technical Information Center. June 1, 1989.
  4. ^ a b Tonndorf, Michael (1999). "Ada conformity assessments: A model for other programming languages?". Proceedings of the 1999 annual ACM SIGAda international conference on Ada (SIGAda '99). Association for Computing Machinery. pp. 89–99. doi:10.1145/319294.319310.