Common Engineering Support Services

Defect Tracking Services

A defect tracking system is a web-accessible, multi-user repository that is the source of record for software defects identified by any of the project stakeholders. Defects are entered, classified, worked, fixed, retested, and closed following the software developer’s defect correction process.

Version Control Services

The Centers for Medicaid & Medicaid Services requires designation of one version control repository as the official repository (source of record) used by the application system for software build and test. The contents of the Version Control System (VCS) repository represent the complete change history of all source code in the system.

At a minimum, the VCS must store and retrieve textual artifacts and establish named baselines for artifacts (also known as configuration items).

Note: CMS ARS Security Control SA-10 (“Developer Configuration Management”) requires configuration management practices.

Continuous Integration Services

The Continuous Integration services orchestrate the flow of source artifacts into and the flow of packages out of the build automation services. Builds are triggered on VCS check-in or on a schedule. Unit tests and integration tests may also be performed if the builds are successful. Results are posted on a Continuous Integration dashboard.

Build Automation Services

The build automation services generate a set of output packages (including documentation and other output products) based on source code, configuration files library packages, and other input sources. The output packages should be stored in the package repository.

Package Repository Services

A package is an installable component that contains instructions for how to install and uninstall the component. It should also contain a list of its dependent packages.

A release is effectively a versioned set of packages. A package repository is a service that contains trusted packages. These packages may have been acquired by contract, obtained via Open Source Software, or produced on behalf of CMS. A package repository contains both source packages and binary packages. Source packages can be used to build new binary packages. Binary packages are suitable for the system operator’s deployment into a production environment.

A package specification is a document or report that describes the contents of the package (sometimes called the manifest) as well as instructions for installation and removal. Typically, packages include mechanisms for dependency identification. Thus, if package A depends on packages B and C, then this is recorded in package A’s specification.

Often, a package builder uses package specifications to construct a package from source artifacts. A package installer takes one or more packages and installs them properly within the target system environment.

Inspection Repository Services

CMS recommends reviews and inspections as very effective practices for finding and preventing defects. See Software Inspection, Tom Gilb and Dorothy Graham, Addison-Wesley, 1993. and Software Inspection v Software Testing: How do they differ?

The inspection repository contains the artifacts of the inspection and the outcomes. Each inspection outcome of a code and design review should be coded with an indication of the number of issues or risks identified in the review to support useful metrics on the results.

Test Automation Services

Developer test automation services are essential for modern software development. Typically, by integrating test automation into the build automation services, tests run automatically after each successful build. Developer testing should include both white and black box testing. Unit testing alone is not sufficient.

Package Deployment Services

The package deployment services are charged with deploying trusted packages from the package repository and deploying them into the appropriate CMS environments (development, test, integration, or production), given appropriate authority. These services record these events in an audit trail. CMS does not specify the use of push- or pull-based mechanisms for package deployment.

Accessibility and Section 508 Test Services

Testing software systems for compliance with Section 508 accessibility requirements is mandatory. While automated testing is helpful, achieving all the requirements does require expert interpretation. An accessibility test service with software tools and experts knowledgeable in the field can help greatly. It is highly recommended that accessibility be built into the design of applications because this will increase the likelihood of successful Section 508 testing.

Performance Test Automation Services

These services can be used to performance test, stress, and soak test applications to determine their behavior under high and sustained load. Soak testing is useful for finding memory or resource leaks that can bring down a system even when performance is within expected parameters.

Security Assurance Services

While static analyzers can be very helpful at finding security vulnerabilities in code, it is important to supplement this analysis with dynamic analysis using tool suites designed to find vulnerabilities in the running system. Such a test bed can subject a system to systematic probing and attacks while still under development to better address security issues early in the software development process. Similarly, using data loss prevention techniques can detect causes of data leakage.

Note: CMS ARS Security Control CM-4(1) requires using a separate test environment before implementing software in an operational (production) environment to look “for security impacts due to flaws, weaknesses, incompatibility, or intentional malice.”