Orchestration of Containers
This topic is based on Research Spotlight Technology Review – Orchestration of Containers (Microservices), CMS TRB Technical Topics, 23 January 2018.
Containers and Microservices are sometimes used interchangeably, but they are two different concepts. Microservices is an architectural style where the application is implemented as a suite of small services, each running in its own process. These processes communicate using lightweight mechanisms (usually HTTP). The goal is to make these lightweight services independently deployable in a fully automated fashion so they may scale as needed. Containers are the vehicle for implementing these microservices and provide a means that enables development groups to more rapidly create iterations to address business problems by piecing together a set of smaller services. Containers are lighter weight compared to VMs. Containers make more efficient use of the underlying architecture and can scale applications to meet fluctuating demand. Containers also facilitate the ability to move applications between different environments or even clouds.
There are two options for implementing containers. One, the bare metal implementation, is when the containers run directly on the host. The other implementation, and probably the most likely for CMS, will be when the containers are implemented within a VM. This is because CMS is making use of more cloud-based infrastructure, where bare metal options are not available. The implemented containers must adhere to CMS ARS security and CMS TRA requirements.
Orchestration
Individual containers can be crafted and managed by a runtime APIs. These runtime APIs may meet the needs for managing one container on one server, but the strength of the container and microservice architecture is the ability to automatically scale the systems. Container orchestration (OR) tools are best suited for managing multiple containers on multiple hosts. These tools can treat an entire cluster of machines as a single entity for deployment and management. They can automate and manage the initial placement, the scheduling and deployment of updates, and perform health monitoring functions while supporting scaling and failover.
Business Rules and Recommended Practices
Business Rules
BR-OR-1: Container Images Must Be Hardened
The container images must be hardened according to security specifications, both at rest and during operation. Ensure conventional security tools can peer into running containers to assess hardening and patching compliance.
BR-OR-2: Use Security Monitoring on Containers
The containers must be subjected to security monitoring, including the review of logs. The logs must be given to CMS security or security personnel must be granted access to review the logs.
BR-OR-3: The Deployment Infrastructure for Containers Must Be Hardened and Monitored
The deployment infrastructure and containerization hosts for containers must be security hardened and monitored.
BR-OR-4: Containers Use Must Respect the Multi-Zone Architecture
Container use must maintain the multi-zone architecture—separation of presentation, application, and data services must be maintained (i.e., vertical separation). For example, a single container within the Presentation or Application Zone cannot be used to contain all elements of an application stack (presentation, application, and data storage services).
BR-OR-5: Libraries of Containers Must Be Maintained in CMS-Only Stores
Libraries of containers, as with any software, must be maintained in CMS-only stores, not shared in Internet libraries (CMS private repository, e.g., private Docker Hub).
BR-OR-6: Required Orchestration Capabilities
Container Orchestration tools must provide the following capabilities to support development and operational (DevOps) needs:
-
Configuration. Container configurations must be described in text files.
-
Service Discovery. OR tools must provide a mechanism to enable the runtime discovery of containers or microservices.
-
Policy-based Placement. To support security, performance, and high-availability requirements, container OR tools must provide a mechanism for defining policies concerning the placement and scaling of containers in the environment, including restrictions such as non-colocation. The policies must also ensure that the container placement and security rules meet CMS ARS security and CMS TRA guidelines.
-
Monitoring. OR tools must be able to track and monitor the health of the system’s containers and hosts. OR tools must run specified health checks at the appropriate frequency, update the list of available nodes, and ensure that the current state of the cluster matches the configuration specified. OR tools must provide or support real-time monitoring to help find performance, compliance, and vulnerabilities issues.
-
Resilience. If a container crashes, the tools must be able to launch its replacement. If a host fails, the OR tools must be able to restart the container(s) to a new host.
Rationale:
These required capabilities establish the minimum baseline for application container orchestration. The configuration capability enables DevOps teams to easily manage configurations via configuration management and version control tools.
The service discovery capability allows containers and microservices to discover one another within the orchestrated environment. This also enables runtime reconfiguration and provides for better resiliency and adaptability.
The policy-based placement capability enables the OR tools to enforce the defense-in-depth strategy of the CMS TRA. A system implemented using containers must still adhere to the security and architecture measures that protect CMS assets from attacks. Even though a service is not implemented within a container, it may not be moved to the edge of the network. CMS expects that all the required defense-in-depth mechanisms still front access to Data Zone services. For example, non-colocation is used to prevent all containers for a service to be placed on the same VM or physical host. Policy-based placement also enables adaptable deployments across development, testing, and production environments.
The monitoring capability acknowledges that containers must be monitored to ensure optimum performance. The resilience capability introduces automation to allow for automatic recovery in case of container or host failure.
Recommended Practices
RP-OR-7: Prefer Container Orchestration Tools That Allow for Container Motion
Container OR tools should be able to move containers across environments and clouds. The placement of the containers must still follow CMS ARS and CMS TRA guidelines. Container OR tools that can operate and move containers across multiple infrastructure providers are preferred.
OR tools should allow for the rolling upgrade of containers across the cluster. The tools will ensure that traffic is routed appropriately, ensuring that a minimal number of containers are available as the other containers are progressively replaced, following the immutable infrastructure model of RP-CA-11.