There are many ways that automation tooling can help streamline existing operations and assist with day to day system administration. Automation with Ansible can help free cycles dedicated to repetitive tasks. Operations personnel can then dedicate more time towards further automation. A virtuous cycle ensues of further automation and less low level Systems Administration work, allowing technical resources to focus more cycles on revenue generating improvements.
We are going to look at three areas where automation can help with Windows systems, or mixed Linux and Windows environments:
- Package Management
- Self Service Gateways
The tool we will cover is Ansible, which works well in managing many types of systems including Windows servers.
We will need a Linux system to act as a controller. Ansible does not run from Windows systems. A VM, physical server or cloud instance will do as long as the controller has network access to ssh or WinRM ports on the systems to be managed. There are some special considerations when configuring Windows target systems and the Ansible controller that will manage them:
- Target Systems must be at or above Powershell version 3
- The python pywinrm module will need to be installed on the Ansible controler pip install pywinrm
- You will need to run the ConfigureRemotingForAnsible.ps1 powershell script on target systems to prep the system for WinRM
- You will need to configure one of the available authentication methods
Windows target host configuration is more involved then Linux system setup so please refer to Windows Support page for detailed steps.
Ansible is very good at deployments, and patching is just a type of deployment. The best method of patching with Ansible is to leverage WSUS (Windows Server Update Services) and Active directory GPOs in conjunction with an Ansible controller. The Windows WSUS server pulls down updates to local storage on the WSUS server. Active Directory GPOs (Group Policy Objects) can be set so that clients will pull updates from WSUS server, and not from external MS sources. This approach can save a great deal of bandwidth and an additional advantage is that the WSUS Web interface allows for selective update approvals and has advanced reporting features. The updates will not be applied to systems via GPO, that will be managed by Ansible.
Once you have set the patches to download but not update via WSUS and GPO, you can then create simple or complex playbooks to update your systems. The following will apply all outstanding critical and security patches, rebooting systems as needed until all patches are applied:
--- - hosts: AllWin tasks: - name: Install all outstanding approved Critical\Security updates rebooting as needed win_updates: category_names: - SecurityUpdates - CriticalUpdates reboot: yes
You can use the rolling update features of Ansible to provide a more HA friendly patching methodology for production systems. This approach can still be largely automated as long as there are multiple systems providing application services while one or more can be down during updates. Also you can choose to blacklist one or more updates so that it is not applied as it normally would be. The playbook below would apply updates to servers in groups of 4, failing if +50% of systems in given run fail, also KBXXXXXX would not be applied to systems:
--- - hosts: importantweb serial: 4 max_fail_percentage: 49 tasks: - name: Install Crit/Sec updates in rolling fashion win_updates: category_names: - SecurityUpdates - CriticalUpdates reboot: yes blacklist: - KBXXXXXX
If required, you can quickly push out a hotfix to systems in event of 0 day attack using win_hotfix module (You can also use win_reg or win_shell to make a config update on a system as a workaround until proper patching can be applied):
--- - hosts: AllWin tasks: - name: Get hotfix win_url: url: http://myrepo.inside.sec/KBXXXXXX.msu dest: C:\Temp\KBXXXXXX.msu - name: Install hotfix win_hotfix: source: C:\temp\KBXXXXXX.msu state: present register: needreboot - win_reboot: when: needreboot.reboot_required
Ansible can be used in conjunction with chocolatey to manage packages on Windows systems the same way that packages can be managed using yum on RHEL based servers and apt on Debian based systems.
With the win_chocolatey module packages can be installed, upgraded and removed:
--- - hosts: AllWin tasks: - name: Install stuff win_chocolatey: name: git.install state: present - name: Upgrade stuff win_chocolatey: name: jdk8 state: latest - name: Remove stuff win_chocolatey: name: php state: absent
Of course, using publicly available packages might not be acceptable to the security organization in your company. In that case, you can build your own Chocolatey server and configure you client systems to only communicate and install packages from your local system. You can then create and internally publish your own corporate packages
There is even a ansible-galaxy role for setting up a chocolatey system making it east to test functionality:
Note you can also use Ansible win_chocolatey as another way to install patches assuming there is, or you have created, a package for the update
Self Service Gateways
I once worked for a company where account reps would periodically need to publish files to a couple of locations (FTP and S3) and then perform an application reload. The account team did not have access to all the systems needed to perform these periodic app reloads, so Ops personnel would have to coordinate and work with them to make these changes. Even worse, these reloads had to be performed at client’s discretion, often late at night.
A better way of addressing these kinds of repeatable tasks is to provide tooling so that users can execute their tasks themselves. For this we use Ansible Tower or, where cost is a concern, the upstream OSS package that Tower is based on: AWX. There a Ansible Galaxy role for installing AWX so it can be setup for testing with ease.
AWX/Ansible Tower will provide a Web based GUI and a REST API to Ansible that can be used to run Ansible code against systems as Jobs. AWX allows a secure method to allow groups of users to execute just those Jobs that relate to tasks they need to complete.
AWX can use LDAP or other shared auth providers to allow users access, and rights can be made very granular. Also, system access and access methods are abstracted from users which is security plus, but also allows easy onboarding and removal of new personnel (No need to change passwords when someone leaves).
Tier 1&2 operations personnel can be shifted to primarily use AWX thereby reducing those touching systems directly. In a mixed environment, Tier 1&2 Ops teams can levage Ansible to
- Perform user management
- Perform service and process restarts
- Address reboots in consistent manner
- Copy files, update configs
- Act as ad-hoc reporting solution, in a pinch
The use of Ansible to perform basic system administration tasks will help improve security posture, add consistency to procedures, and greatly reduce System administration overhead. All this is achieved at minimal cost in resources.