How Oteemo Led a DevSecOps Transformation
Oteemo’s client is a national leader in diagnostic information services, serving one in three of adult Americans and half of the physicians and hospitals in the United States.
The Client has a wide array of products and services that dozens of teams work on. While development is ongoing, the Client was in search of a solution to upgrade legacy code, increase efficiency, improve security, and lower total costs.
The Client initially reached out to Oteemo to help alleviate these expanding issues in the fall of 2019, but it wasn’t until the COVID-19 pandemic revealed the importance of a reimagined development model.
Internal Development Opportunities
The Client’s internal development team relied on a DevSec operating model across dozens of its software products. In March of 2020, as the COVID-19 pandemic took hold, these teams were exceeding their capacities and capabilities. Internal teams were working at 200% capacity, waivers were required for nearly all developments to move at the required speed, and costs were piling up. The Client needed a better solution and wanted to move to the cloud.
DevSecOps Assessment
Oteemo approached the assessment with the goals of assessing their current DevSecOps operating model and providing recommendations to improve their software life cycle.
Oteemo created a thorough assessment of the Client’s existing development model and outlined the gaps between existing models and the cloud based DevSecOps model the Client would like to move to. This assessment included interviews with over 50 employees, horizontally across teams, as well as vertically from developers and QA engineers to director level executives.
Portfolio Assessment Beyond DevSecOps
In addition to the DevSecOps assessment, Oteemo executed a full portfolio assessment. Oteemo reviewed 25 applications across the Client’s suite of products and reviewed a number of factors applying scores to a variety of domains. Oteemo assessed the capabilities to discover where assistance was needed, and find the teams that needed the most help.
Sample Roadmap to DevSecOps Operating Model
Assessment Findings
AGILE, CULTURE, & COLLABORATION
FINDING : Oteemo found an opportunity to develop a deeper understanding of Agile and the Agile philosophy. The Client was currently using an Agile model that was not working and could not scale. Resources were operating very lean and there was a lack of standardization around processes and tools.
RECOMMENDATION : Define agile ownership and create a controlled vocabulary to establish a standardization of tools. The Client should develop Agile ceremonies between Business Analysts and Scrum Masters to tie together business and IT and promote collaboration across teams by defining a common goal and shifting to a product funding model. Additionally, Oteemo recommended The Client provide capacity for teams to reduce technical debt and create safe space and grace periods to enforce higher quality.
RELEASE ENGINEERING, QA, & AUTOMATION
RECOMMENDATION : Oteemo recommended defining and enforcing standards to improve consistency and adopting a product centric approach to share a common goal, productizing DevOps team capabilities to support the organization and leveraging automation to gain efficiencies.
INFRASTRUCTURE AUTOMATION & CONTINUOUS MONITORING
FINDING : Oteemo discovered a knowledge gap in infrastructure automation which increases risks and costs while impeding time-to-market. There was a lack of understanding of the requirements to run healthily in cloud environments, different teams use different technologies for monitoring codes, and overall teams felt there were too many outages and the outages lasted too long. This was due to the lack of comprehensive monitoring.
RECOMMENDATION : Oteemo recommended augmenting the infrastructure teams so they automate in the cloud (AWS/Azure). This would allow them to support the growing demands for provisioning infrastructure, provide demos, training and documentation, and more. Additionally, the standardization of the DevSecOps toolset around automation will help gain insight and gather metrics for current projects. Streamlining knowledge sharing will reduce the pressure on the Dev and Ops teams during on call hours.
SECURITY
FINDING : The existing infrastructure enabled security to be integrated into their development effort. The security team wasn’t properly integrated into the production cycle. A feedback loop was missing between the security teams and the dev teams, and each team had a very different remediation plan.
RECOMMENDATION: Security needed to become more ingrained in the development process. Oteemo recommended that teams should partner with the security team to understand findings, change the engagement to have short but frequent touchpoints, automate security checklists, review and automate portions of the security auditioning, and leverage available security tools to reach regulatory compliance.
Oteemo, Boots on the Ground
After the completion of the DevSecOps assessment, Oteemo launched a pilot program with two development teams. Oteemo implemented the recommendations on a small scale to test adoption with different teams, and encourage other teams to prepare for change and adoption by showing a successful use case. To begin, Oteemo proposed criteria and actively participated in the selection process of two pilot applications to transition to the new development model. The pilot applications and teams were selected to create an upscale message and encourage team members to continue the adaptation process in the rest of the organization. Oteemo took one of the pilot applications and broke the single large team of 30+ people into three smaller teams. Oteemo assisted in changing the organization code to a branching strategy and adapted the way they structure their features and code changes. Changes were tooled around build environments using jenkins and gitlab saas offerings. This exposed a number of security changes, so Oteemo updated the interaction model with security. Oteemo instituted gates that prevented teams from introducing new vulnerabilities. Additionally, Oteemo created regular touchpoints with the security and product team to create a unified goal of security throughout the development process.
In the end, Oteemo created a new Continuous Integration environment, including secrets detection, code quality analysis, dependency vulnerability scans, static application security testing (SAST), Dynamic Application Security Testing (DAST) and container security scanning. Oteemo also integrated coverage metrics for the tests performed.
The Oteemo team transitioned two teams to the reimagined model, and set a pathway forward for transitioning additional client teams to the new model.
Continued Engagement
Oteemo completed the transition of two applications and is actively working on four more, two of which are in an advisory role. By the end of 2022, a total of 7 teams will be transitioned to the new model, with an estimated complete portfolio transition by 2024. Oteemo continues to provide support by directly onboarding new development teams into the new model along with providing oversight to teams executing their own onboarding.
Success Story
Oteemo helped the client modernize their software development practices and achieve tangible benefits. The changes made by the development teams started to be smaller and decoupled from each other, leveraging feature flags. The code review process was restructured and streamlined to save time, and gates were introduced to prevent deploying artifacts that decreased code quality or introduced new important security vulnerabilities.
The infrastructure was bootstrapped with reusable modules across applications, providing the necessary building blocks to quickly provision resources in a consistent manner.
Oteemo worked closely with the security team and the application development team to change the dynamic between the two groups. Security practices shifted left with the introduction of multiple security scans directly within the pipeline. This shortened feedback loops from months to minutes, drastically reducing costs to fix.
0 Comments