How Urban Science is Implementing Continuous Delivery to Save Thousands of Man Hours a Year — and Change the Auto Industry in the Process
Detroit, Michigan has been the center of the American auto industry since the turn of the 20th century, when Henry Ford churned out a few cars each day at his Mack Avenue factory. Nowadays, in addition to Ford, Buick, Cadillac and the rest of the GM family call Motor City home. Behind all the horsepower and sheet metal of the production line, car manufacturers also employ numerous software applications and tools from various vendors as part of their business. One such organization in the automotive industry that you may not have heard of is Detroit-based Urban Science – a company that is changing the way the car industry does business.
Urban Science is a global automotive retail performance expert that serves nearly every automotive OEM in over 70 countries. From Acura to Volvo and just about every manufacturer in between, Urban Science is finding new and innovative ways for auto companies to increase market share and improve profitability. Their goal is to identify and solve the toughest business challenges of this massive industry. They work with manufacturers to help them understand how people are buying, servicing their cars, and using their cars. Basically, if there is any kind of statistic around the automotive industry, Urban Science is tracking and interpreting it. Their work helps manufacturer gain insights across the entire product lifecycle. For example, not only can they tell you which Ford dealer is selling the most yellow Mustangs with ragtops, but they can give Ford insights into how likely the client is to bring that car back to that same dealer for repairs, and if they do so, what that means the client will do the next time they are ready to purchase a car.
You can imagine the infrastructure and software updates required to support such an operation — with its demanding customers, high stakes business, and massive amount of data being collected from all over the world and analyzed 24/7.
Mark Priolo, Software Configuration Manager at Urban Science, is tasked with this challenge. Straddling the line between Configuration Management and Release Management, Priolo is responsible for building and managing the software release processes and IT environments to deliver Urban Science’s offering to its customers. And he’s doing it by implementing DevOps automation and Continuous Delivery (CD) as part their software delivery practices.
The Drivers
Across its portfolio and various services, Urban Science develops and releases dozens of applications, developed and managed by hundreds of developers and IT release managers.
As Urban Science’s applications continued to get more complex, their corresponding deployments and release processes became more complex as well. Limited test coverage and lack of standardized processes and environments were delaying critical software rollouts to their clients, leading to missed releases. In addition, infrastructure utilization was inefficient, with growing infrastructure costs and increased management overhead. For the company to remain competitive and effectively serve its customers – something had to be done.
While Urban Sciences’ IT operations teams were incredibly strong, the complex nature of these deployments continued to increase the likelihood of errors. It was identified that what was being asked of ops team simply wasn’t sustainable. For example, there were teams that were rolling out applications which required having 15-page documents with go-to logic in them (I.E. “Go to Page 7 and do this…then go to page 3 and do this”). Not only were the deployments complex and time-consuming to develop, the deployment process itself could take hours.
So, Priolo set out to implement automation and CD practices to accelerate this pipeline – aiming to reduce manual tasks, one-off configurations and deployment errors, while increasing test coverage, feedback loops and release cadence.
Doing so, however, was not without its growing pains.
The Challenges on the Path to Continuous Delivery
The first thing Priolo was challenged with was figuring out what to automate. Sure, the old way of doing things is painful, but at the same time, it was working. However, the fact was there was a ton of man power going into doing things manually. Even when the decision was made that, “Yes, we should automate,” the question remained: should they try to automate what’s working or just concentrate on new stuff and make sure new builds are automated from the beginning. The initial decision was to focus on automating the delivery processes for only new applications and workflows, and – for the time being – keep the old processes as-is.
Another challenge of automation was team-adoption. Although the executive management recognized the need to adopt DevOps practices and implement CD, these practices were not required for teams to use. Rather, teams were given a choice if they wanted to automate, or continue doing things the old, manual, way.
After a while, the organization took a step back to analyze and re-evaluate their rollout strategy. It was clear things could be improved – and part of this realization came directly from looking at tickets assigned to old applications’ release processes that were over 2 years old, and were simply not progressing, while newer, CD-enabled developments were gliding through the pipeline. Now, people were asking “ “Can we automate the old stuff, too?”
Streamlining Your Path to CD
➤You are more alike than you realize:
The fact was, that while every application’s code was unique, and some deployment processes required very unique configurations or tasks — 90% of the delivery process throughout the lifecycle was the same across all applications. Priolo and his team set out to find those similarities in the commit-to-release pipeline across the different applications. Once they could map the process, they came up with a robust pipeline model that would work with that 90%. They then proceeded to rolling out this pipeline, and all its inherent automation tasks, to about half a dozen teams who started using it.
The results were extremely encouraging: with a relatively small number of teams and applications included, the project had already saved over 1,800 man hours annually.
➤Look at the big picture, focus on the biggest pain for the biggest impact
Priolo approached automation by looking at it from the perspective of identifying the opportunities for big wins and focusing on those. The goal wasn’t to fix local processes that were broken along the way, but rather to discern pain points and give the most bang for the buck.
Being a small team, Priolo didn’t want to have automation creep – ad-hoc fixing of different processes for various teams to address specific scenarios. The focus had to have been on reusability, system-level view and the ability to scale. These would allow his efforts to have the most impact for the entire organization, rather than achieving onlylocal optimizations, which may not amount to much in the large scheme of things (- or – may just be moving the bottlenecks to other areas in the pipelines.)
Therefore, Urban Science focused on three different pipeline models that would be applicable for the vast majority of use cases. By doubling down on those three, Urban Science can service the majority of teams and application, giving them just about everything they need.
➤Bring others on board!
While the initial efforts to implement automation started off with just that half dozen teams buying-in, Urban Science now has over 70 teams taking advantage of that model. Depending on the specific application and use case, some teams have chosen to release monthly, weekly, and even multiple times a day.
These three pipeline models and the standardization across teams allow the entire organization to become more agile, and increase product quality and release velocity and throughput. This sort of release cadence and speed was just not possible with trying to schedule times for manually manage all the processes and environments configurations involved in the delivery pipeline.
➤Think ‘Process as Code’
After modeling the key release pipelines and standardizing their processes, Urban Science turned their focus on treating their Process as Code. Using a technology that enabled them to essentially export their pipeline models and all related automation as code, they were able to have their DevOps processes themselves be programmable. This meant that they could be versioned, reuse and stored in their source code repository- along with their application code and environment configuration.
With the process itself being codified, Urban Science could more easily on-board new applications and teams, update their code to support additional use cases, and more easily extend certain pipelines for specific needs- – achieving predictable, repeatable, releases processes across the organization.
➤Do it end-to-end
To accomplish all this, Urban Science leveraged Electric Flow – an end-to-end DevOps Release Automation platform from Electric Cloud, to simplify build, test, deployment and release of multi-tiered applications.
A successful DevOps transformation takes the right people and culture, the right processes, and also the right tooling. From a tooling perspective, it was crucial for Urban Science to have a unified, single platform that can support their entire end-to-end delivery process, across all teams, and be able to orchestrate all the point tools, workflows and environments that are part of the process.
This ensured the ability to standardize their pipelines across teams and applications, and scale to support the entire organization (rather than cobble-together a different chain of tools or snowflake configurations for various tasks). In addition, this provided both Development and IT shared control and visibility into their software pipeline, across development, testing and packaging stages.
To ensure compliance and organizational control, the pipeline also supports both automated and manual approval gates as code is promoted to further stages along the pipelines and into higher environments. (For example, code can be promoted automatically upon passing a battery of test, or await a confirmation from a supervisor via the platform, before proceeding to the next stage.)
Continuous Improvement and the bottom line:
By adapting new ways of thinking and new technology solutions, Urban Science is now able to drive their software production within a Release Pipeline model. In this approach, all the tasks and workflows throughout the pipeline – from code commit to Release into Production – are being completely automated, and standardized as much as possible across teams.
This has had a great impact on the business.
DevOps and CD are essentially a journey – along a path to continuously optimize your software delivery, to improve IT and organizational performance. Automating the end-to-end Release pipeline is what “greases the wheels” for Urban Science software innovation. Now, the teams are freed to focus on developing great features, rather than on building ad-hoc workflows or doing manual work to manage the path that the code has to go through until it is ready to be released to end users.
DevOps release automation and Continuous Delivery practices have enabled Urban Science to accelerate their software delivery – saving time, lowering costs and increasing productivity. They are now able to better serve their automotive clients – building new and innovative solutions that are changing the way auto manufacturers produce, sell and services their cars.
Urban Science achieved:
- Faster and more frequent deployments. Urban Science went from 12 deployments per week to 40 deployments a week.
- Fewer person resources required. Manual processes replaced by automation reduced person resources needed for deployment by 78%.
- Fewer process errors. Tedious, error-prone manual operations have been eliminated, improving predictability and reliability, and reducing the number of deployments needed.
- Better software quality. Automated deployments mean more time is spent testing instead of on the deploy process.
