New Podcast
Managing Traffic During Peak Demand; Plus, Microsoft, Akamai Outages

Industry

How To Create a Culture of Trust in Automation

By Tris Clark
| | 16 min read

Summary

It's no longer a matter of if but when and how your organization will automate its systems. Establishing a culture of automation is crucial for enhancing network performance and freeing up NetOps' time to concentrate on strategic work that brings more value to the broader organization.


Automation is a hot theme in tech circles these days. And like any emerging trend, it can be contentious, particularly within risk-aware teams like the network organization. These teams are already grappling with enterprise networks that have grown enormously in the past few years as they scale and evolve to meet business needs that leverage the dynamic potential of SaaS, cloud, and SASE technologies. With so much complexity already inherent in the modern WAN, the notion of introducing automation into the IT systems equation can be a scary prospect for a NetOps team that prioritizes reliability.

But with IT and operations leaders now familiar with the efficiency, productivity, and agility gains that automation promises, it is no longer a question of whether your organization will automate its systems but when and how. How can I assure that my company's network and applications will continue to operate seamlessly when so much is being managed by complicated systems outside of my direct view? In this blog, we’ll take a look at how you can create a culture of automation that not only takes network performance to new heights, but also frees up NetOps’ time to focus on strategic work that’s more rewarding and of greater recognized value to the broader organization.

Digital Acceleration, Automation Now 

Why is automation necessary for NetOps teams today? The answer lies in a confluence of mutually reinforcing challenges that have pushed automation from “nice to have” to a necessity. Today’s most influential companies and organizations are scaling and adapting their business models, products, and services at ever faster speeds. Customers are driving this activity through heightened expectations, meaning that network operators must go beyond digital transformation toward a process of digital acceleration.

Tech talent shortages are another crucial factor strengthening the rationale for automation. Some of the industry’s most seasoned network engineers are retiring, whereas others are moving to cloud and security roles. CIOs and Network Directors often lament that they can’t attract network talent due to stressful (sometimes tedious) work focused on finding and fixing issues in increasingly complex environments. As a result, there is a palpable desire within senior IT circles to reinvigorate networking and make it more satisfying. Alas, the brutal reality for NetOps leaders is that they need to match their peers in cloud, security, and DevOps teams in terms of the business value they create and the dynamic working environments they can offer.

Automation is not a silver bullet for all of NetOps challenges. Still, it can remove many barriers to speed and scale—especially by removing some of the more fiddly, tactical tasks that distract senior network engineers from more important strategic projects, or demoralize junior engineers without the skills or experience to solve the problem at hand. Automation embeds intelligence within the DNA of the network organization, funneling human creativity and expertise to meet new networking challenges. With a culture of automation, NetOps becomes the “yes we will” team that thrives and scales to meet opportunity.

Know Your Digital Ecosystem, Master Your Fears  

Organizational cultures that succeed are smart, osmotic, and adaptive. Within this context, leaders can cultivate a culture of automation by developing a deeper understanding of the surrounding environment, which in turn enables teams to enact meaningful change while minimizing risks. As we noted earlier, NetOps teams have a very understandable fear of breaking the network. Automation moves faster than any human ever could, eliciting fears of an accidental misconfiguration that generates a cascade of issues. That is why a greater understanding of your IT environment’s nuances is key to safely deploying automation. 

Perhaps counterintuitively, even the makeup of a technology environment can significantly influence a team’s culture. Unpredictable platforms breed stress, which leads to tactical decision-making by IT and suboptimal digital experiences for customers and users. Ultimately, I believe data and context shape human beings and how our culture evolves, hence why digital uncertainty is a productivity killer.

To achieve a culture of automation—and make the most of future digital experiences—you first need to understand your organization’s historical network and application performance data. Or as the ancient Romans more elegantly put it, Respice Prospice, meaning “Learn from the past, and look to the future.” As a result, there are several "black box" environments that need to be visualized and understood before a team embarks on an automation journey:

  • Know the Internet - The Internet is a best-effort entity. It is ever-changing and beyond the scope of traditional monitoring solutions. However, with Internet and cloud visibility, it is possible to map and understand your service providers’ behavior and comprehend the infrastructure that will drive your organization’s delivery of digital experiences and services. The mix of cloud, ISPs, and CDN providers supporting your organization will be unique, and you need to know how they behave, as this will impact what you choose to automate and when. 

  • Map your SaaS stack: Similar to the ISPs, SaaS applications have behaviors that will shift over time. Each application is different, and user experiences will vary based on location. But since you don’t own the application’s infrastructure, you will need to shift your team’s psychology from finding and fixing the issue directly to one where they identify evidence and escalate it to the proper provider, who can make changes to routing decisions or adjust architecture. A common operating language is vital to expediting issues with these vendors externally—and across different teams internally. When teams trust that they can rely on a single point of truth it cultivates a successful automation culture across teams and within the wider digital ecosystem.

  • Demystify your "modern" network: Traditional techniques, like flow data capture, mean that while on-premise networks have always been observable, they are not black boxes. However, new approaches like SD-WAN and SASE have interwoven Internet and SaaS traffic into the mix, raising digital opacity and obscuring traditional monitoring tools' historical insight. The evolving complexity generated by this influx of new traffic requires network teams to actively map and understand its behaviors to safely proceed with further automation safely.

Evolving Culture Through Process, Incentives, and Direction 

But technology fluency is just one part of the automation journey. Two other essential areas are organizing for automation and maintaining an appetite for automation. Part of this revolves around defining roles, responsibilities, and process, while another part of it is related to human emotional rewards. People like to know where they are going, how they can get there, and the benefits. Again, a sense of certainty is the common thread. Let’s look at processes first. Below are some tactical approaches you can try:

  • Experiment with non-critical workflows: Your first selection of automation tests should be with non-production areas—ideally, these are tasks that are not overly complex. Failures at this stage are to be studied, discussed, and internalized across the team. Building institutional memory of what does and doesn’t work is your mantra.  

  • Use Internet and cloud monitoring to verify that your automated solution picks the most optimal routes and circuits. Test, test, test, and verify. Before you "flip the switch" to full automation, you must have an intermediary period where you test the environment and check that the results data aligns with the solution’s recommendation. Most solutions on the market today require substantial human intervention, but the direction of travel is towards closed-loop automation. Even when you’ve passed "the Rubicon" and have had a successful full automation launch, Internet behavior constantly changes, so be sure to periodically test and be ready to tweak network parameters and policy. 

  • Adopt an ecosystem mindset and get ready to share that data externally: As we previously noted, some issues your organization experiences with application or network performance may sit with a third-party provider in your digital supply chain. Your automated system may adapt to this issue over time. For example, it may proactively switch routes to avoid a problem it’s forecasting based on historical data. But before you even begin your automation project, you should work closely with your provider to proactively bulletproof the routes and policies they are using to support you. Refrain from thinking in terms of static service-level agreements and pointing the finger to lay blame. Instead, look at this as an ongoing service-level relationship where you and your third-party provider constantly learn from each other and create your own "closed loop" of digital knowledge. While in this collaborative process, optimizing your ecosystem with the use of recurring data benchmarks increases your automation efficiency and keeps your provider honest. 

Keeping Momentum, Inspiring Knowledge Sharing and Innovation

Like any IT initiative, automation projects will ultimately succeed or fail based on results and internal promotion of said results. When building a culture of automation, it’s essential to model best practices, internally celebrate breakthrough moments, achieve some quick wins, and market back successes to the broader business, which may help protect and grow your existing budget. Below are some general organizational principles to follow at this stage:

  • Think at scale and remember that even incremental improvement supports the organization’s bottom line over time: Problem avoidance through automation is one of the main benefits we can expect to become mainstream in the near future. And the compound benefits—time saved, digital experiences enhanced, and brand reputations protected—add up over time. Why bother troubleshooting and rectifying a network after the fact, when you can proactively see and avoid an issue in the first place? A culture that embraces automation visualizes, understands, and avoids issues instinctively. As my colleague Mike Hicks mentioned in his SD-WAN policy management blog post, predictive SD-WAN approaches can improve the end-user experience by “avoiding major downtime and hyper-tuning site-level performance.” Moreover, he notes how it can reduce help desk tickets by avoiding issues before they manifest and affect users, which means fewer people must lift a finger when a problem does occur. When you need to change policies across hundreds of sites, you can win in increments. For example, if a company has 600 employees, even a 1% improvement in network performance and, more importantly, employee and customer application experience can mean a substantial increase to the company’s bottom line. 

  • Model team behavior with integration champions and Hack Days: While broader organizational shifts can produce positive results over the long term, such as more secure and resilient networks by bringing NetOps and SecOps closer together, most automation successes take shape because of integration champions who visibly carried the flag in front of their colleagues. The best integration champions tend to be people who can straddle multiple disciplines—for example, security and networking—bridging both worlds. This hybrid mindset helps unfamiliar teams more easily visualize each other's concerns and priorities, increasing the chances of data sharing, collaborative partnerships, and platforms being used strategically at scale across silos. In addition to appointing integration champions, hosting automation-centric hack days can inspire innovative solutions that teams can demo for one another. Regular brown bag lunch-and-learns are another great way to disseminate the latest best practices. 

  • Share automation success with the wider business: NetOps teams have historically struggled to communicate their results back to their non-IT colleagues. This typically arises because the network team speaks in technical terms that do not obviously translate into the business metrics that the wider business prioritizes. To broaden your appeal and engage supporters, IT teams must ensure there’s a direct throughline from your automation gains, to your colleague’s bottom line or corporate targets. Always highlight productivity increased, revenue generated or assured, and digital scale achieved. It’s also vital that any automation initiative plays back results to non-IT stakeholders in terminology they’ll find compelling. If you’re unsure what the best frame is, connect with an ally from the marketing team to help you showcase your team’s results in the most high impact way possible. 

We’ve just barely touched the surface of what automation can help us achieve, and it’s exciting to imagine the productivity leaps we’ll be able to achieve together. Just imagine how much further we can go. As with any complex topic, there will no doubt be many more lessons and best practices we can all observe and internalize in the coming years. 


If you’re interested in diving deeper into the topic of network transformation, then I encourage you to also read our eBook: Opportunities, Challenges, and Solutions in the SD-WAN Era.

Subscribe to the ThousandEyes Blog

Stay connected with blog updates and outage reports delivered while they're still fresh.

Upgrade your browser to view our website properly.

Please download the latest version of Chrome, Firefox or Microsoft Edge.

More detail