Black Hat is a cybersecurity conference renowned for its intensity, its cutting-edge research, and its notoriously demanding network. Every year, the Black Hat Network Operations Center (NOC) faces the daunting task of maintaining a stable and reliable network for thousands of attendees, speakers, and exhibitors.
Troubleshooting App Performance at Black Hat Asia 2023
ThousandEyes has become an important part of the Black Hat NOC. At Black Hat Asia 2023, the Cisco team introduced ThousandEyes to solve an issue with the network latency. At the conference, an important sales application, used for engaging with event prospects, was having issues connecting to the server. The sales team reached out to the NOC leaders to report the application slowness, which they suspected might be due to the network.
Using Cisco ThousandEyes Cloud Agents, together with Endpoint Agent installed on laptops, we configured scheduled synthetic tests to probe the application domain. This immediately showed that consistent latency from the host device to the server was around 200ms, with frequent spikes up to 600ms (about half a second).
Furthermore, ThousandEyes helped visualize the traffic path for the app domain. Using this, we noticed that the domain was hosted in AWS (Amazon Web Services) in Dublin, with the traffic path going through Paris. Each hop added latency, which was causing the reported issues.
As a result of this problem-solving, Cisco was invited to bring ThousandEyes as an official part of the support for Black Hat, starting at USA 2023. No company can buy its way into the Black Hat NOC; it is invitation-only.
Going All In at Black Hat Asia 2025
Deploying ThousandEyes at Black Hat is a rigorous process involving a lot of hardware (some shown in Figure 1 below), configuration, testing, troubleshooting, and running around the conference center.
To enhance scalability and stability, we upgraded the hardware hosting the ThousandEyes agent from the previous Raspberry Pi to a more robust Orange Pi, and we managed the deployment of the agent and host configuration through an Ansible script. This way, when the training room assignments change, we can instantly push the updated network configuration.
We worked with Black Hat NOC team to design the best distribution of agents in the network, as part of infrastructure quality assurance. We also switched from the built-in wireless adapter to an external one to achieve more stable wireless measurements.
Pushing Boundaries
To enhance our visibility, we worked with the Arista team and successfully deployed a Docker agent in Arista’s data center switch. The Arista agent can operate within the default network space on the switches, which have 8GB of system memory, and can perform HTTP, DNS, and network layer tests.
Although agents on Arista switches are not currently supported by the Cisco support team, this proof of concept (POC) marks a significant breakthrough for the ThousandEyes team.
Enhancing the ThousandEyes Dashboard
In addition to our standard deployment tasks, we made several improvements to the service. First, we revamped our dashboard to compare the latency and throughput measurements to the SaaS target, the internal servers in each conference room, the Arista switches, and the NOC members' laptops. The color-coded grid widget always provides a crystal-clear visual indication of any issues.
We also tracked the agent response times.
ThousandEyes introduced a new feature this year: the global dashboard filter. This filter facilitates easy troubleshooting by filtering only agents or tests—instantly reconfiguring the entire dashboard. This flexibility helps us highlight degradation quickly.
Every dashboard widget comes with a drill-down link to quickly access the relevant tests, which are configured with proper filters and agents. This feature helps with quick troubleshooting. Additionally, in the test view, we could use the multi-metrics view to correlate the test results across the different layers to isolate the issue. In this case, ThousandEyes helped us identify network loss when we relocated the agents to different rooms, and we could immediately address this.
Streamlining Data With Splunk
To provide seamless integration between ThousandEyes and Splunk, a new ThousandEyes App for Splunkbase was released. With this app, we could easily link the Splunk Cloud and Enterprise platform to ThousandEyes via OAuth, using HEC to stream test telemetry, and automatically pull activity logs or events. Compared to the previous deployment using the streaming API, this saved a lot of time for integration. We pulled the relevant ThousandEyes metrics into the SNOC dashboard for network metrics, as well.
Gaining Visibility With iOS Agents
We were able to install ThousandEyes Mobile Endpoint Agents on the event registration iPads, which are currently in user testing phase (POV) before publication on the Apple App Store. ThousandEyes Mobile Agents were deployed as a package in iOS and used to monitor the registration website. At the same time, the agent was able to run the speed test on the fly for network measurement. The advantage of Mobile Agents is that we don’t need the extra hardware, and it’s suitable for measuring over Wi-Fi or 5G/LTE. In future NOCs, we could consider deploying them to all check-in iPads across the conference hall and meeting rooms to expand our visibility.
Identifying Wi-Fi Congestion
During the initial two days of training sessions at Black Hat, ThousandEyes agents showed only minor deviations from the baseline during a couple of afternoon sessions. However, multiple smaller session rooms started to show overall latency and link delays to the gateway for two hours.
We used the ThousandEyes Endpoint Agent to validate the Wi-Fi experience, and the Endpoint Agent’s single agent view helped identify the wireless telemetry captured. As my laptop switched the SSID to each room’s dedicated Wi-Fi, we observed a drop in signal quality during the session. After the session finished, this issue resolved itself.
Assuring the Black Hat Experience
Our experience at Black Hat demonstrates the power of ThousandEyes in providing network visibility and enabling rapid problem resolution in even the most demanding environments. As we look ahead, we're excited to continue exploring new ways to leverage ThousandEyes to enhance the Black Hat experience.
Want to see how other Cisco teams participated? Check out the Cisco Security blog: Black Hat Asia 2025 NOC: Innovation in the SOC.
Figure 13. Black Hat NOC showroom
Acknowledgments
Thank you to the Cisco NOC team:
-
Cisco ThousandEyes: Shimei Cridlig and Patrick Yong
-
Cisco Security: Christian Clasen, Shaun Coulter, Aditya Raghavan, Justin Murphy, Ivan Berlinson, Ryan Maclennan, and Jessica Oppenheimer
-
Cisco Meraki: Paul Fidler, with Connor Loughlin supporting
Also, to our NOC partners Palo Alto Networks (especially James Holland and Jason Reverri), Corelight (especially Mark Overholser and Eldon Koyle), Arista Networks (especially Jonathan Smith), MyRepublic and the entire Black Hat / Informa Tech staff (especially Grifter ‘Neil Wyler’, Bart Stump, Steve Fink, James Pope, Michael Spicer, Jess Jung, and Steve Oldenbourg).
About Black Hat
Black Hat is the cybersecurity industry’s most established and in-depth security event series. Founded in 1997, these annual, multi-day events provide attendees with the latest in cybersecurity research, development, and trends. Driven by the needs of the community, Black Hat events showcase content directly from the community through Briefings presentations, Training courses, Summits, and more. As the event series where all career levels and academic disciplines convene to collaborate, network, and discuss the cybersecurity topics that matter most to them, attendees can find Black Hat events in the United States, Canada, Europe, Middle East and Africa, and Asia. For more information, please visit www.blackhat.com.