Will encryption and regulation obsolete DPI?

The death of DPI is exaggerated, but the rise of ETI is not

Network traffic intelligence, including deep packet intelligence (DPI) technology, has been a critical capability in enabling resource allocation, security protections, and network visibility since the early days of IP networking. Meanwhile, regulators and cybersecurity advocates continue to push for more encryption to enhance privacy and security on the Internet and enterprise networks.

All this push for increased privacy presents a conundrum. End-to-end encryption protects information from leakage and compromise by malicious third parties. Yet, the same encryption causes a loss of visibility and control for network administrators, and limiting DPI can prevent security protections from being effective (for example, scanning encrypted web downloads for malware). This visibility is also essential for analytics, policy control, and quality of service management.

As we explain in this article, we don’t expect DPI to go away anytime soon. However, the continued desire for more privacy will drive network operators across enterprises and carriers to turn to advances in encrypted traffic intelligence (ETI) to complement traditional DPI.

The ongoing importance of network visibility

Network visibility is the ability to inspect traffic and discern critical details such as origin, destination, underlying applications and services, technical characteristics like packet size distribution and packets per flow, performance attributes like speed and latency, and behavioral patterns.

This complete picture has become the cornerstone for efficiently operating and securing modern heterogeneous networks. As networks evolve to be more complex, with hybrid multi-cloud architectures, SD-WAN, SASE, and hybrid work models, network visibility is more critical than ever.

For today’s heterogeneous global networks, traffic visibility enables functions like:

• Granular traffic analytics to optimize infrastructure utilization and planning
• Intelligent traffic management for improved performance and customer experience
• Dynamic policy control for usage, access, and charging based on applications
• Rapid threat detection by analyzing traffic patterns and payloads
• Continuous monitoring to identify anomalies and troubleshoot issues proactively

It’s hard to argue that these functions will be less critical in today’s and tomorrow’s increasingly connected world, with networks spanning access, transport, core segments, and data transmission over fiber, mobile wireless, cable, and satellite links. Likewise, ensuring effective traffic management for sophisticated communication applications like unified collaboration, shared design and development, group gaming, real-time industrial controls, telemedicine, or the Internet of Things requires more visibility, not less.

Why is DPI challenging today?

DPI examines packet payloads to accurately identify applications, services, and protocols and extract other attributes. However, encryption standards like transport layer security (TLS), IPSec, and secure shell (SSH) encrypt the packet payload, concealing critical data needed for visibility.

Modern encryption protocols and standards increasingly restrict what data is exposed and analysis DPI tools can perform; examples include:

• TLS 1.3
• TLS encrypted client hello (ECH) which replaces TLS encrypted server name indication (ESNI).
• QUIC (and the accompanying HTTP/3), an increasingly popular secure transport alternative for modern browsers supported by sites like Google, Meta/Facebook, and content delivery networks like Akamai and Cloudflare.
• DNS-over-HTTPS/TLS

With newer protocols, metadata in the packets are often encrypted, making it harder to glean information from the encrypted payload or during protocol handshakes.

Encryption aside, privacy advocates are pushing lawmakers to limit “peeking” into private information by prohibiting DPI beyond layer 3 IP headers. This prohibition creates similar traffic inspection challenges as encryption does.

This lack of visibility severely impacts essential network functions:

• Security - inability to track malware, data leaks, insider threats
• Traffic Management - can’t efficiently route or prioritize traffic based on application awareness
• Analytics - can’t gather granular application and user insights for capacity planning
• Policy Control - can’t enforce dynamic policies based on users, applications, etc.
• Performance Monitoring - limits root cause analysis and detailed monitoring

Nevertheless, if TLS proxy (man-in-the-middle) inspection is allowed for enterprise use cases, DPI can still be used as a critical visibility tool. Note that traditional DPI will always require high-performance network processing, expose private information, and may occasionally prevent applications from working correctly.

Emergence of encrypted traffic intelligence

To address the encryption dilemma, a new approach called encrypted traffic intelligence (ETI) has emerged. ETI combines improved DPI with artificial intelligence (AI) and machine learning (ML) techniques to restore visibility into encrypted network traffic.

ETI solutions aim to deliver several key capabilities:

• Seamlessly integrate into existing physical and virtual network architectures
• Handle increasing traffic volumes and ratio of encrypted packets
• Provide granular classification down to specific applications and service types
• Identify the latest encrypted protocols and cipher suites
• Rapidly adapt to new and evolving protocols and encryption methods
• Identify obfuscated traffic that uses evasion tactics like randomization and mimicry
• Extract actionable insights from metadata for security and performance optimization
• Facilitate a zero-trust architecture that protects confidential data while enabling visibility

AI and ML are critical to ETI

Advanced ML algorithms like convolutional neural networks (CNN), recurrent neural networks (RNN), and long short-term memory networks (LSTM) have been shown by leading vendors and researchers to be able to analyze metadata, generate statistical fingerprints, identify behavioral patterns, and surface other heuristic information within encrypted network traffic.

These techniques can identify latent traffic patterns and self-learn from global, real-world data at enormous scale and speed. Some ETI vendors today use these techniques to pre-train models so that customers can benefit from accurate application classification out-of-the-box.

Using AI/ML, ETI solutions circumvent the fundamental issues with legacy decryption approaches. ETI can offer a future-ready solution that evolves with new encryption standards, evasion tactics, and tightening regulatory compliance.

Build vs. buy - partnering for success

Developing an AI/ML-based encrypted traffic intelligence system requires rare skill sets spanning both the networking domain and advanced ML. Additionally, optimization and performance tuning skills will be essential for efficient operation at carrier-grade scale.

Likewise, pre-training classification models require access to massive, diverse network traffic datasets to ensure accuracy across protocols, applications, and handle edge cases. Unfortunately, many network equipment providers, enterprises, and telecom providers lack specialized in-house resources and data assets.

Therefore, it makes strategic sense to partner with an experienced solution provider investing deeply in ETI innovation. Leveraging a partner’s purpose-built AI/ML algorithms, models, global datasets, and optimization expertise allows rapid deployment with reduced costs and risks. Plus, it provides an ongoing source of continued innovation that will be required as requirements and regulations change.

AvidThink’s conclusion

Pervasive encryption across networks can cause a critical loss of visibility and control. Next-generation traffic intelligence powered by a combination of DPI and AI/ML for encrypted traffic analysis can restore visibility without the overhead and downside of legacy decryption techniques.

ETI continues to support security protections, traffic optimization, granular analytics, and more, that today’s complex heterogeneous networks require. Partnering with the right vendor is essential as networks and customer needs evolve. Network equipment providers and network operators will want to carefully evaluate any network traffic intelligence vendor on their ETI capabilities before picking their technology partners.