Scenario: Introduction

Think of an organization that maintains most of its IT infrastructure on Azure. It applies a segmentation strategy by branch office, where the assets underlying each regional branch office are deployed to their specific landing zone subscription, i.e. SUB-BRANCH-A, SUB-BRANCH-B, etc.

The landing zones share the same Microsoft Entra ID Tenant as their Identity Provider, and may span several Azure regions.

The subscription SUB-OPERATIONS is a technical subscription whose resources implement common services for all the other landing zones. For example, monitoring services, as well as the distribution and management of secrets from the common Tenant to Key Vaults deployed to each particular branch subscription, are two examples of shared services. The development, implementation, maintenance and operation of the common services are the responsibility of specialized operational IT teams.

Security Aspects of the Current Design

As the diagram clearly shows, all communication is secured in two ways:

1. As a first line of defense, access is controlled by identity, through the authentication mechanism implemented at the tenant level. It’s based on Microsoft Entra ID with users, groups, Managed Identities and App Registrations and secrets.
2. At the Transport Layer, all data exchange, including the authentication data, is transmitted on the public internet, and is secured through HTTPS.

This can be considered a baseline design in terms of security. It meets the minimum requirements for industry-grade solutions, but falls short of some security practices that would eliminate well-known security risks.

Some examples of open security risks concerning this design are:

1. An attacker, especially an insider, can obtain (hijack) a valid authorization token for any public-facing resource and use it to break in undetected. There is no additional security layer to prevent exploitation and exfiltration, for example, of the secrets stored in the Key Vaults. The following provides a simple example of how this exploit could compromise the data stored in a Key Vault, which is a typical high-value target: How to Restrict Network Access to Azure Key Vault Using Firewalls & Vnets.
2. The transmission of data on the public internet, although encrypted through TLS, is a public network communication; therefore, it carries corresponding well-known risks and exploits, including data harvesting, etc. Furthermore, it relies on public infrastructure, with all the potential performance downsides.
3. All resources engaging in communication require a fully open public IP address, which makes them accessible from the public internet and therefore intrinsically vulnerable targets.

Goals of the Improved Design

The current implementation enables data flow between SUB-BRANCH-X and SUB-OPERATIONS over the public internet via HTTPS, relying on public IP addresses to provide access to resources. Furthermore, the security is implemented at the following OSI layers:

1. Transport layer (layer 4 OSI model) via HTTPS
2. Session layer (layer 5 OSI model) via identity and Microsoft Entra ID Identity Provider

It’s possible to improve the security of this solution by transmitting data through private IP routes within the Azure backbone and by removing the public endpoint of any service whenever possible, or if not possible, by applying a downstream firewall.

Design Strategy

The design strategy involves making as few changes as possible to the current design at any one time, and gradually transforming it by increasing its complexity in a step-by-step fashion until a new design emerges that meets as many of the design goals as possible.

Improvements Round 1

1. Shift as much of the data traffic as possible from the public internet to the Azure Backbone using Service Endpoints
2. Improve access control to PaaS services through their Service Firewall and network rules that grant access only to known subnets advertised to them as Service Endpoints

Let’s focus on app-a1 and its backend app-ops-1 to illustrate the point and make it easier to introduce the essential topic of Service Endpoint. The article Premium Data Security on a Budget with Azure offers a gentle introduction to this topic; in essence, the technology allows sub-a1-1 to advertise its private IP range to the PaaS service App Service, the backend server app-ops-0.

This has the following important consequences:

1. The traffic between any deployed assets within sub-a1-1 to the public IP address of app-ops-1 will be optimally routed over the private Azure Backbone, which is completely isolated from and independent of the public internet and not accessible from the agents on the public internet. This single-handedly eliminates the corresponding security risks and significantly improves performance.
2. It’s now possible to implement a downstream (cost-free) Service Firewall of app-ops-1 and add an explicit rule to allow traffic only from suba1-1. This makes it impossible for any attackers, including insiders possessing a valid hijacked access token for the backend app-ops-1, to access it unless they operate from the sub-a1-1 subnet.

The same pattern can be replicated and expanded elsewhere in the design providing the same advantages on multiple fronts and integrations. However, before discussing this in detail, a new topic must be introduced:

• Azure App Service Virtual Network Integration – Integrate your app with an Azure virtual network
• Enable virtual network integration in Azure App Service
• How to secure your Azure App Services – dotnetFlix
• Azure App Service and Virtual Network Integration Options

In summary, you can associate an Azure VNet (a virtual network within Azure) as the outbound traffic network of an App Service resource (this includes Azure Functions and Web Jobs). This opens up several possibilities to improve the security posture of the design by leveraging the same Service Endpoint mechanism (Virtual Network service endpoints), just illustrated for the communication from app-a-1 to app-ops-1.

The crucial point to understand from the documentation, Virtual Network service endpoints, is that Service Endpoints offer a simple mechanism to securely connect a network and its resources to many Azure services, improving the security posture considerably with minimal effort at no cost.

1. The se-keyvault takes a private IP address from the outbound network delegated-sub-0 and advertises it to the network boundary for kva1. A network rule can now be enforced on kv-a-1 to allow traffic only from the backend network delegated-sub-0. The cost-free, integrated service firewall of kv-a-1 can now be used to block all traffic from the public internet except for any known IP address ranges as required:
   • How to Restrict Network Access to Azure Key Vault Using Firewalls & Vnets,
   • Virtual network service endpoints for Azure Key Vault.
2. The same pattern is applied between app-ops-1 and the Event Hub eh-a-1: Allow access to Azure Event Hubs namespaces from specific virtual networks – this example is considered in this design because Event Hub can be easily integrated with Azure Key Vault and Event Grid to make it easier to publish and subscribe to other subscriptions as an alternative:
   • Understand event domains for managing Event Grid topics
   • Event hub as an event handler for Azure Event Grid events
3. In the same way, the service endpoint se-sqlserver allows the backend server app-ops-a1 to connect to the backend server on an optimized private route, and now sql-srv-0 can be isolated from the public internet using the integrated service firewall, except for trusted IP ranges if required:
   • Use virtual network service endpoints and rules for servers in Azure SQL Database
   • Azure SQL Database Networking | Connection using Service Endpoint and Private Endpoint

Improvement Round 2

The security of the design can be improved further in the following way.

Suppose that app-a-1 is a front-end app or any app that uses app-ops-1 as its backend API server, while it’s also the case that the app-a-1 can also be used by app-ops-1 to push data or perform queries on resources located on SUB-BRANCH-X.

Then by choosing to implement the receiving end of this communication app-a-1 as an App Service (a PaaS) rather than an Azure VM (IaaS), there is the opportunity to retain the integration app-a-1 => app-ops-1 by using Azure App Service Virtual Network Integration with delegated-sub-1 and the the SE se-appservice and at the same time use the se-app-ops-1 from delegated-sub-0 to enforce secure communication from the opposite side.

At this point, app-a-1 becomes a convenient PaaS and can also be isolated through its downstream, free downstream service firewall, as was the case in all the preceding examples.

Improvement Round 3

Rounds 1 and 2 of security improvements are based on Virtual Network service endpoints and introduce additional security controls at the Application-Layer (layer 7 of the OSI Model) in the design. However, this step introduces a new concept to illustrate another mechanism that adds security control at the Network-Layer (layer 3 OSI Model) / Data-Link (layer 2).

This is the integration of eg-a-1 with app-ops-1, from the Event Grid in the subscription SUB-BRANCH-X to the backend server. This time we implement the Network security for Azure Event Grid resources – Private Endpoint. The Azure Event Grid is a subscription-level global service, and it offers mechanisms to subscribe to event cross-subscription and even cross-tenant:

• Understand event domains for managing Event Grid topics
• Event-Grid: Cross-tenant event delivery using a managed identity

However, there is a much simpler way to implement this integration by using a private endpoint. This integration is suitable and preferable when:

1. Security of the information passed in the event is essential
2. Simple implementation is more important than the cost, or when the volume and frequency of the event are modest, to ensure low running costs

In this case, for example, we assume that the backend server app-ops-1 must be the recipient of the Near Expiry Event (NEE) emitted for the secrets stored in the Key Vault kv-a-1. This is a well-documented, consolidated pattern:

• Monitoring Key Vault with Azure Event Grid
• Automate the rotation of a secret for resources that use one set of authentication credentials – Microsoft
• Rotation tutorial for resources with two sets of credentials

In the implementation represented in the schematic, the Private Endpoint pe-del-sub-0-1 is reserved to resolve to the DNS zone pl-a-1, and can pull the NEE events from the Event Grid through a dedicated private route.

This is based on the following recommended patterns, in particular the mechanism of Pull delivery:

• Deliver events securely over a private link
• Configure private endpoints for Azure Event Grid namespaces
• Network security for Azure Event Grid resources

The difference to the solution based on the service endpoint is that, in this case, there is no longer a public IP address; therefore, the channel is truly between private networks. There is no longer the possibility of accidentally opening a public IP address by misconfiguration of the service firewall at one end of the route.

This security improvement comes at a cost as, contrary to service endpoints which are completely free in Azure, private endpoints add some costs (Azure Private Link pricing), although in this specific case this cost increment will be very negligible as the traffic between the backend service and the Event Grid to detect NEE events is bound to be minimal in all practical cases.

Improvement Round 4

Finally in this last round of improvements, you can see that the technique based on the private endpoint is definitely not an isolated case with the Event Grid; instead, it;s a consolidated Azure pattern for security.

In this case, the SQL Server can be completely isolated from the internet by removing its public IP address and replacing it with the private endpoint pe-del-sub-0-2 in the outbound network of the backend server, which resolves over private DNS to the private IP address of the Azure SQL Server sql-srv-0. The backend server will now have exclusive access to the SQL Server as it should be in most productive scenarios.

• Azure SQL Database Networking | Connection using Service Endpoint and Private Endpoint
• Tutorial: Connect to an Azure SQL server using an Azure Private Endpoint using the Azure portal
• Azure Private Link for Azure SQL Database and Azure Synapse Analytics

The same concept may be applied to other parts of this design in that, in general, any integration based on a service endpoint can be replaced by an integration based on a private endpoint / private link as illustrated.

The factors to consider in the process of replacing service endpoints with private endpoints are the following:

• Whether removing a public endpoint is actually possible, or whether there are exceptions in place where access to that resource from a predefined IP public range may be required, i.e. for operational reasons.
• Whether the additional cost caused by network traffic is compatible with the operational budget.
• Whether it’s acceptable to deal with the operational management of DNS. Private endpoints are automatically configured with the corresponding private Azure DNS at creation, but if company policies require the use of dedicated DNS Servers, then resolution to the associated private link resource must be configured and maintained manually by the administrator of the DNS Server.

Conclusion

The original design implements basic industry-standard security features:

1. Transport layer (layer 4 OSI model) via HTTPS
2. Session layer (layer 5 OSI model) via identity and Microsoft Entra ID Identity Provider

The improved design adds the following security features:

1. Application-Layer (layer 7 of the OSI Model) via service endpoint and network integration
2. Network-Layer (layer 3 OSI Model) / Data-Link (layer 2) via private endpoints

In addition:

• This is achieved with minimal implementation effort, running costs and administrative overhead.
• Any opportunity for exfiltration of secrets from the Key Vault is almost completely eliminated and could be totally eliminated by implementing private endpoint access to the Key Vault.
• The pattern is applicable across any number of subscriptions under the same tenant or different tenants.
• The pattern would also work across different Azure Regions.

References

• Premium Data Security on a Budget with Azure
• Azure SQL Database Networking | Connection using Service Endpoint and Private Endpoint
• How To Restrict Network Access To Azure Key Vault Using Firewalls & Vnets
• Azure App Service Virtual Network Integration – Integrate your app with an Azure virtual network.
• Enable virtual network integration in Azure App Service
• How to secure your Azure App Services – dotnetFlix
• Azure App Service and Virtual Network Integration Options
• Virtual Network service endpoints
• Virtual network service endpoints for Azure Key Vault
• Understand event domains for managing Event Grid topics
• Event hub as an event handler for Azure Event Grid events
• Network security for Azure Event Grid resources
• Network security for Azure Event Grid resources – Private Endpoint
• Event-Grid: Cross-tenant event delivery using a managed identity
• Monitoring Key Vault with Azure Event Grid
• Automate the rotation of a secret for resources that use one set of authentication credentials – Microsoft
• Rotation tutorial for resources with two sets of credentials
• Deliver events securely over a private link
• Configure private endpoints for Azure Event Grid namespaces
• Network security for Azure Event Grid resources
• Azure SQL Database Networking | Connection using Service Endpoint and Private Endpoint
• Tutorial: Connect to an Azure SQL server using an Azure Private Endpoint using the Azure portal
• Azure Private Link for Azure SQL Database and Azure Synapse Analytics
• Azure App Service access restrictions
• Set up Azure App Service access restrictions

Symbols Table

These Solutions are Engineered by Humans

Did you learn from this article? Perhaps you’re already familiar with some of the techniques above? If you find cybersecurity issues interesting, maybe you could start in a cybersecurity or similar position here at Würth Phoenix.