Reaching a Private VM from Azure Managed Grafana Without VNet Peering

Azure Managed Grafana runs in a Microsoft-managed virtual network. You don’t own it, and you can’t peer it to anything. So when a data source needs to reach a service that only exists on a VM inside one of your own VNets — a log server, an internal API, anything that isn’t a recognized PaaS resource — the usual “just peer the networks” answer is off the table.

The supported path is a Managed Private Endpoint on the Grafana side, pointed at a Private Link Service (PLS) you publish on your own VNet, which in turn sits in front of a Standard Internal Load Balancer (ILB) fronting the VM. Three building blocks, one private data path, no public exposure. I recently built this end to end, and while the architecture is clean, the failure modes along the way are the kind that waste an afternoon each if you don’t know what you’re looking at. Here’s the pattern and the five gotchas that actually surfaced.

Why not VNet peering

A Managed Private Endpoint is the only way to reach into a customer VNet from Grafana’s managed network. It can target two kinds of things:

• A recognized Azure PaaS resource (Storage, SQL, Key Vault, App Service, …) — Grafana already knows the DNS zone and groupId for these.
• Your own Private Link Service — the fallback for anything else, including a plain VM-hosted application sitting behind a load balancer.

Since a VM behind an ILB doesn’t fit any PaaS groupId, the Private Link Service is the mechanism, not a workaround.

Architecture

 ┌─────────────────────────┐        ┌───────────────────────────────────┐
 │   Azure Managed Grafana  │        │        Target VNet (yours)         │
 │   (Microsoft-managed VNet)│       │                                     │
 │                           │        │  ┌───────────────┐   ┌──────────┐ │
 │  ┌─────────────────────┐ │  PLS   │  │ Private Link   │   │ Internal │ │
 │  │ Managed Private      │─┼───────►│  │ Service (PLS)  │──►│ Load     │ │
 │  │ Endpoint             │ │  NAT   │  │ (NAT subnet)   │   │ Balancer │ │
 │  └─────────────────────┘ │        │  └───────────────┘   └────┬─────┘ │
 │                           │        │                            │      │
 └─────────────────────────┘        │                       ┌────▼────┐ │
                                      │                       │   VM /  │ │
                                      │                       │ Backend │ │
                                      │                       │ Service │ │
                                      │                       └─────────┘ │
                                      └───────────────────────────────────┘

Request flow, step by step

1. A Grafana data source sends a request to the hostname configured on the managed private endpoint (the privateLinkServiceUrl field).
2. Grafana’s managed network resolves that hostname internally to the managed private endpoint’s own private IP. This DNS mapping is set up automatically when the endpoint is created — it is not a public or customer-managed record.
3. Traffic tunnels through the Private Link infrastructure to the PLS in the target VNet.
4. The PLS’s NAT subnet IP is the actual entry point on the target VNet side. From here, standard Azure networking (NSGs, routing) applies as if traffic originated from that NAT IP.
5. The PLS forwards to the load balancer frontend IP configuration it was created against.
6. The Standard Load Balancer performs DNAT only — it rewrites destination IP/port from the frontend to a healthy backend VM, preserves the source IP (the PLS’s NAT IP), and does not terminate TLS or rewrite the Host header.
7. The backend VM’s NSG evaluates the packet as arriving from the PLS’s NAT IP, destined for the backend VM’s own IP and backend port — not the load balancer’s frontend IP/port. This is the detail most people get wrong first.
8. The backend application receives the request and responds.

Five things that will actually trip you up

1. NSG rules must target the backend VM, not the load balancer frontend

Because DNAT happens before the packet reaches the backend subnet’s NSG, a rule written as “allow <client> → <load balancer frontend IP>” matches nothing — ever. The correct rule is “allow <PLS NAT IP> → <backend VM IP>” on the backend port, not the frontend port. A high-priority catch-all deny rule (the DenyVirtualNetworkOutBound style) will swallow this silently, with no error that points at the real cause.

2. PLS approval is manual by default

A Private Link Service’s autoApproval list is empty out of the box, so every new managed private endpoint connection lands in Pending until someone approves it — Portal → Private Link Service → Private endpoint connections → Approve, or:

az network private-link-service connection update --status Approved

Until approved, there is no working data path. This shows up as a DNS or connection failure on the Grafana side, not an obvious “pending approval” message, so it’s easy to chase the wrong thing first.

3. The domain name field has real validation rules

When targeting a PLS instead of a recognized PaaS groupId, Grafana requires you to supply the hostname it maps to the private endpoint. The value is arbitrary — it doesn’t need to resolve publicly or match a real DNS zone — but:

• It needs at least 3 dot-separated labels (e.g. something.example.internal). A bare hostname like myhost is rejected with PrivateLinkServiceInvalidHostname.
• Avoid .local. It’s reserved for mDNS under RFC 6762, and many embedded/container DNS resolvers special-case .local and route it toward multicast DNS instead of a normal lookup — which fails outright with no responder present, producing a server misbehaving / SERVFAIL error that has nothing obviously to do with Private Link. Use .internal or a made-up subdomain of a domain you actually control.

4. The load balancer won’t save you on protocol mismatches

A Standard Load Balancer is Layer 4 only. If the client requests https:// and the backend only speaks plain HTTP on that port, you get “server gave HTTP response to HTTPS client” — the load balancer can’t terminate or re-originate TLS, and it can’t rewrite the Host header the way an Application Gateway would.

5. The backend may reject the Host header you’re forced to use

Many modern frameworks (Kestrel-based .NET apps in particular) enforce an allow-list of valid Host header values and return a generic 400 Bad Request for anything else. This one is nasty because:

• DNS, NSG, load balancer and TCP handshake all succeed.
• The application error gives no hint that Host is the problem.
• If the same backend is also reachable through a Layer 7 device elsewhere, that device may already be silently rewriting Host to what the backend expects — masking the requirement until this new Layer-4-only path exposes it.

The private-link hostname constraint from #3 (3+ labels) and the backend’s allow-list frequently can’t both be satisfied by naming the private link exactly what the backend wants. If your calling client supports a distinct Host-header override separate from the request URL, that’s the more reliable fix than fighting the naming constraints against each other.

Troubleshooting checklist

What I’d do differently

1. Approve the PLS connection before debugging anything else. Every other symptom in the checklist above looks superficially similar to a stuck approval. Check approval state first and rule it out in thirty seconds instead of chasing NSGs.
2. Pick the private-link hostname up front, with both constraints in mind. Deciding on a .internal-style FQDN with 3+ labels before you hit the ARM validation error (or the backend’s Host-header rejection) saves a round trip.
3. Test the backend directly before testing through Grafana. Hitting the VM’s ILB frontend from a jump box in the same VNet isolates whether a failure is in the Private Link path or in the application itself — cheaper than debugging both layers at once.
4. Write the NSG rule against the backend VM’s IP and port from the start. It’s the single most common mistake in this pattern, and it fails silently rather than with a helpful error.

Summary

This pattern lets a managed PaaS service reach a private, VM-hosted resource without VNet peering, by chaining a Managed Private Endpoint → Private Link Service → Internal Load Balancer. The trade-off is that the path is Layer 4 only end to end unless you introduce a Layer 7 device, so protocol and Host-header expectations on both ends have to line up manually. Nearly every failure mode above comes from that constraint rather than from the Private Link mechanics themselves.