Network testing and diagnostic procedures

One of the most important reasons to use the Blue Cedar Connect Gateway is to allow mobile apps to seamlessly connect to remote resources from any device. However, this connectivity can be disrupted for a variety of reasons and can be tricky to diagnose.

The Connect Gateway includes diagnostic utilities that aid in debugging network connectivity issues (during troubleshooting) and validating proper gateway configuration (during gateway setup).

Use these diagnostic utilities to check connectivity for both the gateway and the mobile app:

• Verify the gateway can send and receive traffic through the backend network.
• Verify the mobile client (the Blue Cedar-secured app) can send and receive traffic through the backend network. The (gateway) tools can mimic sending traffic from the mobile app's network (for IPv4 networks only).

The following sections give an overview of and some guidance for each utility. 

Checking gateway connectivity

These diagnostic utilities are available on the gateway to check connectivity and network traffic. First check the state of the gateway itself, then use the same tools to mimic a mobile client; see Checking mobile client connectivity for syntax.

Pinging endpoints to test gateway traffic

About ping

The ping command is a common method for troubleshooting the accessibility of network devices. It helps determine latency and packet loss, factors that tell you about the round-trip delay in communicating with the destination.

For ping to work, your network and endpoints need to have ICMP enabled. This means that if ICMP is blocked in your network, ping fails. However this failure is not indicative of blocked network traffic—it just means ping doesn't work without ICMP.

See CLI Command Reference for a list of ping options.

To validate if the gateway can reach a network resource, you can ping from the gateway's private interface. If your mobile client address pool is set to NAT addressing then pinging a network resource in this manner is a valid test for both gateway traffic and mobile client traffic: both traffic sources are sourced from the gateway's private interface IP address. 

To perform the ping, log into the gateway and use the following command in operational mode:

> network ping 8.8.8.8
Initiating ping (Ctrl-C to cancel)...
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=116 time=14.6 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=116 time=14.3 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=116 time=14.4 ms

If the destination is unreachable, you may see messages like "38 packets transmitted, 0 received, 100% packet loss" or "Network is unreachable." Please ensure ICMP is enabled in your network.

Using traceroute to check routes from the gateway to endpoints

About traceroute

Traceroute is another utility for troubleshooting network issues. Use traceroute to find the routes that the packets actually take when traveling to their destination. Traceroute shows what network devices are between one point and another, and some data about what those devices are doing. It can also help diagnose routing issues, especially when there is more than one network connection.

Traceroute uses UDP to send packets and may receive some ICMP Type 11 (TTL Exceeded) packets in return. For traceroute to fully operate, ensure that outbound UDP traffic is open on ports 33434–33464.

See CLI Command Reference for a list of traceroute options.

To check the routers along the path to the domain you're trying to reach, you can use traceroute from the gateway's private interface. If your mobile client address pool is set to NAT addressing, then this traceroute is a valid test for both gateway traffic and mobile client traffic: both traffic sources are sourced from the gateway's private interface IP address.

To perform the traceroute, log into the gateway and use the following command in operational mode:

> network traceroute 10.10.99.99
Initiating traceroute (Ctrl-C to cancel)...
traceroute to 10.10.99.99 (10.10.99.99), 30 hops max, 60 byte packets
send: Operation not permitted

The above error indicates that the packet is being routed out the public interface to an external server. Such routes are not allowed: traceroute is intended to find the route that a packet takes via the private interface.

Using dns-lookup to validate Gateway DNS configuration and endpoint name resolution

About dns-lookup

The dns-lookup utility is useful to validate DNS configuration, to locate the IP addresses associated with a domain name, and to check that DNS (Domain Name System) resolution is working for the hosts. This helps you identify if the issue is with the client, a network DNS server, or network firewall rules. (With the dns-lookup command, you can specify any DNS server.)

To validate that the gateway or mobile client can resolve hostnames or FQDN names, run dns-lookup on the hostname or FQDN to make sure they can be resolved.

See CLI Command Reference for a list of dns-lookup options.

Use dns-lookup to validate if the gateway can resolve hostnames from the gateway's private interface. If your mobile client address pool is set to NAT addressing, then using dns-lookup to resolve mobile client DNS resolution in this manner is a valid test for both gateway traffic and mobile client traffic: both traffic sources are sourced from the gateway's private interface IP address. 

If the DNS returns details about the IP address or domain name, your gateway or app is configured and connected. Use dns-lookup with a hostname to look up the IP address, which is shown in the Answer section of the response. 

To perform the dns-lookup, log into the gateway and use the following command in operational mode:

> network dns-lookup hostname

Use the -x option for a reverse lookup, that is, starting with an IP address:

> network dns-lookup -x IP-address

You can use the default name server configured on the gateway (with the above syntax) or specify a name server:

> network dns-lookup hostname nameserver
> network dns-lookup -x IP-address nameserver

To find the default name server, use "network dns-lookup" by itself: the name server being used by the gateway is shown as the "Server" entry.

Use the -t option to specify a record type of interest. (See CLI Command Reference for dns-lookup options.)

For example, to return the IPv6 address for the "google.com" domain, use -t AAAA:

> network dns-lookup google.com -t AAAA
Initiating dns-lookup (Ctrl-C to cancel)...
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 23394
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4000
;; QUESTION SECTION:
;google.com.			IN	AAAA

;; ANSWER SECTION:
google.com.		299	IN	AAAA	2607:f8b0:4006:811::200e

;; Query time: 34 msec
;; SERVER: 10.42.32.80#53(10.42.32.80)
;; WHEN: Thu Nov 19 04:14:47 UTC 2020
;; MSG SIZE  rcvd: 67

If dns-lookup cannot find a DNS entry, the response does not include an Answer section. 

Using test-web-request to verify HTTP(S) connectivity, ports, and TLS/SSL trust via the gateway

About test-web-request

The test-web-request utility is another tool to diagnose potential connectivity issues. Use this command to determine if the gateway can reach a specified URL, and for HTTPS requests, determine if the gateway can trust the identity certificate of that resource. 

Although the ping, traceroute, and dns-lookup commands use different ports than actual gateway requests, the test-web-request command actually makes a request via the configured ports, that is, port 80 for HTTP, port 443 for HTTPS, or the port specified in the URL.

See CLI Command Reference for a list of test-web-request options.

Use test-web-request to validate that the gateway can reach HTTP(S) endpoints (https://www.trusted-resource.com in this example). HTTP(S) requests can be used to test web servers, SCEP servers, or LDAP server reachability. If your mobile client address pool is set to NAT addressing, then using test-web-request to validate in this manner is a valid test for both gateway traffic and mobile client traffic: both traffic sources are sourced from the gateway's private interface IP address.

> request test-web-request url https://www.trusted-resource.com

This is an example of a successful request:

result "Request to https://www.trusted-resource.com succeeded.";

This example indicates a certificate issue:

"Peer certificate cannot be authenticated with given CA certificates";

Typically, this result means trusted certificates for the gateway to reach this URL are not in place—either they have expired or they need to be added. For more information about certificates, see Understanding certificates used by Blue Cedar products and Configuring AAA Public Key Infrastructure.

Checking mobile client connectivity

If the mobile client address-pool is configured on the gateway as static or dhcp addressing, you can use the same utilities described above and emulate a mobile client trying to reach a network resource via the gateway by specifying a source IP address from the mobile client address pool. The following sections describe the option to do this for each utility.

If you have multiple address pools configured, it's a good idea to use a test address from each pool.

Note: If the source address you choose for a network command is already assigned to a mobile client app, that app may see traffic interruptions while the network command is being executed on the gateway. This applies to ping, traceroute, and dns-lookup.

Pinging endpoints to test mobile client traffic

If the mobile client address-pool is configured on the gateway as static or dhcp addressing, you can validate if the mobile client can reach a network resource via the gateway by specifying a source IP address from the mobile client address pool.

For example, to do this with a static address pool, choose an address from the pool, and use it as the source IP address (with the -I option) to ping a device on your internal network. Use the show command in the example to find the range of addresses in the pool:

> show status operational context default addressing-operational
addressing-operational {  pool-type static;
  address-pool mypool {
    start 192.168.40.0;
    end 192.168.50.254;
    ...
  }
}

> network ping -I 192.168.50.3 8.8.8.8
Initiating ping (Ctrl-C to cancel)...
PING 8.8.8.8 (8.8.8.8) from 192.168.50.3 : 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=128 time=19.1 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=128 time=15.8 ms

Using traceroute to check routes from the mobile client to endpoints

If the mobile client address-pool is configured on the gateway as static or dhcp addressing, you can emulate a mobile client trying to reach a network resource via the gateway by specifying a source IP address from the mobile client address pool. Choose a source address from the address pool as described for ping above, and specify it with the -s option.

> network traceroute -s 192.168.50.3 8.8.8.8
Initiating traceroute (Ctrl-C to cancel)...
traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 60 byte packets
1 gateway (10.42.32.1) 0.835 ms 1.196 ms 1.340 ms
2 gi0-1-1-6.1288.agr22.bos01.atlas.cogentco.com (38.88.209.1) 6.708 ms 6.799 ms 6.848 ms
3 be3665.ccr32.bos01.atlas.cogentco.com (154.54.87.109) 7.154 ms 7.508 ms 7.560 ms
4 be3472.ccr42.jfk02.atlas.cogentco.com (154.54.46.34) 13.769 ms be3471.ccr41.jfk02.atlas.cogentco.com (154.54.40.154) 13.967 ms 13.972 ms
5 be3294.ccr31.jfk05.atlas.cogentco.com (154.54.47.218) 14.182 ms 14.503 ms 14.353 ms
6 tata.jfk05.atlas.cogentco.com (154.54.12.18) 14.663 ms 11.031 ms 10.846 ms
7 if-ae-12-2.tcore1.n75-newyork.as6453.net (66.110.96.5) 12.085 ms 12.054 ms 12.082 ms
8 72.14.195.232 (72.14.195.232) 12.671 ms 17.691 ms 12.868 ms
9 108.170.248.97 (108.170.248.97) 12.688 ms 12.689 ms 13.469 ms
10 216.239.62.147 (216.239.62.147) 13.546 ms 142.250.46.197 (142.250.46.197) 14.046 ms 172.253.70.7 (172.253.70.7) 15.424 ms
11 dns.google (8.8.8.8) 15.421 ms 15.034 ms 14.863 ms

See CLI Command Reference for a list of traceroute options.

Using dns-lookup  to validate mobile client DNS resolution

If the mobile client address-pool is configured on the gateway as static or dhcp addressing, you can emulate a mobile client by specifying a source IP address from the mobile client address pool. Choose a source address from the address pool as described for ping above and specify it with the -b option. It is a good idea to perform this test with a client IP address from both the lower and upper bounds of the pool's address range to confirm the entire address pool is allowed through the network firewall.

In this example, dns-lookup uses the virtual interface source address (192.168.50.3) to look for the IPv4 address of google.com via Google's DNS server (8.8.8.8). In the answer section of the response, 172.217.12.142 is returned as the IPv4 address of google.com. 

> network dns-lookup -b 192.168.50.3 google.com 8.8.8.8
Initiating dns-lookup (Ctrl-C to cancel)...
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 32175
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
 
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;google.com. IN A
 
;; ANSWER SECTION:
google.com. 193 IN A 172.217.12.142        
 
;; Query time: 31 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Thu Jul 30 22:38:53 UTC 2020
;; MSG SIZE rcvd: 55

This example emulates a mobile client (-b 192.168.50.3) to look for the IPv6 address (-t AAAA) of google.com via DNS server 1.1.1.1:

> network dns-lookup -b 192.168.50.3 google.com 1.1.1.1 -t AAAA
Initiating dns-lookup (Ctrl-C to cancel)...
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 40043
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;google.com.			IN	AAAA

;; ANSWER SECTION:
google.com.		212	IN	AAAA	2607:f8b0:4006:804::200e

;; Query time: 6 msec
;; SERVER: 1.1.1.1#53(1.1.1.1)
;; WHEN: Thu Nov 19 04:36:52 UTC 2020
;; MSG SIZE  rcvd: 67

Using test-web-request to validate the mobile client can reach HTTP(S) endpoints

You can also specify a source address (if the address pool is configured as static or dhcp) to emulate a mobile app and test whether the gateway trusts the certificates needed from that source address to the target URL. Choose a source address from the address pool as described for ping above.

> request test-web-request src-ip-address 192.168.50.3  
   url https://www.trusted-resource.com

If the endpoint is TLS/SSL protected you will most likely see a trust failure. This means you reached the server, but failed to establish trust from the emulated client. Please see Bypassing SSL trust failure to test SSL traffic for the mobile client below on how to bypass this error if needed.

"Peer certificate cannot be authenticated with given CA certificates";

If there is an issue with port availability, the request times out. To test SSL, make sure the firewall has port 443 open for HTTPS and port 80 for non-SSL traffic. See Configuring firewall ports for the gateway for more details.

Bypassing SSL trust failure to test SSL traffic for the mobile client

Although the source address emulates a mobile client, it is not the mobile client and does not have any required trust certificates to establish SSL/TLS trust. Thus you will get a trust error (cannot be authenticated) unless you bypass the certificate requirement. There is no need to add SSL public certificates to the gateway's trust certificate storage to test the client's ability to send SSL traffic.

To bypass SSL trust failure and see the HTTPS test connection succeed, enable this setting in configure mode:

% set aaa pki insecure-server-certificate-verification enabled
% commit

Then return to operational mode and perform the test-web-request:

> request test-web-request src-ip-address 192.168.50.3 url https://www.trusted-resource.com
result "Request to https://www.trusted-resource.com succeeded.";

To stop bypassing SSL trust failure and see the HTTPS URLs fail to trust the server, disable the same setting in configuration mode:

% set aaa pki insecure-server-certificate-verification disabled
% commit

The test-web-request command returns with this error: "SSL peer certificate or SSH remote key was not OK."