RHEL Networking with NetworkManager, ip, ss and Routing
A network symptom crosses several decisions: link state, NetworkManager profile, address, route, neighbor resolution, local socket, firewall, SELinux, upstream policy and application protocol. Ping answers only one narrow question and may be blocked or deprioritized. Current RHEL administration uses NetworkManager for persistent connection ownership, with ip and ss for kernel evidence.
Keep persistent configuration and runtime evidence distinct
ip address and ip route show kernel state. nmcli connection shows persistent profiles and activation. A one-off ip change may work until the profile reactivates or the host reboots. Make owned persistent changes through NetworkManager, then verify the resulting kernel state.
DNS success, TCP connection and application response are separate. Test the destination name through the configured resolver, route selection for the resolved address, local/upstream policy, TCP/TLS negotiation and application behavior.
Walk the packet path in order
| Layer | Question to answer | Evidence |
|---|---|---|
| Link/profile | Which interface and profile own state? | nmcli device and connection UUID |
| Address/neighbor | Is local addressing valid and peer resolvable? | ip address, ip neigh |
| Route | Which source, gateway and table are selected? | ip route get and rules |
| Policy/socket | Is traffic permitted and is a process listening? | firewall/nft evidence and ss |
| Protocol | Does DNS, TLS and application exchange work? | resolvectl/dig, curl and service logs |
Change a remote profile without losing management
- Record console recovery, retained session and current profile UUID.
- Capture addresses, routes, rules, DNS and an application test.
- Create or clone a candidate NetworkManager profile rather than overwriting the only path blindly.
- Use an approved NetworkManager checkpoint or timed rollback where supported.
- Activate the candidate and test gateway, resolver, management and application paths.
- Confirm persistent state after reconnect/reboot, then remove only the superseded profile.
Inspect profile, route and socket ownership
nmcli general status
nmcli device status
nmcli -f NAME,UUID,TYPE,DEVICE,AUTOCONNECT connection show
nmcli connection show --active
ip -brief link
ip -brief address
ip rule show
ip route show table all
ip route get 203.0.113.10
ip neigh show
ss -lntup
resolvectl status 2>/dev/null || nmcli dev show | grep -E 'DNS|DOMAIN'
tracepath 203.0.113.10- Documentation address
203.0.113.10is not a live test destination; replace it with an approved endpoint. ip route getpredicts kernel selection, including source address; it does not prove upstream return routing.- A wildcard listener may still be blocked by firewall, SELinux or application access policy.
- Packet capture can expose credentials and customer data; filter narrowly and protect evidence.
Evidence and acceptance criteria
| Evidence | Healthy result | Failure meaning |
|---|---|---|
| Profile | Expected UUID owns the intended device and autoconnect | Competing or stale profile can reactivate |
| Address/route | Source, gateway and table match design | Asymmetry, wrong source or default route |
| DNS | Configured resolver returns authoritative expected data | Search domain, split DNS or stale answer problem |
| Socket/policy | Exact process listens and policy permits intended source | Application absent or local denial |
| End-to-end | Intended client receives valid protocol response | Local checks hide upstream/return-path failure |
Worked scenario: a host is reachable but the application is not
SSH works, so the team assumes networking is healthy. The application listens only on loopback after a configuration change. Firewall rules and routes are correct, but remote clients receive connection refused. Adding a broad firewall rule cannot create a listener.
The operator uses ss to prove the bind address, checks the service configuration and changes it to the approved private address. A local request to that address and an intended remote client succeed; an unapproved source remains denied. The correction is at the application bind layer.
Practical how-to cases
Case 1: Create a static profile
Assign address, gateway and DNS to one named NetworkManager connection. Keep console access and an independent session because a correct syntax change can still remove remote reachability.
nmcli device status
sudo nmcli con add type ethernet ifname enp1s0 con-name lab-static ipv4.method manual ipv4.addresses 192.0.2.10/24 ipv4.gateway 192.0.2.1 ipv4.dns 192.0.2.53
sudo nmcli con up lab-static
ip -brief address show enp1s0
ip route| Checkpoint | What to establish |
|---|---|
| Expected result | The intended profile owns the interface and persistent address, route and DNS. |
| If it fails | Wrong interface or gateway can end the remote session even when nmcli accepts the profile. |
| Safe recovery | Reactivate the saved NetworkManager profile from console, restore the prior checkpoint, and confirm address, route and DNS separately. |
Case 2: Add a specific route
Route one test prefix through a deliberate next hop and prove kernel selection. Keep console access and an independent session because a correct syntax change can still remove remote reachability.
sudo nmcli con mod lab-static +ipv4.routes '198.51.100.0/24 192.0.2.254'
sudo nmcli con up lab-static
ip route get 198.51.100.10
nmcli -f ipv4.routes con show lab-static| Checkpoint | What to establish |
|---|---|
| Expected result | The route lookup selects the intended interface, source address and gateway and persists in the profile. |
| If it fails | A route can exist yet fail due to neighbor resolution, return path or upstream policy. |
| Safe recovery | Reactivate the saved NetworkManager profile from console, restore the prior checkpoint, and confirm address, route and DNS separately. |
Case 3: Diagnose DNS separately
Compare configured resolvers, query result and raw IP reachability. Keep console access and an independent session because a correct syntax change can still remove remote reachability.
nmcli dev show enp1s0 | grep -E 'IP4.DNS|IP4.DOMAIN'
getent ahosts example.com
dig example.com A
ip route get 1.1.1.1
ping -c 2 192.0.2.53| Checkpoint | What to establish |
|---|---|
| Expected result | Resolver ownership and query response are distinguished from general IP routing. |
| If it fails | Successful ping to an address does not prove DNS; NXDOMAIN differs from timeout and SERVFAIL. |
| Safe recovery | Reactivate the saved NetworkManager profile from console, restore the prior checkpoint, and confirm address, route and DNS separately. |
Case 4: Trace a listening service
Move from socket to local request, firewall and remote path. Keep console access and an independent session because a correct syntax change can still remove remote reachability.
ss -lntp
ip -brief address
curl -v http://127.0.0.1:8080/
firewall-cmd --get-active-zones
tracepath 192.0.2.20
tcpdump -ni enp1s0 tcp port 8080| Checkpoint | What to establish |
|---|---|
| Expected result | The first layer that loses the request or response is demonstrated with evidence. |
| If it fails | No packets, SYN without reply and completed TCP with HTTP error identify different owners. |
| Safe recovery | Reactivate the saved NetworkManager profile from console, restore the prior checkpoint, and confirm address, route and DNS separately. |
Case 5: Configure IPv6 explicitly
Add one documentation-prefix IPv6 address and route to the lab profile and verify source selection. Keep console access and an independent session because a correct syntax change can still remove remote reachability.
sudo nmcli con mod lab-static ipv6.method manual ipv6.addresses 2001:db8:10::10/64 ipv6.gateway 2001:db8:10::1
sudo nmcli con up lab-static
ip -6 address show dev enp1s0
ip -6 route
ip -6 route get 2001:db8:20::20| Checkpoint | What to establish |
|---|---|
| Expected result | The profile owns the IPv6 address and the kernel selects the intended source and next hop. |
| If it fails | IPv6 link-local reachability does not prove global routing, DNS AAAA or return-path policy. |
| Safe recovery | Reactivate the saved NetworkManager profile from console, restore the prior checkpoint, and confirm address, route and DNS separately. |
Independent practice tasks
- Create DHCP and static profiles and switch safely between them.
- Configure two routes with metrics and predict route selection.
- Diagnose duplicate address symptoms from neighbor evidence.
- Capture a TCP handshake and label SYN, SYN-ACK, ACK and teardown.
For this lesson on RHEL NetworkManager and Routing, complete each task without copying the worked command sequence. Record the initial state, exact change, verification, negative test and recovery command. A task is unfinished if it works now but does not survive a reboot where persistence is required.
Troubleshooting by symptom
| Symptom | Inspect first | Defensible next action |
|---|---|---|
| Name fails, IP works | Configured resolver, search domains and DNS response | Correct DNS ownership, not routes |
| No route to host | Route selection, gateway/neighbor and local policy | Repair exact route/link or interpret policy response |
| Connection refused | Destination socket and active rejection | Start/correct application listener |
| Timeout | Capture both sides, firewall/upstream and return path | Locate first drop; do not assume server firewall |
| Change reverts after reboot | Persistent NetworkManager profile and autoconnect priority | Correct owned profile instead of runtime ip state |
Unsafe operations and recovery boundaries
- Unsafe: deleting the active profile or default route over the only SSH path can strand the server. Keep console and timed rollback.
- Unsafe: broad packet capture can collect secrets and personal data. Filter, encrypt, restrict and retain only as authorized.
- Unsafe: forcing speed/duplex without matching peer design can create loss; verify negotiation and both link ends.
Rewritten knowledge checks
Guided lab and acceptance test
- Create a second lab NetworkManager profile with a static private address.
- Capture route selection before and after activation.
- Run a loopback-only test listener and prove remote refusal, then bind to the approved lab address.
- Permit only one lab source through the firewall and test allowed and denied paths.
- Change DNS for the candidate profile and verify resolver ownership and answer.
- Reactivate/reboot, prove persistence, then remove only the lab profile.