RHEL RAID, Device Failure and Recovery

Anil Jalela · Published Sep 3, 2026 · 7 min read

RAID changes availability and performance characteristics; it does not replace backup. A mirrored array can reproduce deletion or corruption immediately, and a rebuild stresses every remaining member. Device replacement begins with exact serial and slot identity, current array metadata, application impact and a tested recovery path. Guessing from /dev/sdX can remove the healthy member.

Separate array redundancy from data recovery

RAID level, member count, spare policy, metadata version and workload determine the failure tolerance. “Degraded” means expected redundancy is absent, not that data is safe indefinitely. Before rebuild, inspect device health, controller/path errors, kernel logs and backups. A path or cable problem can make a healthy device appear absent.

Resynchronization progress alone does not prove filesystem or application integrity. Monitor latency, errors and remaining members, then validate the upper layers after the array returns to the expected state.

Trace a failed I/O from workload to member

Replace one failed lab member deliberately

1. Confirm the array, workload impact, backup and console access.
2. Map every member path to serial, WWN and physical/virtual slot.
3. Capture mdadm --detail, /proc/mdstat and relevant journal before action.
4. Fail and remove only the verified lab member if it is not already absent.
5. Replace the exact device, reproduce the approved partition layout and add the intended member.
6. Monitor rebuild errors, latency and progress; do not reboot merely to speed recovery.
7. Verify expected array state, upper-layer filesystem and application data after completion.

Inspect mdraid without changing it

cat /proc/mdstat
mdadm --detail /dev/md0
mdadm --examine /dev/sdb1
lsblk -o NAME,PATH,TYPE,SIZE,MODEL,SERIAL,WWN,FSTYPE,MOUNTPOINTS
udevadm info --query=property --name=/dev/sdb
journalctl -k --since '-2 hours' --no-pager
findmnt -S /dev/md0
smartctl -x /dev/sdb 2>/dev/null
watch -n 5 cat /proc/mdstat

• Replace sample devices only after independent identity checks.
• mdadm --examine reads member metadata; compare array UUID and role before adding anything.
• SMART is one evidence source and may be unavailable or incomplete behind controllers.
• Retain the first kernel errors because later rebuild activity can obscure the initial failure.

Evidence and acceptance criteria

Worked scenario: the apparent failed disk is a path failure

A member disappears and the array degrades. A technician replaces the disk named by its current device letter, but the missing member was on another slot and the real fault is a controller path. The replacement removes the remaining valid mirror and turns a recoverable incident into restore work.

The controlled process maps md member UUID to WWN and enclosure slot, correlates transport resets in the kernel journal and verifies the device through the platform. The path is repaired, the correct member is reintroduced under the documented procedure and the independent backup remains protected until application acceptance completes.

Practical how-to cases

sudo mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/vdb1 /dev/vdc1
watch -n 2 cat /proc/mdstat
sudo mdadm --detail /dev/md0
sudo mkfs.xfs /dev/md0

sudo mdadm /dev/md0 --fail /dev/vdc1
sudo mdadm /dev/md0 --remove /dev/vdc1
cat /proc/mdstat
sudo mdadm --detail /dev/md0
findmnt -S /dev/md0

sudo sfdisk --dump /dev/vdb > /root/vdb-layout.sfdisk
sudo sfdisk /dev/vdd < /root/vdb-layout.sfdisk
sudo mdadm /dev/md0 --add /dev/vdd1
watch -n 5 cat /proc/mdstat
sudo mdadm --detail /dev/md0

sudo mdadm --examine /dev/vdb1 /dev/vdc1
sudo mdadm --examine --scan
lsblk -o NAME,SIZE,MODEL,SERIAL,FSTYPE,MOUNTPOINTS
journalctl -k -b | grep -i md

Independent practice tasks

1. Build RAID1 and store checksum-verified test data.
2. Replace one failed member and retain before/after detail output.
3. Explain RAID0, RAID1, RAID5, RAID6 and RAID10 failure tolerance.
4. Recover an inactive lab array without using force.

For this lesson on RHEL RAID and Device Recovery, complete each task without copying the worked command sequence. Record the initial state, exact change, verification, negative test and recovery command. A task is unfinished if it works now but does not survive a reboot where persistence is required.

Troubleshooting by symptom

Unsafe operations and recovery boundaries

• Unsafe: mdadm --create on an existing array can overwrite metadata and destroy assembly evidence.
• Unsafe: forced assembly or adding a member with uncertain event history can select stale data.
• Unsafe: removing a device by Linux letter without serial/WWN/slot correlation can remove the healthy member.

Rewritten knowledge checks

Guided lab and acceptance test

1. Create two disposable loop devices and a small RAID1 array.
2. Format and mount it, write checksummed sample data and record array/member UUIDs.
3. Fail and remove one exact loop member; observe degraded state.
4. Add a replacement loop device and monitor synchronization.
5. Verify checksums and filesystem behavior after recovery.
6. Unmount, stop and remove only the lab array and loop mappings.

Primary references

• RHEL 10 managing storage devices
• RHEL 10 documentation
• RHEL 9 documentation