root@hostname:~# dmesg | tail -n 200
[ 223.850430] XFS (dm-12): Unmounting Filesystem
[ 247.989713] XFS (dm-12): Mounting V5 Filesystem
[ 248.014825] XFS (dm-12): Ending clean mount
[ 248.021340] xfs filesystem being mounted at /pg_data supports timestamps until 2038-01-19 (0x7fffffff)
[ 263.061620] XFS (dm-12): Metadata corruption detected at xfs_inode_buf_verify+0x15d/0x170 [xfs], xfs_inode block 0x9c383240 xfs_inode_buf_verify
[ 263.062166] XFS (dm-12): Unmount and run xfs_repair
[ 263.062439] XFS (dm-12): First 128 bytes of corrupted metadata buffer:
[ 263.062686] 00000000: e5 11 00 00 88 f5 89 06 00 00 05 00 28 02 c8 03 ............(...
[ 263.062951] 00000010: 00 20 04 20 00 00 00 00 c8 9f 62 00 90 9f 62 00 . . ......b...b.
[ 263.063191] 00000020: 58 9f 62 00 20 9f 62 00 e8 9e 62 00 b0 9e 62 00 X.b. .b...b...b.
[ 263.063456] 00000030: 78 9e 62 00 40 9e 62 00 08 9e 62 00 d0 9d 62 00 [email protected].
[ 263.063805] 00000040: 98 9d 62 00 60 9d 62 00 28 9d 62 00 f0 9c 62 00 ..b.`.b.(.b...b.
[ 263.064105] 00000050: b8 9c 62 00 80 9c 62 00 48 9c 62 00 10 9c 62 00 ..b...b.H.b...b.
[ 263.064385] 00000060: d8 9b 62 00 a0 9b 62 00 68 9b 62 00 30 9b 62 00 ..b...b.h.b.0.b.
[ 263.064690] 00000070: f8 9a 62 00 c0 9a 62 00 88 9a 62 00 50 9a 62 00 ..b...b...b.P.b.
[ 263.064978] XFS (dm-12): metadata I/O error in "xfs_imap_to_bp+0x4f/0x70 [xfs]" at daddr 0x9c383240 len 32 error 117
[ 263.066864] XFS (dm-12): Metadata corruption detected at xfs_inode_buf_verify+0x15d/0x170 [xfs], xfs_inode block 0xbb7bbf40 xfs_inode_buf_verify
[ 263.067526] XFS (dm-12): Unmount and run xfs_repair
[ 263.067840] XFS (dm-12): First 128 bytes of corrupted metadata buffer:
[ 263.068179] 00000000: e4 11 00 00 00 ad e4 dc 00 00 00 00 98 00 00 01 ................
[ 263.068491] 00000010: 00 20 04 20 00 00 00 00 08 9f e4 01 10 9e e4 01 . . ............
[ 263.068815] 00000020: 18 9d e4 01 20 9c e4 01 28 9b e4 01 30 9a e4 01 .... ...(...0...
[ 263.069209] 00000030: 38 99 e4 01 40 98 e4 01 48 97 e4 01 50 96 e4 01 [email protected]...
[ 263.069501] 00000040: 58 95 e4 01 60 94 e4 01 68 93 e4 01 70 92 e4 01 X...`...h...p...
[ 263.069865] 00000050: 78 91 e4 01 80 90 e4 01 88 8f e4 01 90 8e e4 01 x...............
[ 263.070199] 00000060: 98 8d e4 01 a0 8c e4 01 a8 8b e4 01 b0 8a e4 01 ................
[ 263.070552] 00000070: b8 89 e4 01 c0 88 e4 01 c8 87 e4 01 d0 86 e4 01 ................
[ 263.070868] XFS (dm-12): metadata I/O error in "xfs_imap_to_bp+0x4f/0x70 [xfs]" at daddr 0xbb7bbf40 len 32 error 117
[ 263.073850] XFS (dm-12): Metadata corruption detected at xfs_inode_buf_verify+0x15d/0x170 [xfs], xfs_inode block 0x9c383240 xfs_inode_buf_verify
[ 263.074615] XFS (dm-12): Unmount and run xfs_repair
[ 263.074954] XFS (dm-12): First 128 bytes of corrupted metadata buffer:
[ 263.075312] 00000000: e5 11 00 00 88 f5 89 06 00 00 05 00 28 02 c8 03 ............(...
[ 263.075661] 00000010: 00 20 04 20 00 00 00 00 c8 9f 62 00 90 9f 62 00 . . ......b...b.
[ 263.076014] 00000020: 58 9f 62 00 20 9f 62 00 e8 9e 62 00 b0 9e 62 00 X.b. .b...b...b.
[ 263.076398] 00000030: 78 9e 62 00 40 9e 62 00 08 9e 62 00 d0 9d 62 00 [email protected].
[ 263.076773] 00000040: 98 9d 62 00 60 9d 62 00 28 9d 62 00 f0 9c 62 00 ..b.`.b.(.b...b.
[ 263.077116] 00000050: b8 9c 62 00 80 9c 62 00 48 9c 62 00 10 9c 62 00 ..b...b.H.b...b.
[ 263.077564] 00000060: d8 9b 62 00 a0 9b 62 00 68 9b 62 00 30 9b 62 00 ..b...b.h.b.0.b.
[ 263.077918] 00000070: f8 9a 62 00 c0 9a 62 00 88 9a 62 00 50 9a 62 00 ..b...b...b.P.b.
[ 263.078278] XFS (dm-12): metadata I/O error in "xfs_imap_to_bp+0x4f/0x70 [xfs]" at daddr 0x9c383240 len 32 error 117
[ 267.928729] XFS (dm-12): Metadata corruption detected at xfs_inode_buf_verify+0x15d/0x170 [xfs], xfs_inode block 0x9c383240 xfs_inode_buf_verify
[ 267.929699] XFS (dm-12): Unmount and run xfs_repair
[ 267.930128] XFS (dm-12): First 128 bytes of corrupted metadata buffer:
[ 267.930498] 00000000: e5 11 00 00 88 f5 89 06 00 00 05 00 28 02 c8 03 ............(...
[ 267.930880] 00000010: 00 20 04 20 00 00 00 00 c8 9f 62 00 90 9f 62 00 . . ......b...b.
[ 267.931288] 00000020: 58 9f 62 00 20 9f 62 00 e8 9e 62 00 b0 9e 62 00 X.b. .b...b...b.
[ 267.931651] 00000030: 78 9e 62 00 40 9e 62 00 08 9e 62 00 d0 9d 62 00 [email protected].
[ 267.932058] 00000040: 98 9d 62 00 60 9d 62 00 28 9d 62 00 f0 9c 62 00 ..b.`.b.(.b...b.
[ 267.932398] 00000050: b8 9c 62 00 80 9c 62 00 48 9c 62 00 10 9c 62 00 ..b...b.H.b...b.
[ 267.932727] 00000060: d8 9b 62 00 a0 9b 62 00 68 9b 62 00 30 9b 62 00 ..b...b.h.b.0.b.
[ 267.933051] 00000070: f8 9a 62 00 c0 9a 62 00 88 9a 62 00 50 9a 62 00 ..b...b...b.P.b.
[ 267.933419] XFS (dm-12): metadata I/O error in "xfs_imap_to_bp+0x4f/0x70 [xfs]" at daddr 0x9c383240 len 32 error 117
[ 267.935092] XFS (dm-12): Metadata corruption detected at xfs_inode_buf_verify+0x15d/0x170 [xfs], xfs_inode block 0xbb7bbf40 xfs_inode_buf_verify
[ 267.935855] XFS (dm-12): Unmount and run xfs_repair
[ 267.936222] XFS (dm-12): First 128 bytes of corrupted metadata buffer:
[ 267.936591] 00000000: e4 11 00 00 00 ad e4 dc 00 00 00 00 98 00 00 01 ................
[ 267.936946] 00000010: 00 20 04 20 00 00 00 00 08 9f e4 01 10 9e e4 01 . . ............
[ 267.937405] 00000020: 18 9d e4 01 20 9c e4 01 28 9b e4 01 30 9a e4 01 .... ...(...0...
[ 267.937863] 00000030: 38 99 e4 01 40 98 e4 01 48 97 e4 01 50 96 e4 01 [email protected]...
[ 267.938294] 00000040: 58 95 e4 01 60 94 e4 01 68 93 e4 01 70 92 e4 01 X...`...h...p...
[ 267.938645] 00000050: 78 91 e4 01 80 90 e4 01 88 8f e4 01 90 8e e4 01 x...............
[ 267.938986] 00000060: 98 8d e4 01 a0 8c e4 01 a8 8b e4 01 b0 8a e4 01 ................
[ 267.939323] 00000070: b8 89 e4 01 c0 88 e4 01 c8 87 e4 01 d0 86 e4 01 ................
[ 267.939701] XFS (dm-12): metadata I/O error in "xfs_imap_to_bp+0x4f/0x70 [xfs]" at daddr 0xbb7bbf40 len 32 error 117
[ 267.941264] XFS (dm-12): Metadata corruption detected at xfs_inode_buf_verify+0x15d/0x170 [xfs], xfs_inode block 0x9c383240 xfs_inode_buf_verify
[ 267.942026] XFS (dm-12): Unmount and run xfs_repair
[ 267.942417] XFS (dm-12): First 128 bytes of corrupted metadata buffer:
[ 267.942832] 00000000: e5 11 00 00 88 f5 89 06 00 00 05 00 28 02 c8 03 ............(...
[ 267.943224] 00000010: 00 20 04 20 00 00 00 00 c8 9f 62 00 90 9f 62 00 . . ......b...b.
[ 267.943636] 00000020: 58 9f 62 00 20 9f 62 00 e8 9e 62 00 b0 9e 62 00 X.b. .b...b...b.
[ 267.944066] 00000030: 78 9e 62 00 40 9e 62 00 08 9e 62 00 d0 9d 62 00 [email protected].
[ 267.944426] 00000040: 98 9d 62 00 60 9d 62 00 28 9d 62 00 f0 9c 62 00 ..b.`.b.(.b...b.
[ 267.944791] 00000050: b8 9c 62 00 80 9c 62 00 48 9c 62 00 10 9c 62 00 ..b...b.H.b...b.
[ 267.945137] 00000060: d8 9b 62 00 a0 9b 62 00 68 9b 62 00 30 9b 62 00 ..b...b.h.b.0.b.
[ 267.945527] 00000070: f8 9a 62 00 c0 9a 62 00 88 9a 62 00 50 9a 62 00 ..b...b...b.P.b.
[ 267.945897] XFS (dm-12): metadata I/O error in "xfs_imap_to_bp+0x4f/0x70 [xfs]" at daddr 0x9c383240 len 32 error 117
[ 273.543750] XFS (dm-12): Metadata corruption detected at xfs_inode_buf_verify+0x15d/0x170 [xfs], xfs_inode block 0x9c383240 xfs_inode_buf_verify
[ 273.544828] XFS (dm-12): Unmount and run xfs_repair
We have an Ubuntu 22.04 server on which there is a Postgres database. During the after done (After patching the kernel) the kernel patch team rebooted the server and the database crashed. This is the second time we have faced this issue, and the database team has not made any database backup, but they have shut down the database services properly.
Now I want to find out why the database got corrupted again and again. Previously I had suggested making a full database backup before the kernel patch and backing it up both times. They are relying on a vnsnapshot (VM snapshot??) only.
Now they are asking me for a root cause analysis.
Please help me to solve this problem. I want to identify why it's getting corrupted again and again. Is it mandatory to back up the database during kernel patch activity, so I can tell them to back up the database next time (at a different time??)?
xfs_repairfrom the live environment. You need to start by repairing the XFS filesystem. THEN, you should schedule maintenance and backup periods. VM snapshots as 'restore 'points before you do updates is generally safe, but they are not long term backup solutions and cause issues when you have tons of snapshots and a chain of them to work with.