Buffer I/O error on dev dm-5 and error, corrupted directory (invalid entries) after 6.12.3 update

[kennymc.c]

I just installed the 6.12.3 update from 6.12.2 but during the installation process I noticed that my license was downgraded to a trial. When I rebooted this was still the case and when I tried to reinstall the key via the url the log just said Downloading and the Installed but it still showed a trial license. Fortunately, the license was then suddenly active again after I waited a few minutes.

I did everything like described in the update thread and updated all plugins and stopped the array before the update process. I also noticed these strange reoccurring log entries during the installation:

Jul 15 15:31:03 smartserver kernel: buffer_io_error: 6 callbacks suppressed
Jul 15 15:31:03 smartserver kernel: Buffer I/O error on dev dm-5, logical block 15627953920, async page read
Jul 15 15:31:03 smartserver kernel: Buffer I/O error on dev dm-5, logical block 15627953921, async page read
Jul 15 15:31:03 smartserver kernel: Buffer I/O error on dev dm-5, logical block 15627953922, async page read
Jul 15 15:31:03 smartserver kernel: Buffer I/O error on dev dm-5, logical block 15627953923, async page read
Jul 15 15:31:03 smartserver kernel: Buffer I/O error on dev dm-5, logical block 15627953924, async page read
Jul 15 15:31:03 smartserver kernel: Buffer I/O error on dev dm-5, logical block 15627953925, async page read
Jul 15 15:31:03 smartserver kernel: Buffer I/O error on dev dm-5, logical block 15627953926, async page read
Jul 15 15:31:03 smartserver kernel: Buffer I/O error on dev dm-5, logical block 15627953927, async page read
Jul 15 15:31:03 smartserver kernel: Buffer I/O error on dev dm-5, logical block 15627953920, async page read
Jul 15 15:31:03 smartserver kernel: Buffer I/O error on dev dm-5, logical block 15627953921, async page read

 

After the reboot I now get a log of these messages:

kernel: FAT-fs (sda1): error, corrupted directory (invalid entries)

 

Could it be due to my usb flash device not working properly anymore, as I have now also noticed many fsck files and a few with unreadable names on the flash?

 

smartserver-diagnostics-20230715-1606.zip

[kennymc.c]

SMB also doesn't seem to accept my accounts anymore and these log entries appearing:

Jul 15 16:21:50 smartserver smbd[28400]: [2023/07/15 16:21:50.699249,  0] ../../source3/smbd/msdfs.c:170(parse_dfs_path)
Jul 15 16:21:50 smartserver smbd[28400]:   parse_dfs_path: can't parse hostname from path smartserver._smb._tcp.local
Jul 15 16:21:50 smartserver smbd[28400]: [2023/07/15 16:21:50.699309,  0] ../../source3/smbd/msdfs.c:180(parse_dfs_path)
Jul 15 16:21:50 smartserver smbd[28400]:   parse_dfs_path: trying to convert smartserver._smb._tcp.local to a local path
Jul 15 16:21:55 smartserver smbd[28676]: [2023/07/15 16:21:55.411055,  0] ../../source3/smbd/msdfs.c:170(parse_dfs_path)
Jul 15 16:21:55 smartserver smbd[28676]:   parse_dfs_path: can't parse hostname from path smartserver._smb._tcp.local
Jul 15 16:21:55 smartserver smbd[28676]: [2023/07/15 16:21:55.411110,  0] ../../source3/smbd/msdfs.c:180(parse_dfs_path)
Jul 15 16:21:55 smartserver smbd[28676]:   parse_dfs_path: trying to convert smartserver._smb._tcp.local to a local path

 

EDIT: Seems to be the usb drive since I also get a "USB drive is not read-write" notification

Edited July 15, 2023 by kennymc.c

[itimpi]

20 minutes ago, kennymc.c said:

I just installed the 6.12.3 update from 6.12.2 but during the installation process I noticed that my license was downgraded to a trial. When I rebooted this was still the case and when I tried to reinstall the key via the url the log just said Downloading and the Installed but it still showed a trial license. Fortunately, the license was then suddenly active again after I waited a few minutes.

I did everything like described in the update thread and updated all plugins and stopped the array before the update process. I also noticed these strange reoccurring log entries during the installation:

Jul 15 15:31:03 smartserver kernel: buffer_io_error: 6 callbacks suppressed
Jul 15 15:31:03 smartserver kernel: Buffer I/O error on dev dm-5, logical block 15627953920, async page read
Jul 15 15:31:03 smartserver kernel: Buffer I/O error on dev dm-5, logical block 15627953921, async page read
Jul 15 15:31:03 smartserver kernel: Buffer I/O error on dev dm-5, logical block 15627953922, async page read
Jul 15 15:31:03 smartserver kernel: Buffer I/O error on dev dm-5, logical block 15627953923, async page read
Jul 15 15:31:03 smartserver kernel: Buffer I/O error on dev dm-5, logical block 15627953924, async page read
Jul 15 15:31:03 smartserver kernel: Buffer I/O error on dev dm-5, logical block 15627953925, async page read
Jul 15 15:31:03 smartserver kernel: Buffer I/O error on dev dm-5, logical block 15627953926, async page read
Jul 15 15:31:03 smartserver kernel: Buffer I/O error on dev dm-5, logical block 15627953927, async page read
Jul 15 15:31:03 smartserver kernel: Buffer I/O error on dev dm-5, logical block 15627953920, async page read
Jul 15 15:31:03 smartserver kernel: Buffer I/O error on dev dm-5, logical block 15627953921, async page read

 

After the reboot I now get a log of these messages:

kernel: FAT-fs (sda1): error, corrupted directory (invalid entries)

 

Could it be due to my usb flash device not working properly anymore, as I have now also noticed many fsck files and a few with unreadable names on the flash?

 

smartserver-diagnostics-20230715-1606.zip 137.78 kB · 1 download

It definitely could as the .fsck files get created during the boot process if corruption is detected on the flash drive, and the error messages look like problems reading from the flash dtive.

 

I would try rewriting the flash drive with a fresh install keeping just the contents of the ‘config’ folder as sometimes rewriting the flash drive helps.   If that does not help then you probably need to replace the flash drive. 

[kennymc.c]

Should I remove the fsck files from the config folder when rewriting the flash?

[itimpi]

5 minutes ago, kennymc.c said:

Should I remove the fsck files from the config folder when rewriting the flash?

Yes.   Any .fsck files relate to corrupted configuration files

[kennymc.c]

I have tried to flash the usb drive again with the unraid flash creator but the process aborted before the end with a can not open device error.
I then used another drive instead and the flashing process passed without errors. I replaced the contents of the config folder with those from the backup without the fsck and files with cryptic names. Seems to boot at least without errors.
Since I can't use the new stick permanently for Unraid I will have to transfer the license two times. The 2nd time it should work as described in the docs by a mail to [email protected] or is this only done in very specific cases?

 

BTW: Is it expected that an unclean shutdown was detected after replacing the failed usb drive?

Edited July 15, 2023 by kennymc.c

[itimpi]

36 minutes ago, kennymc.c said:

BTW: Is it expected that an unclean shutdown was detected after replacing the failed usb drive?

If the drive had failed then yes as Unraid would have been unable to write back to the flash drive that it had successfully stopped the array.

[kennymc.c]

Since I still had problems with SMB login and my password was not accepted, I set it again via the users settings. That helped. I did the same with all other users on Unraid level. The root login was not affected.

[kennymc.c]

Today I wanted to install the 6.12.4 update. When I stopped the array as recommended before the update, I noticed again the same Buffer I/O error messages in the log with exactly the same block numbers, which I already had before the last update and which then caused the flash drive to set itself to read-only after a reboot. This can't indicate a corrupt USB flash drive again, can it? I can restart the array afterwards without any problems. However, I would like to know what these messages mean exactly, since I use a completely new USB flash drive of the same model, which has also lasted the last few years with various updates.

[kennymc.c]

No one who has an idea why these messages appear even with a new USB flash? I have recently rebooted the system and it worked fine so far. Is it possible that these are leftovers from the corrupt old USB flash, since they also have the same block numbers?

[JorgeB]

Post new diags after it happens again.

[kennymc.c]

It happens as soon as you stop the array and then the callback messages reappear every minute until the array is restarted. 

Diags have been saved after the array has been stopped and restarted around two minutes later.

smartserver-diagnostics-20230926-1942.zip

[JorgeB]

dm-5 is this UD disk, not the flash drive:

 

Sep 20 08:30:27 smartserver kernel: XFS (dm-5): Mounting V5 Filesystem
Sep 20 08:30:27 smartserver kernel: XFS (dm-5): Ending clean mount
Sep 20 08:30:27 smartserver kernel: xfs filesystem being mounted at /mnt/disks/WD_Elements_25A3

 

[kennymc.c]

That is my backup usb hdd which only gets powered on during weekly backups. Strange that this only happens when the array is stopped and when the hdd is not powered on.

 

I assume I should run xfs_repair? Which /dev should I choose? There is no dm-5 in /dev, just dm-1 to 4. I tried it with the corresponding id sdf and sdf1 but this results in

 

bad primary superblock - bad magic number !!!

shortly after starting a dry run.

 

I let it run with sdf for many hours (8 TB hdd) but it ends with

 

Sorry, could not find valid secondary superblock

 

Do i have to add something because the drive is luks encrypted? 

[itimpi]

2 minutes ago, kennymc.c said:

I let it run with sdf for many hours (8 TB hdd) but it ends with

Did you include the partition (i.e. /dev/sdf1)?

[JorgeB]

Since it's encrypted would need to use /dev/mapper/sdX1, recommend using the UD inbuilt filesystem check, easier to use., but those errors don't look like a filesystem issue, could be because of trying to stop the array and close LUKS:

45 minutes ago, kennymc.c said:

when the hdd is not powered on.

 

[kennymc.c]

I think I have solved the problem: I simply stopped the array while the USB hdd was turned on and mounted. When the USB hdd then also got unmounted, I switched it off and restarted the array, waited a while and stopped it again. The messages in the log did not appear anymore. Starting the array worked again without any problems.