Docker.img deleted but Service will not start

[adfrost81]

So my docker service started having all sorts of problems and that lead me down the path of deleting the docker.img. Had some issues getting it deleted, but I finally succeeded. Every procedure list I've seen says to simply start the service after deleting it and it will recreate the image from scratch. Mine is not doing that. It will not start the service and will not create the image. Any ideas on what I could be doing wrong here? Seems like a pretty simple procedure.

 

Docker settings:

 

 

The appdata share is set to prefer the cache, so it should be there. I can't tell in the GUI, but the docker directory is empty when I check in the command line.

 

 

I don't know if this is helpful, but here's what the system log shows:

 

 

Edited May 30, 2023 by adfrost81
Added screenshot of system log.

[JorgeB]

Please post the diagnostics.

[adfrost81]

@JorgeB See attached.

trantor-diagnostics-20230530-1821.zip

Edited May 30, 2023 by adfrost81

[JorgeB]

May 29 19:43:36 Trantor kernel: ata1: COMRESET failed (errno=-16)
May 29 19:43:36 Trantor kernel: ata1: limiting SATA link speed to 1.5 Gbps
May 29 19:43:36 Trantor kernel: ata1: hard resetting link
May 29 19:43:41 Trantor kernel: ata1: COMRESET failed (errno=-16)
May 29 19:43:41 Trantor kernel: ata1: reset failed, giving up
May 29 19:43:41 Trantor kernel: ata1.00: disable device
May 29 19:43:41 Trantor kernel: ata1: EH complete

 

Cache1 dropped offline, check/replace cables and post new diags after array start.

[adfrost81]

@JorgeB Alright, I reseated the cable, but I notice this error comes up when I start the array. Would this be related to having added another cache drive (after the issue with Docker started).

 

Seems like the drive is started...

 

 

Docker service still will not start. Attached updated Diagnostics file.

 

 

trantor-diagnostics-20230531-0719.zip

[JorgeB]

It dropped again:

 

May 31 05:13:51 Trantor kernel: ata1: link is slow to respond, please be patient (ready=0)
May 31 05:14:20 Trantor kernel: ata1: COMRESET failed (errno=-16)
May 31 05:14:20 Trantor kernel: ata1: limiting SATA link speed to 3.0 Gbps
May 31 05:14:20 Trantor kernel: ata1: hard resetting link
May 31 05:14:25 Trantor kernel: ata1: COMRESET failed (errno=-16)
May 31 05:14:25 Trantor kernel: ata1: reset failed, giving up
May 31 05:14:25 Trantor kernel: ata1.00: disable device

 

[adfrost81]

Maybe a faulty drive? I don’t see any indications (other than those log entries and that notification) that there is anything wrong with it. 

[JorgeB]

If cables were already replaced try this:

https://forums.unraid.net/topic/134954-warning-crucial-mx500-ssds-world-of-pain-stay-away-from-these/?do=findComment&comment=1255816

 

[adfrost81]

Read through that post and decided to remove the drive from the system to see if I could check/update the firmware on my PC. That was not successful...even with WinBtrfs installed, I could not get the drive to show up in Windows. I put it back in the system, but it wouldn't detect in Unraid either. If it wasn't borked before, it seems like it is now, as it is saying it will overwrite whatever was on there if I re-add it as a cache drive. Before removing it from the system, I tried using the Mover to transfer stuff off, but it was nothing but errors.

 

 

 

I have not transferred anything new to the array in a couple weeks...what is the chance that I will actually lose data? Wouldn't everything be copied over to the array at this point?

 

If I try to Mount the drive as an unassigned device, I get errors.

 

trantor-diagnostics-20230531-2132.zip

Edited June 1, 2023 by adfrost81

[JorgeB]

Post the output of

btrfs fi show

 

[adfrost81]

When I run that command in the terminal, it does nothing.

[JorgeB]

Not a good sign but reboot and post new diags after array start.

 

 

[adfrost81]

Alright, but that will wipe the drive according to the UI, so if there was any data on it, it’ll be gone for sure…

[JorgeB]

There's shouldn't be a "data will be deleted" warning after a reboot, but if there is unassign the device before array start.

[adfrost81]

After reboot, it's still showing the drive will be overwritten if I start the array. However, the command worked this time. Here are the results:

 

root@Trantor:~# btrfs fi show
warning, device 2 is missing
Label: none  uuid: a8311295-b025-4362-b378-ea013fc3a3df
        Total devices 2 FS bytes used 270.68GiB
        devid    1 size 465.76GiB used 296.79GiB path /dev/sdb1
        *** Some devices missing

 

[JorgeB]

Assuming the other pool device is still sdc (adjust the command if it isn't) post the output of:

btrfs-select-super -s 1 /dev/sdc1

then again

btrfs fi show

 

[adfrost81]

Yes, I had added a 1TB SSD that I had unplugged for troubleshooting. That is now plugged in and here is the result:

 

root@Trantor:~# btrfs fi show
Label: none  uuid: a8311295-b025-4362-b378-ea013fc3a3df
        Total devices 2 FS bytes used 270.68GiB
        devid    1 size 465.76GiB used 296.79GiB path /dev/sdb1
        devid    2 size 931.51GiB used 17.03GiB path /dev/sdc1

root@Trantor:~# btrfs-select-super -s 1 /dev/sdb1
using SB copy 1, bytenr 67108864

 

I have not started the array, as it is is showing both drives will be overwritten.

 

 

Not a big deal for the 1TB, since it is empty. It does look like data is still on the original 500GB cache (sdb1), but It seems inaccessible at this point. But was that data also transferred to the array? I'm not clear on how cache works...does it store a copy, or the ONLY copy until it is transferred?

Edited June 3, 2023 by adfrost81

[itimpi]

14 minutes ago, adfrost81 said:

or the ONLY copy until it is transferred?

This

[adfrost81]

1 hour ago, itimpi said:

This

 

I'm really confused on how there is any data on that drive then. The mover is scheduled to run daily, and I had not transferred anything new onto the server even this week, so it should've been empty. Unless the Mover wasn't working or something. I'm just wary of what could be on that drive. It's probably mostly replaceable media, but...

[itimpi]

1 hour ago, adfrost81 said:

 

I'm really confused on how there is any data on that drive then. The mover is scheduled to run daily, and I had not transferred anything new onto the server even this week, so it should've been empty. Unless the Mover wasn't working or something. I'm just wary of what could be on that drive. It's probably mostly replaceable media, but...

It depends on the Share settings whether items get moved to the array.   This only happens for shares with Use Cache=Yes set.  Shares such as ‘appdata’ and ‘system’ are typically set with Use Cache=Prefer which keeps them on the cache pool.   Shares to Use Cache=Only or Use Cache=No are ignored for moving  files after initial placement.

[adfrost81]

6 hours ago, itimpi said:

It depends on the Share settings whether items get moved to the array.   This only happens for shares with Use Cache=Yes set.  Shares such as ‘appdata’ and ‘system’ are typically set with Use Cache=Prefer which keeps them on the cache pool.   Shares to Use Cache=Only or Use Cache=No are ignored for moving  files after initial placement.

 

IIRC, appdata and system were set to Use Cache=prefer setting. I don't think I messed with the defaults. I guess all I can do at this point is start the array, wipe the drives, and hope I didn't lose too much.

[JorgeB]

Unassign both pool devices, start array, stop array, re-assign both pool devices, start array, post new diags.

[adfrost81]

On 6/4/2023 at 5:14 AM, JorgeB said:

Unassign both pool devices, start array, stop array, re-assign both pool devices, start array, post new diags.

 

After a reboot, I was able to get the btrfs-select-super -s 1 /dev/sdb1 command ran, and was then able to mount the drive as an unassigned device. Then I was able to run a scrub. Here are the results:

 

FS: btrfs

Executing file system scrub: /sbin/btrfs scrub start -B -R -d -r /dev/sdb1 2>&1

Scrub device /dev/sdb1 (id 1) canceled
Scrub started: Mon Jun 5 17:58:41 2023
Status: aborted
Duration: 0:01:38
data_extents_scrubbed: 0
tree_extents_scrubbed: 125
data_bytes_scrubbed: 0
tree_bytes_scrubbed: 2048000
read_errors: 1
csum_errors: 0
verify_errors: 0
no_csum: 0
csum_discards: 0
super_errors: 0
malloc_errors: 0
uncorrectable_errors: 0
unverified_errors: 0
corrected_errors: 0
last_physical: 5242880

File system corruption detected!

 

I was able to access the share via the console:

 

root@Trantor:~# cd /mnt/disks/1529101AA26A
root@Trantor:/mnt/disks/1529101AA26A# ls -l
/bin/ls: cannot access 'video': Input/output error
total 0
drwxrwxrwx 1 nobody users 118 Feb  1  2022 appdata/
drwxrwxrwx 1 nobody users   0 Jan 20  2022 domains/
drwxrwxrwx 1 nobody users  26 Jan 20  2022 system/
d????????? ? ?      ?       ?            ? video/
root@Trantor:/mnt/disks/1529101AA26A# 

 

I started the array and checked the video folder on the array...everything appears to be present that was last transferred, so at this point, I'm fine with abandoning this drive. I have a couple new drives I've purchased and will use those for cache going forward. I'm fine with recreating the appdata from scratch.

 

Thanks for all the help @JorgeB @itimpi