tucansam

Yes. Right at 0900 is when I noticed the array was stopped. Everything up to the point of the server time error looked normal. The log covers more than a week and I didn't want to post it all, but I can.

This has now happened again. Array stopped on its own at some point during the night. The GUI worked, however restarting the array had no effect (it tried, the web interface allowed me to click the button to start the array, but after several minutes of waiting, it never happened). Rebooting from the GUI had no effect. I had to 'sudo reboot' from the shell, and now its doing a parity check. This is probably the fourth or fifth time now that this has occured. At the suggestion of you good folks, I have unraid logging to a syslog server, so now I have logs after a reboot. Here's a section that looks pertinent: Feb 1 17:47:14 ffs2 emhttpd: read SMART /dev/sdl Feb 1 17:47:33 ffs2 emhttpd: read SMART /dev/sdp Feb 1 17:47:42 ffs2 emhttpd: read SMART /dev/sdv Feb 1 17:47:49 ffs2 emhttpd: read SMART /dev/sdt Feb 1 17:47:56 ffs2 emhttpd: read SMART /dev/sdx Feb 1 17:48:03 ffs2 emhttpd: read SMART /dev/sdu Feb 1 17:48:15 ffs2 emhttpd: read SMART /dev/sdj Feb 1 17:48:24 ffs2 emhttpd: read SMART /dev/sdb Feb 1 17:48:37 ffs2 emhttpd: read SMART /dev/sdc Feb 1 17:49:48 ffs2 emhttpd: read SMART /dev/sdw Feb 1 18:37:55 ffs2 ntpd[1686]: kernel reports TIME_ERROR: 0x41: Clock Unsynchronized Feb 1 18:37:55 ffs2 rsyslogd: action 'action-2-builtin:omfwd' resumed (module 'builtin:omfwd') [v8.2002.0 try https://www.rsyslog.com/e/2359 ] Feb 1 20:33:05 ffs2 emhttpd: spinning down /dev/sdr Feb 1 20:33:05 ffs2 emhttpd: spinning down /dev/sdq Feb 1 20:33:12 ffs2 emhttpd: spinning down /dev/sdm Feb 1 20:33:12 ffs2 emhttpd: spinning down /dev/sdh Feb 1 20:33:12 ffs2 emhttpd: spinning down /dev/sdj Feb 1 20:33:12 ffs2 emhttpd: spinning down /dev/sdg Feb 1 20:33:12 ffs2 emhttpd: spinning down /dev/sdw Feb 1 20:33:12 ffs2 emhttpd: spinning down /dev/sdd Feb 1 20:33:12 ffs2 emhttpd: spinning down /dev/sdt Feb 1 20:33:12 ffs2 emhttpd: spinning down /dev/sde Feb 1 20:33:12 ffs2 emhttpd: spinning down /dev/sdu Feb 1 20:33:12 ffs2 emhttpd: spinning down /dev/sdb Feb 1 20:33:12 ffs2 emhttpd: spinning down /dev/sdx Feb 1 20:33:12 ffs2 emhttpd: spinning down /dev/sdf Feb 1 20:33:12 ffs2 emhttpd: spinning down /dev/sdv Feb 1 20:33:12 ffs2 emhttpd: spinning down /dev/sdc Feb 1 20:33:12 ffs2 emhttpd: spinning down /dev/sds Feb 1 20:33:12 ffs2 emhttpd: spinning down /dev/sdn Feb 1 20:33:12 ffs2 emhttpd: spinning down /dev/sdl Feb 1 20:33:12 ffs2 emhttpd: spinning down /dev/sdi Feb 1 20:33:12 ffs2 emhttpd: spinning down /dev/sdp Feb 2 03:00:01 ffs2 CA Backup/Restore: It doesn't appear that the array is running. Exiting CA Backup Feb 2 07:00:01 ffs2 crond[1707]: exit status 3 from user root /usr/local/sbin/mover &> /dev/null Feb 2 09:04:10 ffs2 webGUI: Successful login user root from 192.168.0.105 Feb 2 09:04:28 ffs2 emhttpd: shcmd (874446): /usr/local/sbin/set_ncq sdq 1 Feb 2 09:04:28 ffs2 kernel: mdcmd (37): set md_num_stripes 768 Feb 2 09:04:28 ffs2 kernel: mdcmd (38): set md_queue_limit 80 Feb 2 09:04:28 ffs2 kernel: mdcmd (39): set md_sync_limit 5 Feb 2 09:04:28 ffs2 kernel: mdcmd (40): set md_write_method Feb 2 09:04:28 ffs2 emhttpd: shcmd (874447): echo 128 > /sys/block/sdq/queue/nr_requests Feb 2 09:04:28 ffs2 emhttpd: shcmd (874448): /usr/local/sbin/set_ncq sdp 1 Feb 2 09:04:28 ffs2 emhttpd: shcmd (874449): echo 128 > /sys/block/sdp/queue/nr_requests Feb 2 09:04:28 ffs2 emhttpd: shcmd (874450): /usr/local/sbin/set_ncq sde 1 Feb 2 09:04:28 ffs2 emhttpd: shcmd (874451): echo 128 > /sys/block/sde/queue/nr_requests Feb 2 09:04:28 ffs2 emhttpd: shcmd (874452): /usr/local/sbin/set_ncq sdf 1 Feb 2 09:04:28 ffs2 emhttpd: shcmd (874453): echo 128 > /sys/block/sdf/queue/nr_requests Any ideas?

Not sure what is happening. Don't think its a power issue (as in, not a loss of power). Came home to no shares, array stopped. Uptime for server is 1 day. Looks like /var/log/syslog starts just after 1PM yesterday. Second time this has happened in a few weeks or so. Nothing has changed; I'm working on other projects and unraid has been left to its own devices for months now, aside from docker and plugin updates. Shutdown was unclean, as soon as I entered my key and started the array, it started a parity check. I posted about this before but can't find the thread... Someone posted a way to persist log info after a reboot so I can post it the next time it happens. If that person would be kind enough to explain it again, I'd like to see if this happens a third time..... Thanks.

Yesterday's parity check was running fine when I went to bed. Woke up this morning to a server that was running, but whose array had been shut down. Parity check bubble in the upper right corner of the browser window said it completed with errors, but couldn't give me specifics. I entered my passphrase and started the array, which took a really long time to come online. Now another (automated, I assume due to an unclean shutdown?) parity check has begun. Diags attached. sat_ffs2-diagnostics-20211218-0827.zip

Errors were found but SMART says "passed." I remain confused. SMART report after extended test attached. ffs2-smart-20211019-1731.zip

Up to 454 rear errors now. Running extended test as we speak.

Just finished a parity check over the weekend and now this. Diags attached. ffs2-diagnostics-20211018-1829.zip

OK I'm just doing them in the order that the GUI is popping up the options........

Just did this with Disk 16.... It is not working. Formatting and rebooting had zero effect, although it fixed Disk 15. I have added four more disks, all of which were previously used (its been years) in this server, all of which are now clearing. I will update shortly, I presume I will need to format all of them and then I will have five disks which refuse to add to the array because "Unmountable: Volume not encrypted"

That worked. But now the disk is showing up as added, and not added.....

Do the LSI cards with the four external ports, and no internals, need to be flashed to IT mode also? I picked up a Dell branded LSI and it sees no disks attached... But the old card (that was giving CRC errors) saw them. Is this a firmware issue that is easily fixed?

Wow, I really appreciate that!!! Thanks!

The disk existed in the array long ago, like maybe a year or two ago. Its been sitting on a shelf since. How do I get the disk to add? And will I have this problem with subsequent former disks that I add?

I did format it....... Diags attached. ffs2-diagnostics-20210915-1955.zip

No. I just stopped the array, selected one of the disks from the drop down list of the next empty slot, and started the array.

So I physically added four drives. All were mountable via the new "mount" button (used to use unassigned devices) except one drive, for some reason it wouldn't mount but was recognized. I added one of the drives to the array after stopping it, and then restarted. That's when I got the format error. Formatted it, and now am getting the not encrypted error. The disk is showing up in the list of array members, however it is not mounted. Yet I am getting green balls and assume parity is valid. So I'm not sure what is happening.

All my disks are XFS, encrypted. I added a disk, it belonged to the array a while ago and has been sitting on a shelf after an upgrade. After adding the new disk, I started the array. Unraid said it was unmountable because it wasn't formatted (it was, I mounted it and read files just fine). So I clicked the format box and now it says its unmountable because its not an encrypted volume. But if I click on the disk it clearly says its "XFS encrypted." I've added or upgraded disks on this system dozens of times and have never had this happen.

Dammit. I've got a 650w PS with those 5-way SATA splitters, and cables I've used in other servers with no issue. I suppose its the LSI controller card. I've eliminated everything else, essentially.

OK. I got rid of the expander and all cables. I now have the LSI with four external ports, an 8088 -> 4x SATA cable, and my drives. Just plugged four drives into said cable, and one immediately threw 14000 CRC errors. In addition, two of the four drives connected show up as "Dev 4" and "Dev 6" and show all SMART attributes. The other two are "sds" and "sdt" and show no SMART attributes and will not let me run a SMART test. I'm curious if the CRC error reporting is because unraid has noticed the drives... that is to say, I am hoping the error count does not climb. I'm not going to add them to the array. Should I mount them individually and move some data around to test them, to see if the error counts rise? Is there a way to reset the CRC error count?

I am re-evaluating the setup and am going to try an 8086 -> 4-port SATA cable directly connected to the external port of the LSI controller. The expander, the cable from the expander to the drives, and the cable from the LSI to the expander can all be eliminated as variables.

As in replace each one, one at a time, until I figure out the bad part?

As a matter of fact, I do have an expander. However, the first time this problem occurred was before the 2nd HBA for the expander was installed. Right now nothing is connected to the expander. I have an LSI HBA and all of the array members, except the parity disks, are connected to that HBA. I have a second LSI that will eventually connect to the expander. The expander is reserved for when I get a JBOD completed for expansion.

You mean the cache disk? I've never used one, the system is not configured to use one.

root@ffs2:/mnt# ls /bin/ls: cannot access 'user': Transport endpoint is not connected RecycleBin/ disk11/ disk14/ disk4/ disk7/ disks/ user0/ disk1/ disk12/ disk2/ disk5/ disk8/ remotes/ disk10/ disk13/ disk3/ disk6/ disk9/ user/ root@ffs2:/mnt# ls user /bin/ls: cannot access 'user': Transport endpoint is not connected root@ffs2:/mnt# All shares gone, not accessible at all. 'ls /mnt/user' fails. Happened a few weeks ago too. Reboot fixed it. Never did find out what caused it. Diags attached. My SSD, mounted with unassigned devices, is fully accessible. Starting and stopping the array had no effect, must be a reboot to get it back up. Any ideas? ffs2-diagnostics-20210905-1528.zip

When I play videos on my Roku, the first 3-5 seconds of audio is missing. Videos play, but there is no sound at all. Eventually the sound catches up and then all is well. Has been doing this now for about a week, no change in the server config that I am aware of. When I play videos on the Plex app on my LG TV, audio is not sync'd with video. If I change from "original quality" to the one just under it, ie, "original quality 10.8mbps" to "10mbps" then it works just fine. I only watch on the TV a few times a month, so not sure when it started happening. In the browser on my PC none of these problems exist. No problems of any kind existed a month or so ago.