Using a unRAID Server durring Parity build

[kcossabo]

I am at another break point on this. I have learned a lot, but, after 4 days of a parity build, the server failed at 7:27am 

1) I have a sys log Server set up, and nothing was logged since 4:38
 

Date	Time	Level	Host Name	Category	Program	Messages
2024-03-31	04:38:50	Notice	SpaceStation	user	SysDrivers	SysDrivers Build Complete
2024-03-31	04:38:38	Notice	SpaceStation	user	network	reload service: nginx
2024-03-31	04:38:23	Info	SpaceStation	daemon	avahi-daemon	Service "SpaceStation" (/services/sftp-ssh.service) successfully established.
2024-03-31	04:38:23	Info	SpaceStation	daemon	avahi-daemon	Service "SpaceStation" (/services/smb.service) successfully established.
2024-03-31	04:38:23	Info	SpaceStation	daemon	avahi-daemon	Service "SpaceStation" (/services/ssh.service) successfully established.
2024-03-31	04:38:22	Notice	SpaceStation	user	root	Starting Nginx server daemon...
2024-03-31	04:38:22	Info	SpaceStation	daemon	avahi-daemon	Server startup complete. Host name is SpaceStation.local. Local service cookie is 262143705.
2024-03-31	04:38:22	Info	SpaceStation	user	emhttpd	shcmd (31): /etc/rc.d/rc.nginx start
2024-03-31	04:38:22	Notice	SpaceStation	user	root	Starting php-fpm done
2024-03-31	04:38:22	Info	SpaceStation	user	emhttpd	shcmd (30): /etc/rc.d/rc.php-fpm start
2024-03-31	04:38:22	Info	SpaceStation	user	emhttpd	Autostart disabled (device configuration change)
2024-03-31	04:38:21	Info	SpaceStation	daemon	avahi-dnsconfd	Successfully connected to Avahi daemon.
2024-03-31	04:38:21	Notice	SpaceStation	user	root	Starting Avahi mDNS/DNS-SD DNS Server Configuration Daemon: /usr/sbin/avahi-dnsconfd -D
2024-03-31	04:38:21	Info	SpaceStation	user	emhttpd	shcmd (25): /etc/rc.d/rc.avahidnsconfd start
2024-03-31	04:38:21	Info	SpaceStation	daemon	avahi-daemon	Registering new address record for 10.10.102.13 on br0.IPv4.
2024-03-31	04:38:21	Info	SpaceStation	daemon	avahi-daemon	Registering new address record for 10.10.10.5 on br2.40.IPv4.
2024-03-31	04:38:21	Info	SpaceStation	daemon	avahi-daemon	Network interface enumeration completed.
2024-03-31	04:38:21	Info	SpaceStation	daemon	avahi-daemon	New relevant interface br0.IPv4 for mDNS.
2024-03-31	04:38:21	Info	SpaceStation	daemon	avahi-daemon	Joining mDNS multicast group on interface br0.IPv4 with address 10.10.102.13.

 

2) hardware platform reported that the OS Failed

 

	
	2024-03-29 01:57:58 (UTC-0400)	System Software event: System Event sensor, OEM System Boot Event was asserted	
	2024-03-29 01:58:11 (UTC-0400)	System Software event: System Event sensor, Timestamp Clock Synch was asserted	
	2024-03-29 01:58:17 (UTC-0400)	System Software event: System Event sensor, Timestamp Clock Synch was asserted	
	2024-03-29 01:59:24 (UTC-0400)	System Software event: OS Boot sensor, boot completed - boot device not specific was asserted	
	2024-03-31 07:26:17 (UTC-0400)	System Software event: OS Stop Or Shutdown sensor, Run-time Critical Stop was asserted	
	2024-03-31 07:35:24 (UTC-0400)	System Software event: System Event sensor, OEM System Boot Event was asserted	
	2024-03-31 07:35:27 (UTC-0400)	System Software event: System Event sensor, OEM System Boot Event was asserted

 

3) now the parity rebuild has restarted
 

Total size:	12 TB	
Elapsed time:	22 minutes	
Current position:	103 GB (0.9 %)	
Estimated speed:	45.1 MB/sec	
Estimated finish:	3 days, 1 hour, 12 minutes

 

Three 12TB Drives, and a 12TB Parity takes 3 days to build?

 

Are you suppose to be able to use the server during the parity build? I know that writing to the drive throws everything off, so I hydrated the 20TB of data before I started the Parity build, but with PLEX and a few arr's running I am about to rebuild to ESXi or Proxmox to see if it is unRAID or the hardware.

QUESTIONs
 

1) Is 3 days to rebuild normal? This is as long or longer than a SHR rebuild on my NAS?
2) Do I need to not use the server for the 3 days?
3) beyond writing syslog to a remote server is there any other forensic data that I can pull on why this is CRASHING, to be clear the server was on, heat was low, the OS CRASHED.  

[Hoopster]

4 hours ago, kcossabo said:

Three 12TB Drives, and a 12TB Parity takes 3 days to build?

No, not normal.  My 8TB parity takes a little over 16 hours to build.  A reasonable rough estimate is about 2 hours per TB.  Your 12TB parity should rebuild in approximately 24 hours if everythng with your server is as it should be.  Clearly, there are some problems.

 

You can use your server during rebuild altough that can slow down the rebuild depending on how heavy the use.  You should minimize the use.  Another option is to use the Parity Check Tuning plugin and spread a parity build over several low-use periods such as at night.  I typically do my rebuild in two roughly 8-hour (midnight to 8am) sessions.

Edited March 31 by Hoopster

[kcossabo]

@Hoopster - Thank You 

Found (at least) one of the issues. One of the two internal SATA Adaptors was set to IDE not AHCI. I changed them both to AHCI and I moved from (about) 50mbps to 150mbs, and now the parity rebuild (no workloads) is 24 hours. I am 12 hours in and seems to be tracking to the 24 hour mark.

 

Problem 1 - Set the SATA controllers to AHCI

[itimpi]

Your times are still long enough that you might want to consider setting the checks to only run in off-peak times by using the Parity Check Tuning plugin.

[Hoopster]

2 hours ago, kcossabo said:

I am 12 hours in and seems to be tracking to the 24 hour mark.

Glad to hear you found the issue and your parity rebuild is progressing normally.  FYI, towards the end it will typically start to slow down quite a bit as it reaches the inner tracks of the drives.  Still, roughly 2 hrs. a TB, perhaps just a a bit more, it typical with larger drives.