deano_southafrican

@JorgeB Don't know what happened to my response that accompanied the above diagnostics... Anyway, disk1 is still unmountable after swapping to a brand new SATA cable and putting it onto the other PSU SATA string which is now the same as disk2 and disk3. However, the rebuild for disk2 is still going ahead... Does that mean that I might be able to rebuild disk 1 afterwards? Or will the data be corrupt? I'm very confused... Initially it slowed down to 25MB/s for the rebuild and then ramped back up to full speed.

tower-diagnostics-20251015-1903.zip

RIP, I think Disk1 has failed now after the read errors. So Disk2 is being emulated but obviously now Disk1 data will be lost. I'm wondering now if I should just pull the old drives and put in the new 12TB drive and then pull from backups and resync parity... Updated diagnostics... tower-diagnostics-20251015-0024.zip

Hey @JorgeB, so the preclear of the new 14TB parity drive took forever, and then finally, after nearly 30 hours it managed to do the parity swap. So followed my plan and I'm letting it rebuild disk 2 (the old parity drive is now disk 2 after the swap). But I'm getting read errors on disk 1 and it's causing the rebuild to go at like KB/s and I can hear the drive struggling. At this point I just have a feeling everything is going to fail. I've attached the diagnostics. I can't see anything obvious besides some errors due to swapping my cache drives and not having changed the mapping yet. I'm wondering, can I not just take out the drive that's being rebuilt (disk 2) and put in my new 12TB data drive now already and attempt to rebuild? Also wondering if there's anything else that could be causing slow-downs or issues? Or perhaps it's messed up and I should be pulling from backups? Appreciate any advice here... tower-diagnostics-20251014-2351.zip

Thank you for your help, I greatly appreciate it. The drive is attached to the third (of four) SATA power connectors directly attached to the PSU, no splitters or anything. So a bit strange. I've removed the drive an tried to connect it to my PC in one of those SATA HDD USB docks in an attempt to check the drive info in Crystal Disk Info but the drive won't read at all despite spinning up, other drives do work in this way, I've tested 3 other drives today. So I've moved all my data from my other cache pools onto a single new NVMe drive and removed the PCIE to SATA adapter that had all of my SSD's connected. The HDD's in my array were all connected to the SATA ports directly on the motherboard. I've removed the failed drive and connected in its place the 14TB drive that will be my new Parity drive. I disabled Docker and turned off VMs before all of this maintenance work. Now I'm running pre-clear on the new drive. The array is offline. So obviously the parity drive is now emulating the failed disk which is why I could still see and access my data. So I'm trying to figure out if I can use parity swap to replace the failed disk, then rebuild parity to the new 14TB drive. Then I'm guessing I'll have to remove one HDD, replace it with the new 12TB drive, and rebuild from parity, at which point I can move the data from the other two drives to the new 12TB drive and then remove those? I have backups as mentioned so I'm not overly stressed if it goes badly. But just trying to avoid having to rebuild from backups as I'm bottlenecked by my internet speed. Really appreciate you taking the time to read this and give me advice.

I also don't understand how the parity swap would work for multiple data disks and single parity. Would it be a case of moving the parity drive to the data pool and syncing parity to the newer, larger drive? Would that mean the smaller, old parity drive assumes the roll of replacing the defective drive and then the pool rebuilds parity to the new parity drive?

@JorgeB I have replaced SATA cables multiple times, power is off the same string of SATA power connectors as the other drives (I guess could still be an issue). Would it still be safe to try the parity swap? Short of replacing the motherboard and PSU I don't really know what else to do.

@JorgeB Some of the issues were caused by a bad SATA PCIe expansion card. I'm trying to resolve all of this by upgrading from what I have now to two large data drives instead four small drives. Additionally I'm replacing my three small cache drives with a 2TB NVMe SSD and a 2TB 2.5" SSD. This will leave me with one spare SATA port (my board only has 4 SATA ports) for swapping the drives around etc. I'm open to suggestions. Also, rebuilding the array from scratch could potentially mean an opportunity to switch to ZFS, although that would require an upgrade which I haven't done due to the janky drive situation. tower-diagnostics-20251006-1606.zip

Hey! TLDR: don't know how to go about replacing 4 x 2TB drive array (1 drive failed) with 2 x 12TB drives. I'm in a weird spot and I'm trying to figure out the best way to go about this, please tell me what you'd do or even perhaps what not to do? So I have a 4 drive array, 1 parity drive and 3 data drives, all are 2TB in size. I have cloud backups, and physical backups at home of all my important data and have tested pulling them back down - if it all goes horribly wrong, I'll be alright. One of my data drives (disk2) had issues and failed and has subsequently been automatically (not sure how) ejected from the pool. The SMART data says the drive is fine but I don't really trust it. It still shows up as unmounted in unassigned drives plugin. But that's besides the point. I have ordered 2 12TB drives to replace the drives in my array. Now I need to figure out how to swap them all out. I have I think a few options: 1 - Try remount the failed drive and then swap parity, then data drives. (bad idea) 2 - Swap parity and then data drives and then replace the data from the missing drive (not sure this would work because it might destroy parity?). 3 - Pull all the drives and Rsync all the data back onto the new array (nice idea but slow due to internet bottleneck). 4 - Put one of the new drives into the system and copy across all the data, remove it, replace the drives (will parity be able to sync if the data drive already has data when creating a new array?). 5 - Replace the failed drive with the 2TB USB HDD I have and then replace parity, then rebuild from parity (Can you add a USB drive into the main pool?) 6 - Just buy another 2TB HDD and do it properly (meh) Anything I may have overlooked? I realise some ideas are more stupid than others but they're all options. I haven't had to do this before so any guidance, tips/tricks would be greatly appreciated!

Adding the Organisation Owner permission did the trick. Now it looks like it connects to Atlas but then it just crashes. I can access the front-end while the connection is happening but then the entire container crashes... Any ideas? Unhandled Rejection at: Promise { <rejected> Error [ConnectionTimeoutError]: Connect timeout for 5000ms, GET https://cloud.mongodb.com/api/atlas/v1.0/groups/6532986a69795331fcdc584d/clusters/panko-gpt/fts/indexes/pankogpt/vectordatas? -2 (connected: false, keepalive socket: false, socketHandledRequests: 1, socketHandledResponses: 0) headers: {} at Timeout._onTimeout (/usr/src/backend/node_modules/urllib/lib/urllib.js:988:15) at listOnTimeout (node:internal/timers:573:17) at process.processTimers (node:internal/timers:514:7) { requestId: 9, data: undefined, path: '/api/atlas/v1.0/groups/6532986a69795331fcdc584d/clusters/panko-gpt/fts/indexes/pankogpt/vectordatas?', status: -2, headers: {}, res: { status: -2, statusCode: -2, statusMessage: null, headers: {}, size: 0, aborted: false, rt: 5004, keepAliveSocket: false, data: undefined, requestUrls: [Array], timing: null, remoteAddress: '', remotePort: '', socketHandledRequests: 1, socketHandledResponses: 0 } } } reason: Error [ConnectionTimeoutError]: Connect timeout for 5000ms, GET https://cloud.mongodb.com/api/atlas/v1.0/groups/6532986a69795331fcdc584d/clusters/panko-gpt/fts/indexes/pankogpt/vectordatas? -2 (connected: false, keepalive socket: false, socketHandledRequests: 1, socketHandledResponses: 0) headers: {} at Timeout._onTimeout (/usr/src/backend/node_modules/urllib/lib/urllib.js:988:15) at listOnTimeout (node:internal/timers:573:17) at process.processTimers (node:internal/timers:514:7) { requestId: 9, data: undefined, path: '/api/atlas/v1.0/groups/6532986a69795331fcdc584d/clusters/panko-gpt/fts/indexes/pankogpt/vectordatas?', status: -2, headers: {}, res: { status: -2, statusCode: -2, statusMessage: null, headers: {}, size: 0, aborted: false, rt: 5004, keepAliveSocket: false, data: undefined, requestUrls: [ 'https://cloud.mongodb.com/api/atlas/v1.0/groups/6532986a69795331fcdc584d/clusters/panko-gpt/fts/indexes/pankogpt/vectordatas?' ], timing: null, remoteAddress: '', remotePort: '', socketHandledRequests: 1, socketHandledResponses: 0 } } > tsc && vite build vite v5.3.3 building for production... transforming... ✓ 516 modules transformed. rendering chunks... computing gzip size... dist/index.html 0.45 kB │ gzip: 0.29 kB dist/assets/logo-BBQ4Illo.png 8.74 kB dist/assets/index-DpyhJGoW.css 31.88 kB │ gzip: 6.79 kB dist/assets/index-CaIQrqW5.js 440.05 kB │ gzip: 131.40 kB ✓ built in 2.65s > [email protected] build > tsc Init Atlas Configurator API running on http://localhost:5002 Username found: pankogpt Atlas Index: Using specified cluster: panko-gpt Connecting to MongoDB URL mongodb+srv://pankogpt:[email protected]/?retryWrites=true&w=majority&appName=panko-gpt Using database: pankogpt Server ready! ** Press ANY KEY to close this window **

Hey @dracon! I'm not sure if I've done something wrong but I just can't get it to communicate with the database properly. Network access has been set, all the correct names/tokens. Just won't save any data and I'm getting this in the logs: Init Atlas Configurator API running on http://localhost:5002 MongooseError: Operation `discordconfigs.find()` buffering timed out after 10000ms at Timeout.<anonymous> (/usr/src/backend/node_modules/mongoose/lib/drivers/node-mongodb-native/collection.js:185:23) at listOnTimeout (node:internal/timers:573:17) at process.processTimers (node:internal/timers:514:7) MongooseError: Operation `settings.findOne()` buffering timed out after 10000ms at Timeout.<anonymous> (/usr/src/backend/node_modules/mongoose/lib/drivers/node-mongodb-native/collection.js:185:23) at listOnTimeout (node:internal/timers:573:17) at process.processTimers (node:internal/timers:514:7) MongooseError: Operation `settings.findOne()` buffering timed out after 10000ms at Timeout.<anonymous> (/usr/src/backend/node_modules/mongoose/lib/drivers/node-mongodb-native/collection.js:185:23) at listOnTimeout (node:internal/timers:573:17) at process.processTimers (node:internal/timers:514:7) MongooseError: Operation `settings.findOne()` buffering timed out after 10000ms at Timeout.<anonymous> (/usr/src/backend/node_modules/mongoose/lib/drivers/node-mongodb-native/collection.js:185:23) at listOnTimeout (node:internal/timers:573:17) at process.processTimers (node:internal/timers:514:7) MongooseError: Operation `settings.findOne()` buffering timed out after 10000ms at Timeout.<anonymous> (/usr/src/backend/node_modules/mongoose/lib/drivers/node-mongodb-native/collection.js:185:23) at listOnTimeout (node:internal/timers:573:17) at process.processTimers (node:internal/timers:514:7) MongooseError: Operation `settings.findOne()` buffering timed out after 10000ms at Timeout.<anonymous> (/usr/src/backend/node_modules/mongoose/lib/drivers/node-mongodb-native/collection.js:185:23) at listOnTimeout (node:internal/timers:573:17) at process.processTimers (node:internal/timers:514:7) MongooseError: Operation `discordconfigs.find()` buffering timed out after 10000ms at Timeout.<anonymous> (/usr/src/backend/node_modules/mongoose/lib/drivers/node-mongodb-native/collection.js:185:23) at listOnTimeout (node:internal/timers:573:17) at process.processTimers (node:internal/timers:514:7) MongooseError: Operation `whatsappconfigs.find()` buffering timed out after 10000ms at Timeout.<anonymous> (/usr/src/backend/node_modules/mongoose/lib/drivers/node-mongodb-native/collection.js:185:23) at listOnTimeout (node:internal/timers:573:17) at process.processTimers (node:internal/timers:514:7) MongooseError: Operation `settings.findOne()` buffering timed out after 10000ms at Timeout.<anonymous> (/usr/src/backend/node_modules/mongoose/lib/drivers/node-mongodb-native/collection.js:185:23) at listOnTimeout (node:internal/timers:573:17) at process.processTimers (node:internal/timers:514:7) MongooseError: Operation `settings.findOne()` buffering timed out after 10000ms at Timeout.<anonymous> (/usr/src/backend/node_modules/mongoose/lib/drivers/node-mongodb-native/collection.js:185:23) at listOnTimeout (node:internal/timers:573:17) at process.processTimers (node:internal/timers:514:7) Any ideas?

Hey, if anyone is willing to chip in here, I'm looking for some opinions as to what to do next. I'm pretty sure after a couple unclean powerdowns (we have terrible power and I've had a UPS and an inverter die), I have been having issues with a single disk constantly throwing errors and I haven't been able to complete a parity sync for ages now, one because it's incredibly slow, and two because I was previously concerned about ruining the parity (this has gone out the window now). So basically I was getting this in my logs when parity would run: Apr 11 08:25:04 Tower kernel: mdcmd (37): nocheck cancel Apr 11 08:25:05 Tower sSMTP[15964]: Creating SSL connection to host Apr 11 08:25:06 Tower sSMTP[15964]: SSL connection using TLS_AES_256_GCM_SHA384 Apr 11 08:25:10 Tower sSMTP[15964]: Sent mail for [email protected] (221 2.0.0 closing connection d16-20020adff2d0000000b003418364032asm968558wrp.112 - gsmtp) uid=0 username=root outbytes=775 Apr 11 08:25:26 Tower kernel: ata9.00: failed to read SCR 1 (Emask=0x40) Apr 11 08:25:26 Tower kernel: ata9.01: failed to read SCR 1 (Emask=0x40) Apr 11 08:25:26 Tower kernel: ata9.02: failed to read SCR 1 (Emask=0x40) Apr 11 08:25:26 Tower kernel: ata9.02: exception Emask 0x100 SAct 0x0 SErr 0x0 action 0x6 frozen Apr 11 08:25:26 Tower kernel: ata9.02: failed command: READ DMA Apr 11 08:25:26 Tower kernel: ata9.02: cmd c8/00:00:18:4a:00/00:00:00:00:00/e0 tag 17 dma 131072 in Apr 11 08:25:26 Tower kernel: res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout) Apr 11 08:25:26 Tower kernel: ata9.02: status: { DRDY } Apr 11 08:25:27 Tower kernel: ata9.15: SATA link up 1.5 Gbps (SStatus 113 SControl 310) Apr 11 08:25:27 Tower kernel: ata9.01: limiting SATA link speed to 1.5 Gbps Apr 11 08:25:28 Tower kernel: ata9.00: SATA link up 1.5 Gbps (SStatus 113 SControl 310) Apr 11 08:25:28 Tower kernel: ata9.01: SATA link down (SStatus 0 SControl 310) Apr 11 08:25:28 Tower kernel: ata9.02: hard resetting link Apr 11 08:25:28 Tower kernel: ata9.02: SATA link up 1.5 Gbps (SStatus 113 SControl 310) Apr 11 08:25:28 Tower kernel: ata9.00: configured for UDMA/133 Apr 11 08:25:29 Tower kernel: ata9.02: configured for UDMA/133 Apr 11 08:25:29 Tower kernel: ata9.02: device reported invalid CHS sector 0 Apr 11 08:25:29 Tower kernel: ata9: EH complete Apr 11 08:25:29 Tower kernel: md: recovery thread: exit status: -4 Apr 11 08:25:37 Tower kernel: ata9.00: failed to read SCR 1 (Emask=0x40) Apr 11 08:25:37 Tower kernel: ata9.01: failed to read SCR 1 (Emask=0x40) Apr 11 08:25:37 Tower kernel: ata9.02: failed to read SCR 1 (Emask=0x40) Apr 11 08:25:37 Tower kernel: ata9.02: exception Emask 0x100 SAct 0x0 SErr 0x0 action 0x6 frozen Apr 11 08:25:37 Tower kernel: ata9.02: failed command: READ DMA Apr 11 08:25:37 Tower kernel: ata9.02: cmd c8/00:20:40:5f:6c/00:00:00:00:00/e9 tag 20 dma 16384 in Apr 11 08:25:37 Tower kernel: res 50/00:00:37:5e:6c/00:00:00:00:00/ed Emask 0x4 (timeout) Apr 11 08:25:37 Tower kernel: ata9.02: status: { DRDY } Apr 11 08:25:38 Tower kernel: ata9.15: SATA link up 1.5 Gbps (SStatus 113 SControl 310) Apr 11 08:25:38 Tower kernel: ata9.01: limiting SATA link speed to 1.5 Gbps Apr 11 08:25:38 Tower flash_backup: adding task: /usr/local/emhttp/plugins/dynamix.my.servers/scripts/UpdateFlashBackup update Apr 11 08:25:38 Tower kernel: ata9.00: SATA link up 1.5 Gbps (SStatus 113 SControl 310) Apr 11 08:25:39 Tower kernel: ata9.01: SATA link down (SStatus 0 SControl 310) Apr 11 08:25:39 Tower kernel: ata9.02: hard resetting link Apr 11 08:25:39 Tower kernel: ata9.02: SATA link up 1.5 Gbps (SStatus 113 SControl 310) Apr 11 08:25:39 Tower kernel: ata9.00: configured for UDMA/133 Apr 11 08:25:39 Tower kernel: ata9.02: configured for UDMA/133 Apr 11 08:25:39 Tower kernel: ata9: EH complete This lead me down the rabbit hole. First a few quick SMART tests - all fine. Then extended SMART tests - also fine. Once, I had to run XFS_repair to fix an issue where the drive wasn't mounting. I have swapped the SATA cable with a previously used one and a brand new one, I also swapped to a SATA port on a PCIe-SATA adapter so as to test if it was maybe the port on the motherboard (which I thought would be fine since the other 3 don't have issues. So now I'm pretty convinced there is actually an issue with the drive. All the data (as far as I can tell) seems to be in-tact and I have cloud backups of the entire NAS stored per disk (ie. /mnt/disk1 rsyncs to cloud/disk1 and so on for each of the 3 data disks in my array). When writing to or reading from the drive I don't see any performance issues. EDIT: As I'm posting this it seems the drive is finally dead. It is unmountable and I can't get anything to work. I'm going to have to replace it. So any advice as to how to do that safely is appreciated. So I have a few actual questions. Is there something I haven't considered or may have overlooked? I believe I need to replace the disk. If I do replace the disk, should I let it sync with Parity first and then try to restore the data from the cloud backups or should I prevent parity from running, restore from backups and then sync? Would it be silly to try copy this data straight to the new disk (considering there may well be data corruption issues? Any other ideas/input would be greatly appreciated! I have attached diagnostics just in case but I have just rebooted a few times trying to change cables etc so I'm not sure what you might get from it. Thanks a million, appreciate you for reading through this! tower-diagnostics-20240411-0931.zip

Reddit is going wild but seems far more sensible over here. I have a Pro license and in all honesty I'm unlikely to ever need another license and if I did I'd be willing to purchase a year long license as outlined in the post. We can all be brutally honest and say the amount of value we've gotten out of unRAID for the price has been incredible. That being said, it'd be annoying to have to worry about a subscription model or annual license charge. A lifetime option should exist and if you need to increase the price then so be it. Everyone needs to put down the pitch forks. In future though, please make your announcement before releasing the update for everyone to find and get riled up about.

Just for interest sake, it seems there was a corruption issue with the flash drive. Time will tell but my server gave up and was offline for a bit. Ran Memtest and passed. Tried booting and there was an issue with the bz boot loader or something, it refused to boot. Anyway, I have restored from backup and moved over to a new flash drive and have my server up and running. We'll have to wait to see if everything works but at least for now it seems to be alright.

Thanks @JorgeB, appreciate you taking a look. I'm going to run memtest, I had a hunch a while back either one of my RAM modules is bad or my mobo is dying. Will look into corruption on cache as well. Please point me in the right direction for resources or some way I can learn to look through my diagnostics to find corruption issues. I really just want to make sure my flash drive is fine as I don't want to keep making backups of a bad configuration. I have attached the complete diagnostics. tower-diagnostics-20231202-1021.zip