GreenGoblin

HI All, I have just been notified that my Cache drive has become read only. Reading the advice it says to post on here with my diagnostics? Any ideas? last time something like this happened I had a faulty drive and so I reduced my RAID 1 cache down to a single drive and now that is causing me issues too. I seem to have a lot of bad luck when it comes to my cache drives tower-diagnostics-20240104-1059.zip

Hi @JorgeB, Thanks for that. I have been away so I'm now coming back to this and I see this: [85499.102333] BTRFS info (device sde1): read error corrected: ino 0 off 6939942928384 (dev /dev/sdb1 sector 277269152) [85499.102408] BTRFS info (device sde1): read error corrected: ino 0 off 6939942932480 (dev /dev/sdb1 sector 277269160) [85499.124169] BTRFS error (device sde1): bad tree block start, want 6939943682048 have 6945702219815404465 [85499.160851] BTRFS error (device sde1): bad tree block start, want 6938820476928 have 3118346403840 [85499.165297] BTRFS error (device sde1): bad tree block start, want 6939949596672 have 16089162085870613913 [85499.232679] BTRFS error (device sde1): bad tree block start, want 6939975647232 have 10018932829234446372 [85499.240721] BTRFS error (device sde1): bad tree block start, want 8565548875776 have 3383689706639153690 [85499.240741] BTRFS: error (device sde1: state A) in btrfs_run_delayed_refs:2157: errno=-5 IO failure [85499.240748] BTRFS info (device sde1: state EA): forced readonly [85499.240760] BTRFS info (device sde1: state EA): balance: ended with status: -5 Any ideas on how I should proceed? I have a spare SSD connected and unassigned which has the data form the cache drives backed up on it. tower-diagnostics-20230916-0207.zip

The old drive is completely dead for what I can tell. I removed it from the case thinking I could just replace it with another 2tb SSD. or is that not the case? as of right now I'm still getting the error for file system not existing. I have been away so not been able to deal with this for a while.

Now I can't seem to replace it as it keeps on saying it is unmountable and the file system doesn't seem to exist. I have tried the first part on here to no avail but I can't get the second part to work. Any help would be greatly appreciated. tower-diagnostics-20230713-0158.zip

So using the upload script, what would I need to change to move files back to my local drives? as Google has kindly informed me I'm over my limits sadly. I have tried messing with it myself but to no avail. I tried changing the local and remote paths around but clearly I'm not doing something correctly. I even asked ChatGPT and tried that but still can't get it to work. #!/bin/bash ###################### ### Upload Script #### ###################### ### Version 0.95.5 ### ###################### ####### EDIT ONLY THESE SETTINGS ####### # INSTRUCTIONS # 1. Edit the settings below to match your setup # 2. NOTE: enter RcloneRemoteName WITHOUT ':' # 3. Optional: Add additional commands or filters # 4. Optional: Use bind mount settings for potential traffic shaping/monitoring # 5. Optional: Use service accounts in your upload remote # 6. Optional: Use backup directory for rclone sync jobs # REQUIRED SETTINGS RcloneCommand="move" # choose your rclone command e.g. move, copy, sync RcloneRemoteName="gdrive_media_vfs" # Name of rclone remote mount WITHOUT ':'. RcloneUploadRemoteName="gdrive_media_vfs" # If you have a second remote created for uploads put it here. Otherwise use the same remote as RcloneRemoteName. LocalFilesShare="/mnt/user/local1" # location of the local files without trailing slash you want to rclone to use RcloneMountShare="/mnt/user/mount_rclone1" # where your rclone mount is located without trailing slash e.g. /mnt/user/mount_rclone MinimumAge="1d" # sync files suffix ms|s|m|h|d|w|M|y ModSort="ascending" # "ascending" oldest files first, "descending" newest files first # Note: Again - remember to NOT use ':' in your remote name above # Bandwidth limits: specify the desired bandwidth in kBytes/s, or use a suffix b|k|M|G. Or 'off' or '0' for unlimited. The script uses --drive-stop-on-upload-limit which stops the script if the 750GB/day limit is achieved, so you no longer have to slow 'trickle' your files all day if you don't want to e.g. could just do an unlimited job overnight. BWLimit1Time="01:00" BWLimit1="0" BWLimit2Time="08:00" BWLimit2="3000" BWLimit3Time="16:00" BWLimit3="3000" # OPTIONAL SETTINGS # Add name to upload job JobName="_daily_upload" # Adds custom string to end of checker file. Useful if you're running multiple jobs against the same remote. # Add extra commands or filters Command1="--exclude downloads/**" Command2="" Command3="" Command4="" Command5="" Command6="" Command7="" Command8="" # Bind the mount to an IP address CreateBindMount="N" # Y/N. Choose whether or not to bind traffic to a network adapter. RCloneMountIP="192.168.11.4" # Choose IP to bind upload to. NetworkAdapter="eth1" # choose your network adapter. eth0 recommended. VirtualIPNumber="1" # creates eth0:x e.g. eth0:1. # Use Service Accounts. Instructions: https://github.com/xyou365/AutoRclone UseServiceAccountUpload="N" # Y/N. Choose whether to use Service Accounts. ServiceAccountDirectory="/mnt/user/appdata/other/rclone/service_accounts" # Path to your Service Account's .json files. ServiceAccountFile="sa_gdrive_upload" # Enter characters before counter in your json files e.g. for sa_gdrive_upload1.json -->sa_gdrive_upload100.json, enter "sa_gdrive_upload". CountServiceAccounts="15" # Integer number of service accounts to use. # Is this a backup job BackupJob="N" # Y/N. Syncs or Copies files from LocalFilesLocation to BackupRemoteLocation, rather than moving from LocalFilesLocation/RcloneRemoteName BackupRemoteLocation="backup" # choose location on mount for deleted sync files BackupRemoteDeletedLocation="backup_deleted" # choose location on mount for deleted sync files BackupRetention="90d" # How long to keep deleted sync files suffix ms|s|m|h|d|w|M|y ####### END SETTINGS ####### ############################################################################### ##### DO NOT EDIT BELOW THIS LINE UNLESS YOU KNOW WHAT YOU ARE DOING ##### ############################################################################### ####### Preparing mount location variables ####### if [[ $BackupJob == 'Y' ]]; then LocalFilesLocation="$LocalFilesShare" echo "$(date "+%d.%m.%Y %T") INFO: *** Backup selected. Files will be copied or synced from ${LocalFilesLocation} for ${RcloneUploadRemoteName} ***" else LocalFilesLocation="$LocalFilesShare/$RcloneRemoteName" echo "$(date "+%d.%m.%Y %T") INFO: *** Rclone move selected. Files will be moved from ${LocalFilesLocation} for ${RcloneUploadRemoteName} ***" fi RcloneMountLocation="$RcloneMountShare/$RcloneRemoteName" # Location of rclone mount ####### create directory for script files ####### mkdir -p /mnt/user/appdata/other/rclone/remotes/$RcloneUploadRemoteName #for script files ####### Check if script already running ########## echo "$(date "+%d.%m.%Y %T") INFO: *** Starting rclone_upload script for ${RcloneUploadRemoteName} ***" if [[ -f "/mnt/user/appdata/other/rclone/remotes/$RcloneUploadRemoteName/upload_running$JobName" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Exiting as script already running." exit else echo "$(date "+%d.%m.%Y %T") INFO: Script not running - proceeding." touch /mnt/user/appdata/other/rclone/remotes/$RcloneUploadRemoteName/upload_running$JobName fi ####### check if rclone installed ########## echo "$(date "+%d.%m.%Y %T") INFO: Checking if rclone installed successfully." if [[ -f "$RcloneMountLocation/mountcheck" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: rclone installed successfully - proceeding with upload." else echo "$(date "+%d.%m.%Y %T") INFO: rclone not installed - will try again later." rm /mnt/user/appdata/other/rclone/remotes/$RcloneUploadRemoteName/upload_running$JobName exit fi ####### Rotating serviceaccount.json file if using Service Accounts ####### if [[ $UseServiceAccountUpload == 'Y' ]]; then cd /mnt/user/appdata/other/rclone/remotes/$RcloneUploadRemoteName/ CounterNumber=$(find -name 'counter*' | cut -c 11,12) CounterCheck="1" if [[ "$CounterNumber" -ge "$CounterCheck" ]];then echo "$(date "+%d.%m.%Y %T") INFO: Counter file found for ${RcloneUploadRemoteName}." else echo "$(date "+%d.%m.%Y %T") INFO: No counter file found for ${RcloneUploadRemoteName}. Creating counter_1." touch /mnt/user/appdata/other/rclone/remotes/$RcloneUploadRemoteName/counter_1 CounterNumber="1" fi ServiceAccount="--drive-service-account-file=$ServiceAccountDirectory/$ServiceAccountFile$CounterNumber.json" echo "$(date "+%d.%m.%Y %T") INFO: Adjusted service_account_file for upload remote ${RcloneUploadRemoteName} to ${ServiceAccountFile}${CounterNumber}.json based on counter ${CounterNumber}." else echo "$(date "+%d.%m.%Y %T") INFO: Uploading using upload remote ${RcloneUploadRemoteName}" ServiceAccount="" fi ####### Upload files ########## # Check bind option if [[ $CreateBindMount == 'Y' ]]; then echo "$(date "+%d.%m.%Y %T") INFO: *** Checking if IP address ${RCloneMountIP} already created for upload to remote ${RcloneUploadRemoteName}" ping -q -c2 $RCloneMountIP > /dev/null # -q quiet, -c number of pings to perform if [ $? -eq 0 ]; then # ping returns exit status 0 if successful echo "$(date "+%d.%m.%Y %T") INFO: *** IP address ${RCloneMountIP} already created for upload to remote ${RcloneUploadRemoteName}" else echo "$(date "+%d.%m.%Y %T") INFO: *** Creating IP address ${RCloneMountIP} for upload to remote ${RcloneUploadRemoteName}" ip addr add $RCloneMountIP/24 dev $NetworkAdapter label $NetworkAdapter:$VirtualIPNumber fi else RCloneMountIP="" fi # Remove --delete-empty-src-dirs if rclone sync or copy if [[ $RcloneCommand == 'move' ]]; then echo "$(date "+%d.%m.%Y %T") INFO: *** Using rclone move - will add --delete-empty-src-dirs to upload." DeleteEmpty="--delete-empty-src-dirs " else echo "$(date "+%d.%m.%Y %T") INFO: *** Not using rclone move - will remove --delete-empty-src-dirs to upload." DeleteEmpty="" fi # Check --backup-directory if [[ $BackupJob == 'Y' ]]; then echo "$(date "+%d.%m.%Y %T") INFO: *** Will backup to ${BackupRemoteLocation} and use ${BackupRemoteDeletedLocation} as --backup-directory with ${BackupRetention} retention for ${RcloneUploadRemoteName}." LocalFilesLocation="$LocalFilesShare" BackupDir="--backup-dir $RcloneUploadRemoteName:$BackupRemoteDeletedLocation" else BackupRemoteLocation="" BackupRemoteDeletedLocation="" BackupRetention="" BackupDir="" fi # process files rclone $RcloneCommand $LocalFilesLocation $RcloneUploadRemoteName:$BackupRemoteLocation $ServiceAccount $BackupDir \ --user-agent="$RcloneUploadRemoteName" \ -vv \ --buffer-size 512M \ --drive-chunk-size 512M \ --tpslimit 8 \ --checkers 8 \ --transfers 4 \ --order-by modtime,$ModSort \ --min-age $MinimumAge \ $Command1 $Command2 $Command3 $Command4 $Command5 $Command6 $Command7 $Command8 \ --exclude *fuse_hidden* \ --exclude *_HIDDEN \ --exclude .recycle** \ --exclude .Recycle.Bin/** \ --exclude *.backup~* \ --exclude *.partial~* \ --drive-stop-on-upload-limit \ --bwlimit "${BWLimit1Time},${BWLimit1} ${BWLimit2Time},${BWLimit2} ${BWLimit3Time},${BWLimit3}" \ --bind=$RCloneMountIP $DeleteEmpty # Delete old files from mount if [[ $BackupJob == 'Y' ]]; then echo "$(date "+%d.%m.%Y %T") INFO: *** Removing files older than ${BackupRetention} from $BackupRemoteLocation for ${RcloneUploadRemoteName}." rclone delete --min-age $BackupRetention $RcloneUploadRemoteName:$BackupRemoteDeletedLocation fi ####### Remove Control Files ########## # update counter and remove other control files if [[ $UseServiceAccountUpload == 'Y' ]]; then if [[ "$CounterNumber" == "$CountServiceAccounts" ]];then rm /mnt/user/appdata/other/rclone/remotes/$RcloneUploadRemoteName/counter_* touch /mnt/user/appdata/other/rclone/remotes/$RcloneUploadRemoteName/counter_1 echo "$(date "+%d.%m.%Y %T") INFO: Final counter used - resetting loop and created counter_1." else rm /mnt/user/appdata/other/rclone/remotes/$RcloneUploadRemoteName/counter_* CounterNumber=$((CounterNumber+1)) touch /mnt/user/appdata/other/rclone/remotes/$RcloneUploadRemoteName/counter_$CounterNumber echo "$(date "+%d.%m.%Y %T") INFO: Created counter_${CounterNumber} for next upload run." fi else echo "$(date "+%d.%m.%Y %T") INFO: Not utilising service accounts." fi # remove dummy file rm /mnt/user/appdata/other/rclone/remotes/$RcloneUploadRemoteName/upload_running$JobName echo "$(date "+%d.%m.%Y %T") INFO: Script complete" exit

Thanks for the info guys I really appreciate it, I had to fix the file system in the end in order to get it to work on disk4. Disk 5 is going back to the seller as its faulty it seems but I'm not using it atm.

I’m getting the following types of errors and I don’t believe based on what I can see that my drive is full. Drive mounted read-only or completely full. disk4 is not defined / installed in the array. This will cause errors when writing to the array. Fix it here: NOTE: Because of how the UI works, disk4 will not appear in on this page. You will need to make a change (any change), then revert the change and hit apply to fix this issue i have tried to changing the configuration back and forth on the share to sort the error above but that doesn’t work. I don’t seem to be able to get dockers to download anything either so there definitely seems to be an issue but everything has been running smoothly until recently. I haven’t made any config changes leading up to these issues. The only thing I can think of is that I had an electrician turn off the circuit the server is on so it tripped to ups power and shut down. please see my attached diagnostics. I’d appreciate any help on this as I’m lost as to what’s the issue. Normally I can figure it out but with this I’m out of my depth. tower-diagnostics-20230218-2100.zip

[NOW RESOLVED] I recently moved hardware (drives remained the same just moved to a new mobo and cpu) and before the move everything ran smoothly and now it keeps on having issues with the mount. Having said that I believe Rclone also received an update around the same time so I'm not sure what could be the issue. Script location: /tmp/user.scripts/tmpScripts/Rclone_mount/script Note that closing this window will abort the execution of this script 27.09.2020 21:05:38 INFO: Creating local folders. mkdir: cannot stat '/mnt/user/mount_rclone/gdrive_media_vfs': Transport endpoint is not connected 27.09.2020 21:05:38 INFO: *** Starting mount of remote gdrive_media_vfs 27.09.2020 21:05:38 INFO: Checking if this script is already running. 27.09.2020 21:05:38 INFO: Script not running - proceeding. 27.09.2020 21:05:38 INFO: *** Checking if online 27.09.2020 21:05:39 PASSED: *** Internet online 27.09.2020 21:05:39 INFO: Mount not running. Will now mount gdrive_media_vfs remote. 27.09.2020 21:05:39 INFO: Recreating mountcheck file for gdrive_media_vfs remote. 2020/09/27 21:05:39 DEBUG : rclone: Version "v1.53.1" starting with parameters ["rcloneorig" "--config" "/boot/config/plugins/rclone/.rclone.conf" "copy" "mountcheck" "gdrive_media_vfs:" "-vv" "--no-traverse"] 2020/09/27 21:05:39 DEBUG : Creating backend with remote "mountcheck" 2020/09/27 21:05:39 DEBUG : Using config file from "/boot/config/plugins/rclone/.rclone.conf" 2020/09/27 21:05:39 DEBUG : fs cache: adding new entry for parent of "mountcheck", "/" 2020/09/27 21:05:39 DEBUG : Creating backend with remote "gdrive_media_vfs:" 2020/09/27 21:05:39 DEBUG : Creating backend with remote "gdrive:crypt" 2020/09/27 21:05:41 DEBUG : mountcheck: Modification times differ by -12h2m28.852390437s: 2020-09-27 21:05:39.806390437 +0100 BST, 2020-09-27 08:03:10.954 +0000 UTC 2020/09/27 21:05:43 INFO : mountcheck: Copied (replaced existing) 2020/09/27 21:05:43 INFO : Transferred: 32 / 32 Bytes, 100%, 13 Bytes/s, ETA 0s Transferred: 1 / 1, 100% Elapsed time: 3.9s 2020/09/27 21:05:43 DEBUG : 4 go routines active 27.09.2020 21:05:43 INFO: *** Creating mount for remote gdrive_media_vfs 27.09.2020 21:05:43 INFO: sleeping for 5 seconds 2020/09/27 21:05:44 Fatal error: Can not open: /mnt/user/mount_rclone/gdrive_media_vfs: open /mnt/user/mount_rclone/gdrive_media_vfs: transport endpoint is not connected 27.09.2020 21:05:48 INFO: continuing... 27.09.2020 21:05:48 CRITICAL: gdrive_media_vfs mount failed - please check for problems. Stopping dockers If I reboot the server then they will mount for a short while before then getting terminated and not mounting again until I restart it. Any help would be greatly appreciated in trying to figure out what has gone wrong or that I have set incorrectly. Not sure what I had done wrong but now it seems to work again after being down for a couple of days. tower-diagnostics-20200927-2104.zip