Go to a command prompt and enter these commands:
ulimit -Hn 80000
ulimit -Sn 80000
If that lets your system work, then add those commands to your "go" file so the hard and soft file limits will get set whenever you reboot.
You have a lot of these messages in your log:
Oct 17 12:47:01 jlw-unRaid smbd[12066]: synthetic_pathref: opening [downloads/FileBot/jre/legal/jdk.crypto.cryptoki/pkcs11wrapper.md] failed
Oct 17 12:47:01 jlw-unRaid smbd[12066]: [2022/10/17 12:47:01.591558, 0] ../../source3/smbd/files.c:1193(synthetic_pathref)
Stop them by adding this to the smb-extra.conf:
logging = 0
If you are not using any MACs to access the shares, turn off "Enhanced macOS interoperability:" in SMB settings.