NAS

It is important that users who choose a non subscription model, even if that is just implicit by the fact they use only the traditional unRAID product, that there be no phone home or other services that reach out of the system by lieu of the subscription services running in "off" mode of or any other mechanism. I cannot stress this enough. Feel free to add value in whatever way suits your business but dont break that trust model whilst doing so.

Whilst it is not ideal that the poster did not follow normal security reporting etiquette it is clear there is an issue and it is off our own making. See versus http://www.slackware.com/security/list.php?l=slackware-security&y=2020 tl;dr we are long overdue an update but we have slipped into the old habit of waiting for the development branch to be ready and ignoring the stable branch. It is not the end of the world but its a habit we need to break again ASAP

@unRate can you post a few representative examples to set context. Nothing should be `fixed over a year ago` but 280-290 days is unfortunately possible.

This is a very interesting poll and I commend the people responsible. I do however question why `SSD Array option with Trim support` is on this poll though. The other items in the poll are feature enhancements, nice to haves or power user edge cases..... but supporting SSDs in 2020 should be a basic capability for a NAS not something we poll to see if its optionally wanted.

This is a pretty big deal if it is true. I wouldnt know how to pull this off right now though so more details required.

I think I can actually replicate this now. If i mount a USB drive and copy files continuously to my SSD cache drive which also is the location of my docker loopback image then after a few minutes docker starts to not respond which obviously ruins the web GUI as well. I routinely copied files in this way in all previous versions and the cache drive seems fine (its pretty new) and there are no errors in any log that I can see. The SSD is attached to a motherboard SATA port directly. I am pretty sure it is IO WAIT as load sky rockets. Will wait and see if its an "only me" problem.

Except if you a company or any public body or a registered non profit or anyone with liability cover or any organization that has to comply is ISO accreditation or worldwide equivs or has independent audits or anyone covered by EU GDPR or or.... context is import and whilst most people here are home users plenty unRAID users are not Nice addon. I really mean that. It absolutely has its place but the default advice should always be "be secure unless you really really understand the risks of not being".

I faced an odd issue where the web gui would load but not completely. Specifically the dash wouldnt load but disk view would excluding the action buttons at the bottom. Docker view wouldnt load at all but settings would. Dockers containers did not work or were super slow (hard to say) Manually restarting one container instantly kicked web gui and docker into a working state. I cant replicate.

tl;dr to fully fix this Intel garbage users are going to pay a performance price. The price varies wildly based on user workloads and is basically impossible to predict however I have been in conversation where some devops have seen insane edge cases performance drops. I would suggest the right way to do this is to fix it by default but document an opt out for those that want to accept the risk because it is not possible for normal humans to really understand this beginning to end.

So one of the problems with cache_dirs is that it is at its core a hack. Quite a bit of the user setup is based on educated guesswork and very little info is presented or logged to know how well it is functioning e.g. how many files are being cached, how often is it waking up disks. Most of the performance feedback we can gather is based on timings. Some ideas: track timings and present them as a time based graph for easier analysis store timings outside of rotating logs for easier long term analyisis estimate memory usage In theory we can can reasonably estimate maximum memory requirements by counting potential inode entrys and multiplying it by the inode size . This would allow what/if scenarios to be built where each share can be presented in terms of estimated memory requirements and if we do end up adding a depth per share option the user can tune their settings (and potentially reorganize their data) to cache as efficiently as possible. This will take some reading and experiemtation and unless someone is capable of reading the kernel code to understand how cache pressure impacts this it will always be worst case..... but to my eye this is the direction to go

Sorry to be so long replying. I wanted to wait until I was sure my assumptions fixed the issue before I posted again and time got away from me. tl;dr my fundamental issue was at some point the addon update dropped the includes but not the excludes so it appeared to be working but actually was doing nothing. Changing the settings mostly fixed the issue. I am keen to throw some other ideas into the pot but the most important matter at hand is to deal with the PRs. Forking and getting users to manually pull addons from the internet is sub optimal. There has been some discussion on this above, where are we today with it.

I do see the point you are making but statistically it does not matter, any sufficiently large group when asked "what they want" will tend towards 100% coverage of any set of options. This if charted will obviously be a histogram but a forum is a very blunt tool for this kind of requirement gathering. You will have much better results if you either: ask users for their use cases and do the requirements analysis in reserve from there or estimate the base requirements internally and offer a trial working set of features and follow on by capturing the use cases people aren't able to meet with the trial, deciding form there if they are cases you want to support. The second option is what I would do as it allows you to internally debate the 10% effort 90% gain tipping point.

Is there a way to add basic support as a trial? I dont know the effort on this but I do know that asking people on the internet what they "need" is generally a bad idea, much like trying to herd cats with an error margin greater than the sample set, you will just tend towards "everything" the more people you ask.

Something weird is going on. I can see cache dirs running and even though I now have /bin/bash /usr/local/emhttp/plugins/dynamix.cache.dirs/scripts/cache_dirs -e appdata -l off -p 0 -D 9999 the cache seems to be perpetually needing disks spun up. How would you like me to debug this further? Update: it is this obvious, have all my cache includes been removed so I end with exclusions only and no inclusions or is there there an implied * include?

Having a hard time learning to love the new adaptive feature. The idea sounds great in practice but I have found that effectively cache dirs doesn't work any more, with all lower level folder access waking up some disks. Certainly this will be user specific but without feedback on how the adaptive caching is working, ideally showing depth per top level folder, timings etc it seems to me people would be better served with a default manual setup. On this note I don't know how other people do this but my ideal use case is to be able to set a global minimum depth and exceptions per folder. Currently I exclude 30% of my array as the data is rarely accessed but very large numbers of files.... however whilst this helps cache dirs a single stray click on one of these top level folders busts the cache and disks spin up. I would love to get into a discussion on how we could enhance this feature as without a word of a lie, it is the most critical one I operate.

Coincidentally I was thinking about this the other day along with S3 and it seems, to me at least, that whilst everyone will make a case for features they want, adding protocol support is the one sure fire way to attract new users. I cannot quantify how many users this would be but we can say that every user in the world that wants iSCSI or S3 doesnt buy unRAID. Seems like an "easy" win to me.

I cant see anything but I wanted to check to make sure since this is such a long thread. I am in the process of migrating to XFS and I have found that one RFS disk that will mount happily with unraid wont mount at all with UD. dmesg shows [Tue Jun 19 15:40:36 2018] REISERFS warning (device sdc1): sh-2021 reiserfs_fill_super: can not find reiserfs on sdc1 so before I get into this are there any know relevant issues? Update: well it turns out its pretty obvious why UD cant mount this reiser disk.... thats because it is xfs. # file -sL /dev/sdc* /dev/sdc: DOS/MBR boot sector; partition 1 : ID=0xee, start-CHS (0x0,0,2), end-CHS (0x3ff,255,63), startsector 1, 4294967295 sectors, extended partition table (last) /dev/sdc1: SGI XFS filesystem data (blksz 4096, inosz 512, v2 dirs) any ideas why UD is trying to mount a XFS disks as ResierFS?

LT are wrong then But seriously I think all we need here is normal nix separation of error states. The reason I bring this up is that after manually tracking down where cache space had gone i noticed that i have had mover errors for weeks so this is based on real world issues. Also it is not my experience here that it only runs once but PECAK is entirely feasible. Will pay more attention and report back.

Some quick feedback. I am new to this plugin. First off my compliments. It is super useful and very powerful Couple of enhancements for consideration: we suggest turning off mover logging which makes sense except that it also removes mover error logging.. we should not be hiding errors. not sure what the best fix would be here. when you enter the settings dialogue it actually kicks off a scan. I would suggest this is a break with the GUI design model and that entering settings should be limited to just that. Activating a scan should be a positive user action especially since it can take tens of minutes to complete. Kudos

Polite bump for (it wont let me quote a quote... silly forum :): tl;dr Even when 100% successfully complete, a preclear run done via the unassigned devices addon only ever has the "cancel" option. Expected behavior would be that cancel state is removed when complete and drive status image changes to pre-cleared.

Agree. This would be ideal but it is disingenuous of me to assume that these wasnt a good reason that this scheme was used in the first place and the same complications still exist today. I think this is the one change that needs to happen soon. My focus on this topic is driven by a strong potential of user data loss and whilst there is an inelegance to users having two types of almost identical disks that are not fully compatible that is an issue that can be addressed in the longer term (much as it irks me about all the time I have wasted moving data to standardize on XFS disks only to now find they are still non standard) I agree here as well and this wording puts a finger on one point I was struggling to make. Also it is important to accept that there is a real difference between the mindset of a user bringing a disk from elsewhere (importing) and one plugging in a disk created on unRAID (native). The user that has stayed native during the whole story may not even think to consider that there may be issues and that is where things can go badly wrong.

@Squid to be fair I specifically and intentionally never said it was a bug in UD for three reasons, 1. I didn't know enough to comment, 2. to the user it makes no difference anyway and 3. I really didn't want to start pointing as that only deflects from the issue.... but that plan kinda back fired lol Anyways based on what I know so far I have to agree with the general sentiment that this must change and soon. It is without question a requirement that there should be absolute compatibility between UD formatted disks and unRAID and if a new common ground needs agreed that better fits with norms then it needs to happen soon. But I stand by my assertation that this is a bug, even if it is intentional and the code is sound; we simply cant have a situation where users can make a simple assumption/mistake and lose a disk of data with one button click. Whatever you need from me you have it because this needs to go away before someone gets hurt.

The bug is that there are scenarios where disks with data could end up being erased. These might be rare but data safety trumps everything. I have not looked into it more since I had a backup to use to recover but I have here (still I think) a disk that was full of data and has ended up unmountable by unRAID and UD. It was (created using UD, mounted using UD, rsync using unRAID, umount using UD and then added to array but not mounted as the format warning stopped me. Also as far as I know unRAID cannot format a disk outside of the array? and if you add it to the array you can no longer remove it without a parity outage. The warning is a good step and I appreciate it as it will no doubt save others. Perhaps someone can explain or point me at more detail as to what is so special about unRAIDs' XFS format options that we cant replicate it? I would rather expend time on a direct fix than a work around or recap of where we were

Add me to the list of people. I am very surprised about this limitation and would like to be pointed at more info. Are we treating this as a bug to fix (because IMHO it is a serious one)

Thanks everyone, I have done so. Anyone know what the back story to this is? 2017.07.09c ..Add: Warning in the format and partition dialog that a UD partitioned disk cannot be added to the array.