Dynamix - V6 Plugins


bonienl

2973 posts in this topic Last Reply

Recommended Posts

Thanks, dlandon.

 

I want to have a github version-tracking of non-zipped, so I can track changes properly. It would we valuable for the community if someone hosted such a one on Github. It would probably be better if someone else hosts the cache_dirs, maybe someone who responds as quickly as you do dlandon :)

 

I've changed my unRAID repository to contain the raw file, and added links to dlandon and dynamix githubs. https://github.com/arberg/unRAID

 

I've updated the cache_dirs a bit, but its still work in progress. I hope to make the script prevent spamming disks, which was my idea for the adaptiveness in the first place. So I might make the script back off a lot sooner and for a longer period when it detects it is spamming disk.

 

Alex

 

PS on your GitHub comment

>> Thank you Alex R. Berg for your updates to cache_dirs.

You are welcome :)

Link to post
  • Replies 3k
  • Created
  • Last Reply

Top Posters In This Topic

Top Posters In This Topic

Popular Posts

Notice: You must be running unRAID version 6.1 or later to use these plugins The easiest way of installing plugins is thru Community Applications. This is an apps installation manager develope

Today has been a productive day   I have updated several Dynamix plugins, mainly bug fixes. Please check your plugins to get the updates.

I'll put this on my todo list. Had several (failed) attempts in the past, and need a good review of the approach to take.

Posted Images

22 minutes ago, Alex R. Berg said:

Thanks, dlandon.

 

I want to have a github version-tracking of non-zipped, so I can track changes properly. It would we valuable for the community if someone hosted such a one on Github. It would probably be better if someone else hosts the cache_dirs, maybe someone who responds as quickly as you do dlandon :)

 

I've changed my unRAID repository to contain the raw file, and added links to dlandon and dynamix githubs. https://github.com/arberg/unRAID

 

I've updated the cache_dirs a bit, but its still work in progress. I hope to make the script prevent spamming disks, which was my idea for the adaptiveness in the first place. So I might make the script back off a lot sooner and for a longer period when it detects it is spamming disk.

 

Alex

 

PS on your GitHub comment

>> Thank you Alex R. Berg for your updates to cache_dirs.

You are welcome :)

I can add an unzipped script to my github and you can do pr's to make changes and I'll release it as the zipped version for the plugin.

Link to post
48 minutes ago, Alex R. Berg said:

Thanks, dlandon.

 

I want to have a github version-tracking of non-zipped, so I can track changes properly. It would we valuable for the community if someone hosted such a one on Github. It would probably be better if someone else hosts the cache_dirs, maybe someone who responds as quickly as you do dlandon :)

 

I've changed my unRAID repository to contain the raw file, and added links to dlandon and dynamix githubs. https://github.com/arberg/unRAID

 

I've updated the cache_dirs a bit, but its still work in progress. I hope to make the script prevent spamming disks, which was my idea for the adaptiveness in the first place. So I might make the script back off a lot sooner and for a longer period when it detects it is spamming disk.

 

Alex

 

PS on your GitHub comment

>> Thank you Alex R. Berg for your updates to cache_dirs.

You are welcome :)

@Alex R. Berg you are all set on my github.  I can do some testing for you and I'll release the updated script when it's ready.

Link to post

Good. I've committed my changes for now to my own repository here: https://github.com/arberg/cache_dirs

When I think its worthy of your time, i'll post a pull-request to your repository. Or wait a week and have a look, if I should suddenly get busy elsewhere and not return.

 

I have not read up on all the issues posted here on cache-dirs, but as far as I have picked up (and from my own experience) cache-dirs still in my adaptive version spams to much on the disks. I found some definite design-issues with my adaptive scan, where it would keep trying to hard to get to it previous best depth. I have committed my changes, and will keep looking at it.

 

Link to post

I've considered updating the help for the cache-dir plugin. I've found it valuable to set max-scan depth even though I've now made the adaptive scanning. Its valuable to make sure not to have more files than needed under cache-dirs scanning eye, to reduce disk-spam, and I might write that in the documentation. 

 

Do we have any kind of guideline anywhere for how to write help? If I update FolderCachingSettings.page and make a pull-request, will BergWare do his magic?

https://github.com/bergware/dynamix/blob/master/source/cache-dirs/FolderCachingSettings.page

 

 

Best Alex

Link to post
2 hours ago, Alex R. Berg said:

I've considered updating the help for the cache-dir plugin. I've found it valuable to set max-scan depth even though I've now made the adaptive scanning. Its valuable to make sure not to have more files than needed under cache-dirs scanning eye, to reduce disk-spam, and I might write that in the documentation. 

 

Do we have any kind of guideline anywhere for how to write help? If I update FolderCachingSettings.page and make a pull-request, will BergWare do his magic?

https://github.com/bergware/dynamix/blob/master/source/cache-dirs/FolderCachingSettings.page

 

 

Best Alex

Yes he will.  He's very responsive and would probably appreciate the pr.  Do a pull request and you'll see how the help is done.  Its pretty easy.  The plugin currently does not set up the script for adaptive scan.  I've asked bonienl to add that as an option.

Link to post
On 12.3.2018 at 1:22 PM, bonienl said:

I'll put this on my todo list. Had several (failed) attempts in the past, and need a good review of the approach to take.

 

Hi bonienl,

is there any intermediate fix for the problem with the S3 Script? Maybe a downgrade to the old algorithm to determinate if a disk is spun down or a way to make the hdparm check optional?

Link to post
On 7.4.2018 at 5:40 PM, darkside40 said:

 

Hi bonienl,

is there any intermediate fix for the problem with the S3 Script? Maybe a downgrade to the old algorithm to determinate if a disk is spun down or a way to make the hdparm check optional?

I would be interested too
Link to post

Cache_dirs update

 

I've update the cache_dirs plugin. Made many many small changes that I consider improvements. I've update the dynamix plugin folder, fixed a bug in the start script, and added options to the settings page and added help info.

 

I've forked dlandon and dynamix githubs, and pushed my changes. I have updated the dynamix-txz and cache_dirs.zip in my forks, so if you use this temporary plg attached here which points to my forks, you can test it. The new plugin requires the new cache_dirs script because I've added new options to the script.

 

The current attached cache_dirs plugin does not start when the server reboots on my server where it auto-start the array. I hope someone can fix that for me. 

 

If you send me logs from /var/log/cache_dirs.log and /var/log/cache_dirs_lost_cache.csv I can inspect them, though I'll probably quickly tire of looking at logs, but it would be helpful to see how cache_dir is behaving on other systems.

 

# Version 2.2.0g- When cache lost, don't wait for disks idle, instead decrease depth
#               - Now scans each disks and cache in separate processes. Thus removed option -u for scan user
#               - Fixed loss of adaptive after 1 week
#               - Weekly rescan only when disks been idle long
#               - Apaptive scan will not retry increasing level to previous seen stable level until disks have been idle for a long time (20s)
#               - -q now also kills subproccesses immediately (so cache_dirs does not prevent stopping the array)
#               - Added file-count and depth adjustment by filecount
# arberg 

 

If your disks are never idle for 20 seconds the adaptive scanning will no longer work for you. If the disks are idle once in a while, I believe it will work far better than before.

 

Best Alex

dynamix.cache.dirs.zip

Link to post
10 hours ago, nuhll said:

Will this get updated automatic?

 

Its not yet promoted to the automatic update stream. That will happen when its tested a bit more and approved. Till then you can try the changes by replacing the cache-dirs plugin with the version i linked. You can just copy the file over the existing in /boot/config/plugins/dynamix.cache.dirs.plg and reboot (or run something like installplg or remove and install from gui). 

 

Best Alex

Link to post
On 4/7/2018 at 5:40 PM, darkside40 said:

 

Hi bonienl,

is there any intermediate fix for the problem with the S3 Script? Maybe a downgrade to the old algorithm to determinate if a disk is spun down or a way to make the hdparm check optional?

 

Sorry, I don't have time right now to look into this, and have to keep it on my todo list.

Link to post
14 hours ago, bonienl said:

 

Sorry, I don't have time right now to look into this, and have to keep it on my todo list.

 

Fair enough mate, Is there a http command that i could send the server to tell it to manually sleep?

 

This way in the meantime i could have Home Assistant put it to sleep once my media players have been switched off for the night?

Link to post

Cache_dirs update 2.2.0

 

I think I'm done with the 2.2.0 update. I've attached a version of the plugin that points to my forked github so you can test it until it goes live. Note that because of how the plugin system works

 

I've fixed the issues mentioned in my previous comment, about the plugin not starting on boot, it was my bad.

 

Users notes

  • The new cache_dirs scans your disks when theive been idle for 60 seconds. This can now be adjusted in the options. If the disks are idle once in a while, it seems to work far better than before.
  • Cache-pressure 0 works impressively for my 1.5 million files and 16 GB of memory, but it may crash the system, because linux docs says it will not release cache-memory even if the system is running out of memory.  

 

Programmers notes

I've moved the cache_dirs to the dynamix plugin github. Te plugin and the cache-dirs version are tied together, the current cache-dirs page does not work properly with older cache_dirs versions. The plugin contained the MD5 of the cache_dirs.zip, and if the cache_dirs.zip was updated but the plugin was not updated, it would cause 'unexpected' behaviour, such as redownload on each reboot.

Since I've moved cache_dirs to dynamix github I've added a note that its not under copyright of dynamix. You can move it back out if you want. The newest script is also committed in dlandons github (in my fork). 

 

Future ideas

  • The cache_dirs log is very useful to tweak your personal system. If it was possible to copy the Syslog.htm page and point it to /var/log/cache_dirs.log and put a link to it in the Folder Caching page it would be very useful for users.
  • It would be nice with restart buttons like on the PhAzE's Resilio sync plugin.

Release notes

# Version 2.2.0 - When cache lost, don't wait for disks idle, instead decrease depth
#               - Now scans each disks and cache in separate processes. Thus removed option -u for scan user
#               - Fixed loss of adaptive after 1 week
#               - Weekly rescan only when disks been idle long
#               - Apaptive scan will not retry increasing level to previous seen stable level until disks have been idle for a long time (20s)
#               - -q now also kills subproccesses immediately (so cache_dirs does not prevent stopping the array)
#               - Added file-count and depth adjustment by filecount 
# arberg 

 

Best Alex

 

dynamix.cache.dirs.zip

Link to post
  • 2 weeks later...
On 11.4.2018 at 8:02 AM, bonienl said:

 

Sorry, I don't have time right now to look into this, and have to keep it on my todo list.

Okay i just changed the script by myself using the HDD Disk active check method form 3.0.4.

Works perfect till now.

Link to post

Here you can find the modified script:

https://pastebin.com/knv869Aq

 

The modifications are done between line 233-236, just commenting out the new check and adding the old style activity check from 3.0.4.

 

I have also added an archive which includes the plg (because you need to change the md5sum of the txz after repackaging) and the txz with the script version mentioned above. Ignore the config, you will only need the plg and the txz.

For me it works, wouldnt take any guarantee  if it does for you.

 

It is just a really ugly hacky fix till bonienl has the time to make an official fix, because the 3.0.6 didnt work for me most of the time.

 

sleep.tar.gz

Edited by darkside40
Link to post
11 minutes ago, darkside40 said:

Here you can find the modified script:

https://pastebin.com/knv869Aq

 

The modifications are done between line 233-236, just commenting out the new check and adding the old style activity check from 3.0.4.

 

I have also added an archive which includes the plg (because you need to change the md5sum of the txz after repackaging) and the txz with the script version mentioned above. Ignore the config, you will only need the plg and the txz.

For me it works, wouldnt take any guarantee  if it does for you.

 

It is just a really ugly hacky fix till bonienl has the time to make an official fix, because the 3.0.6 didnt work for me most of the time.

 

sleep.tar.gz

Thank you very much
Link to post

Thanks for the modified script. I am also giving it a try. I disabled sleep for the moment. I installed the .txz file into the dynamic.s3.sleep folder on the flash drive and copied the .plg file overwriting the existing .plg file on the flash drive and restarted the plugin. Is that all that has to be done?

 

Link to post
On 4/13/2018 at 4:09 AM, Alex R. Berg said:

Cache_dirs update 2.2.0

 

I think I'm done with the 2.2.0 update. I've attached a version of the plugin that points to my forked github so you can test it until it goes live. Note that because of how the plugin system works

 

I've fixed the issues mentioned in my previous comment, about the plugin not starting on boot, it was my bad.

 

Users notes

  • The new cache_dirs scans your disks when theive been idle for 60 seconds. This can now be adjusted in the options. If the disks are idle once in a while, it seems to work far better than before.
  • Cache-pressure 0 works impressively for my 1.5 million files and 16 GB of memory, but it may crash the system, because linux docs says it will not release cache-memory even if the system is running out of memory.  

 

Programmers notes

I've moved the cache_dirs to the dynamix plugin github. Te plugin and the cache-dirs version are tied together, the current cache-dirs page does not work properly with older cache_dirs versions. The plugin contained the MD5 of the cache_dirs.zip, and if the cache_dirs.zip was updated but the plugin was not updated, it would cause 'unexpected' behaviour, such as redownload on each reboot.

Since I've moved cache_dirs to dynamix github I've added a note that its not under copyright of dynamix. You can move it back out if you want. The newest script is also committed in dlandons github (in my fork). 

 

Future ideas

  • The cache_dirs log is very useful to tweak your personal system. If it was possible to copy the Syslog.htm page and point it to /var/log/cache_dirs.log and put a link to it in the Folder Caching page it would be very useful for users.
  • It would be nice with restart buttons like on the PhAzE's Resilio sync plugin.

Release notes

# Version 2.2.0 - When cache lost, don't wait for disks idle, instead decrease depth
#               - Now scans each disks and cache in separate processes. Thus removed option -u for scan user
#               - Fixed loss of adaptive after 1 week
#               - Weekly rescan only when disks been idle long
#               - Apaptive scan will not retry increasing level to previous seen stable level until disks have been idle for a long time (20s)
#               - -q now also kills subproccesses immediately (so cache_dirs does not prevent stopping the array)
#               - Added file-count and depth adjustment by filecount 
# arberg 

 

Best Alex

 

dynamix.cache.dirs.zip

Seems to be working nicely.  Hopefully @bonienl can make it official.  @bonienl let me know if you meed any help.

Link to post
12 hours ago, bilben1 said:

Thanks for the modified script. I am also giving it a try. I disabled sleep for the moment. I installed the .txz file into the dynamic.s3.sleep folder on the flash drive and copied the .plg file overwriting the existing .plg file on the flash drive and restarted the plugin. Is that all that has to be done?

 

 

Should be. After a reboot you should see Script Version 3.0.6.1 on the S3 Sleep settings page.

Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.