Cache_dirs scan level depth "auto" not enough?


Recommended Posts

I've had an ongoing issue with drives and their contents staying in cache_dirs memory for what seems like forever, but it's definitely much longer than needs to be.  I've never gotten a straight forward answer as to why or what could be causing this issue, but I'm going to lay it all out in hopes that someone can shine some light:

 

I currently have a 3 drive array, all contain parts of Movies and Television.

When adding a new series in sonarr, any time I add a new series, drive 3 will spin up.  Drive 3 is the newest of the drives, all drives are XFS formatted, not in spinup groups, all have the same default 30 minute spin down delay.  This is the only time this drive spins up, aside from actually using the drive to watch a show, etc.

 

I understand sonarr has to check the array to see if any files exist for this new show, that's not the odd part.

The odd part is, when using couchpotato to download a movie, once said movie completes downloading, unpacking, the second unpack finishes, ONLY drives 1 and 2 spin up.  Again, same XFS formatting, no spingroups, same 30 min delay.  Drive 3 NEVER spins up.

 

The same principle occurs when just trying to browse the share in windows through samba network.  Drives 1 and 2 will spin if I go into /Movies, Drive 3 will spin if I go into /Television.

 

 

So I ask, is there something wrong with my default settings of cache_dirs? Default essentially, and have included drives: Movies and Television.

I have literally no other idea what could possibly be causing this.  Docker paths all point downloads to cache, Move files to array, which obviously just symlinks them.

 

I can provide any pics of settings, setup, etc as need be.  I'd really just like some, any, help regarding this.  It's driving me absolutely crazy attempting to figure out what could be causing this.  Open Files plugin can only help so much.

Link to comment
  • Replies 60
  • Created
  • Last Reply

Top Posters In This Topic

Top Posters In This Topic

Posted Images

I have to admit being confused by your posting, so I'll start with some questions.

 

I've had an ongoing issue with drives and their contents staying in cache_dirs memory for what seems like forever, but it's definitely much longer than needs to be.  I've never gotten a straight forward answer as to why or what could be causing this issue, but I'm going to lay it all out in hopes that someone can shine some light:

I don't understand what's the issue, don't you want the directory entries to stay in memory?  Cache_dirs doesn't have memory, it just influences the kernel to not dump the directory entries out of the kernel's directory buffers.

 

I currently have a 3 drive array, all contain parts of Movies and Television.

When adding a new series in sonarr, any time I add a new series, drive 3 will spin up.  Drive 3 is the newest of the drives, all drives are XFS formatted, not in spinup groups, all have the same default 30 minute spin down delay.  This is the only time this drive spins up, aside from actually using the drive to watch a show, etc.

 

I understand sonarr has to check the array to see if any files exist for this new show, that's not the odd part.

The odd part is, when using couchpotato to download a movie, once said movie completes downloading, unpacking, the second unpack finishes, ONLY drives 1 and 2 spin up.  Again, same XFS formatting, no spingroups, same 30 min delay.  Drive 3 NEVER spins up.

 

The same principle occurs when just trying to browse the share in windows through samba network.  Drives 1 and 2 will spin if I go into /Movies, Drive 3 will spin if I go into /Television.

You've listed 3 different situations, where drives are spinning up or not spinning up, and I'm not sure what is wrong or unexpected.  The behavior seems normal to me, so clarify which situations do not seem right to you.  When is it that drives are spinning up and you feel they should not be, and when drives are not spinning when you feel they should be.

Link to comment

I have to admit being confused by your posting, so I'll start with some questions.

 

I've had an ongoing issue with drives and their contents staying in cache_dirs memory for what seems like forever, but it's definitely much longer than needs to be.  I've never gotten a straight forward answer as to why or what could be causing this issue, but I'm going to lay it all out in hopes that someone can shine some light:

 

I don't understand what's the issue, don't you want the directory entries to stay in memory?  Cache_dirs doesn't have memory, it just influences the kernel to not dump the directory entries out of the kernel's directory buffers.

 

 

I DO want the entries to stay in memory, but from how it appears to be acting over the last few weeks, it seems that the settings either aren't sticking, or something within cache_dirs settings parameters I should be putting back.  I should mention I'm also on 2.1.1 the latest.  See pic attached for current setup.

 

I currently have a 3 drive array, all contain parts of Movies and Television.

When adding a new series in sonarr, any time I add a new series, drive 3 will spin up.  Drive 3 is the newest of the drives, all drives are XFS formatted, not in spinup groups, all have the same default 30 minute spin down delay.  This is the only time this drive spins up, aside from actually using the drive to watch a show, etc.

 

I understand sonarr has to check the array to see if any files exist for this new show, that's not the odd part.

The odd part is, when using couchpotato to download a movie, once said movie completes downloading, unpacking, the second unpack finishes, ONLY drives 1 and 2 spin up.  Again, same XFS formatting, no spingroups, same 30 min delay.  Drive 3 NEVER spins up.

 

The same principle occurs when just trying to browse the share in windows through samba network.  Drives 1 and 2 will spin if I go into /Movies, Drive 3 will spin if I go into /Television.

 

You've listed 3 different situations, where drives are spinning up or not spinning up, and I'm not sure what is wrong or unexpected.  The behavior seems normal to me, so clarify which situations do not seem right to you.  When is it that drives are spinning up and you feel they should not be, and when drives are not spinning when you feel they should be.

 

In the past, lets say for example, I've decided to add a new show to sonarr, that doesn't exist on the server yet.  Technically all of those folders within /mnt/user/Television should be cached within memory from cache_dirs running the whole time.  Adding a new series, no spin up occurs. Series page just creates itself within Sonarr.  No spin up.

 

Now, however, and also with Radarr, any time I add a new series that doesn't exist or a new movie that doesn't exist, sometimes all 3 drives will spin up.  All 3 contain /mnt/user/Movies and /mnt/user/Television.

 

Is this due to my current settings for cache_dirs? Like what would cause this behavior to completely change?  Have I somehow exceeded the limit of it being cached into memory?  I don't consider the amount of folders within Movies or Television to be so enormous that this would basically seem like nothing is stored in memory.  1010 movies, each within folders within /Movies, 200 series, within /mnt/user/TV

 

Edited.  Added 2 other pics, Shares (all setup the same way), and dockers tab

cache.jpg.6d0e64ec8ed745bd9d44fcae6ddfd802.jpg

Shares.jpg.9de6df41fdd6bd867c7751b9440ccc24.jpg

dockers1.jpg.1b8581a3db0fdd39cfa9bfdb8fa7d02e.jpg

Link to comment

This is not causing problems, but one thing I would change is your Includes and Excludes.  I always recommend setting the includes to be *only* those shares you actually want to be cached, then drop any excludes, not needed.  Now, you are caching everything except 2 folders, including junk and temporaries and other folders that don't need to be cached, wasting memory and a little time.  Just click on Includes, and click on Movies and Television.  The other settings are fine.

 

When files are downloaded and saved to disk, that disk needs to spin up of course.  Just browsing cached disks should not cause spin ups, unless you are really low in available RAM, or there's something beside the simple directory entry being requested by the browsing tool.  That may apply to your Windows browsing.  If the Windows tool (Explorer or anything else) wants to check internal metadata (like resolution, minutes, etc) or thumbnails or a file icons, then those aren't cached by CacheDirs, and the drive will have to spin up to access that data.  If the file itself has to be opened for other info or data, then CacheDirs can't help.  If that's the problem, then you should be able to stop that by configuring the browsing tool not to access such extra info (icons, thumbnails, and other metadata).

Link to comment

Correct, browsing the disk (whether via MC, or ftp, etc) no spin up is created.  Windows - totally different story, I get that.

However, shouldn't the same apply to a download or adding a new series to sonarr? Doesn't cache come into play that way too?

 

If sonarr completes an episode downloading, nzbget unpacks, sends it to the array, nothing spins up.  Not anything at all, it grabs metadata, the images needed from metadata about the episode, the show, etc and the file gets updated in sonarr saying the file is now there in the listing.  Never have an issue with an existing show in the database.

 

If couch or radarr handle a movie, nzbget downloads, unpacks, grabs metadata... the second nzbget finishes downloading, every drive spins, even though those files are technically just moving to the Movie folder held on cache....  I don't understand how one can work yet the other can't.  Or better yet, how sonarr can work perfectly for ages, couch can work for weeks without a problem, and then just one day decides to start spinning up drives again.

Link to comment

If sonarr is saving a file to disk, that disk MUST spin up.  So if the array data drives aren't spinning up, then sonarr is not saving the files to the array.  It must be saving them to a Cache drive or an unassigned drive (which are spinning).

 

Sorry correct yes, they're saving to the cache drive, to be later moved by mover.  The same applies to movies, but movies cause the spinups, tv does not.  Yet they're both going in the end to the same place, mnt/cache/Movies or /mnt/cache/Television.

Link to comment

Just wanted to know if anyone else has any input, or possibly suffering from the same spinups due to Movies, yet not tv.  All dockers relating to downloading are setup the same way, and all paths essentially lead to downloads going to the cache, NOT the array directly.

 

The spinups seem to occur right as files (again, movies only) are finished unpacking and moving to the pick up directory (also cache: /mnt/cache/Downloads/Movies), where couch, radarr, what have you, sees, picks up and renames and moves to /mnt/user/Movies/Movie.Name, but Movies share uses the cache, so technically it's /mnt/cache/Movies.

 

Again, any help would be appreciated with this, this is an issue I've dealt with far too long, that's come and gone, with having absolutely no clue what causes the spin ups.  If sonarr handles this so well and doesn't spin for anything (file, metadata, series images, etc), why do movies do it?

 

See attached image for docker setup

docker.jpg.940ead8f6cd74b4c41335221f058d1ae.jpg

Shares.jpg.1251920a246c14db2b071b2710d8de87.jpg

cache.jpg.15ceea88425f528b958a63e1c9a98ab2.jpg

Link to comment

The spinups seem to occur right as files (again, movies only) are finished unpacking and moving to the pick up directory (also cache: /mnt/cache/Downloads/Movies), where couch, radarr, what have you, sees, picks up and renames and moves to /mnt/user/Movies/Movie.Name, but Movies share uses the cache, so technically it's /mnt/cache/Movies.

Not exactly.  First thing that has to happen when copying anything to /mnt/user is that the system has to check to see if the file and/or folder already exists within the array disks so that it knows if it has to over write the existing file.

 

So technically any write to a cache enabled share may possibly spin up disks.

 

TBH, I've always found cache dirs to be somewhat finicky.  I only have it enabled on two specific shares both of which contain thousands of folders, and its always been a bit of a hit/miss whether the entries are cached in memory to prevent a spin up.  Most of the time, the majority of entries are cached, because an access to one of those shares does not spin up all of the disks.  I'm far to busy to actually investigate the options available within cache dirs to try and eliminate the problem altogether, and it doesn't particularly bother me.

 

 

Link to comment

The spinups seem to occur right as files (again, movies only) are finished unpacking and moving to the pick up directory (also cache: /mnt/cache/Downloads/Movies), where couch, radarr, what have you, sees, picks up and renames and moves to /mnt/user/Movies/Movie.Name, but Movies share uses the cache, so technically it's /mnt/cache/Movies.

Not exactly.  First thing that has to happen when copying anything to /mnt/user is that the system has to check to see if the file and/or folder already exists within the array disks so that it knows if it has to over write the existing file.

 

So technically any write to a cache enabled share may possibly spin up disks.

 

TBH, I've always found cache dirs to be somewhat finicky.  I only have it enabled on two specific shares both of which contain thousands of folders, and its always been a bit of a hit/miss whether the entries are cached in memory to prevent a spin up.  Most of the time, the majority of entries are cached, because an access to one of those shares does not spin up all of the disks.  I'm far to busy to actually investigate the options available within cache dirs to try and eliminate the problem altogether, and it doesn't particularly bother me.

 

I understand, so it definitely isn't that my setup is incorrect, this does in fact happen to basically everyone.  And you're right, sometimes it's only 1 or 2 drives, other times it's all that contain the top level folder... there doesn't seem to be any rhyme or reason.  I just always thought something was wrong because adding something to sonarr or adding any files with sonarr never spins up a drive, where as with movies, 9 times out of 10 it will occur.

 

If there was a way of keeping a "Movies" and "Television" as part of the cache drive at all times (essentially mover wouldn't move the entire /Movies directory over to the array) this completely stops this from happening.  Downloads point to /mnt/cache/X, entries stay on cache awaiting mover, no spin up occurs with movies and tv this way, and since this is technically a symlink location, anything from plex to kodi will see this link regardless.... but I don't know how something like this would even be done.

Link to comment

Unless of course there is some way to have a constant folder stay in /mnt/cache, and not have mover move that top folder? So there's always a /Movies in /mnt/cache/Movies?

Just trying to get as little pointless spinups as possible

If you set a user share as either cache-no or cache-only, it won't get moved.

 

cache-no will not write new files to cache.

cache-only will write new files only to cache.

 

In either case, the user share will still be able to read files from all disks unless a disk is excluded from Global Share Settings. So if you have some files for a cache-only share on disk1, for example, they would be included in reads of the share.

Link to comment

Unless of course there is some way to have a constant folder stay in /mnt/cache, and not have mover move that top folder? So there's always a /Movies in /mnt/cache/Movies?

Just trying to get as little pointless spinups as possible

If you set a user share as either cache-no or cache-only, it won't get moved.

 

cache-no will not write new files to cache.

cache-only will write new files only to cache.

 

In either case, the user share will still be able to read files from all disks unless a disk is excluded from Global Share Settings. So if you have some files for a cache-only share on disk1, for example, they would be included in reads of the share.

 

I understand what you mean, but cache-no would mean, if say it was Movies, all files going to Movies would just get written directly to array.  Cache-only would mean all those files would stay on the cache, much like appdata... so both of those aren't what I want.

 

Maybe I didn't explain it right, what I meant was to basically have an always there folder called /Movies or /Television in /mnt/cache/ that is always there, that way I can have a docker write to it all the time.  When mover moves, it takes the entire contents of those files waiting to move to the array, top folder included, so at the end of the day those /Movies and /Television folders, regardless of being empty of not, I would like to still exist.  That way they match up with their exacts on the array, but I can avoid getting the spin up that I seem to be having by Radarr/Couch/whatever is creating.

 

Point docker to final dir of /mnt/cache/Movies/X = no spin up

Point docker to final dir of /mnt/user/Movies = spin up

 

Hopefully I clarified this enough.

TL;DR = if those folder that get moved to the array, the top level folder (/Movies and /Television in this case), to have that very top level never move from /mnt/cache/

Link to comment

Unless of course there is some way to have a constant folder stay in /mnt/cache, and not have mover move that top folder? So there's always a /Movies in /mnt/cache/Movies?

Just trying to get as little pointless spinups as possible

If you set a user share as either cache-no or cache-only, it won't get moved.

 

cache-no will not write new files to cache.

cache-only will write new files only to cache.

 

In either case, the user share will still be able to read files from all disks unless a disk is excluded from Global Share Settings. So if you have some files for a cache-only share on disk1, for example, they would be included in reads of the share.

 

I understand what you mean, but cache-no would mean, if say it was Movies, all files going to Movies would just get written directly to array.  Cache-only would mean all those files would stay on the cache, much like appdata... so both of those aren't what I want.

 

Maybe I didn't explain it right, what I meant was to basically have an always there folder called /Movies or /Television in /mnt/cache/ that is always there, that way I can have a docker write to it all the time.  When mover moves, it takes the entire contents of those files waiting to move to the array, top folder included, so at the end of the day those /Movies and /Television folders, regardless of being empty of not, I would like to still exist.  That way they match up with their exacts on the array, but I can avoid getting the spin up that I seem to be having by Radarr/Couch/whatever is creating.

 

Point docker to final dir of /mnt/cache/Movies/X = no spin up

Point docker to final dir of /mnt/user/Movies = spin up

 

Hopefully I clarified this enough.

TL;DR = if those folder that get moved to the array, the top level folder (/Movies and /Television in this case), to have that very top level never move from /mnt/cache/

Mover just works with user shares, not folders within user shares. Actually, a user share is a top level folder, but there is no way to get its subfolders treated differently by mover. If you want mover to process something differently than something else, you will have to put it in a different share.

 

Another possibility would be to replace mover with your own script.

Link to comment

Unless of course there is some way to have a constant folder stay in /mnt/cache, and not have mover move that top folder? So there's always a /Movies in /mnt/cache/Movies?

Just trying to get as little pointless spinups as possible

If you set a user share as either cache-no or cache-only, it won't get moved.

 

cache-no will not write new files to cache.

cache-only will write new files only to cache.

 

In either case, the user share will still be able to read files from all disks unless a disk is excluded from Global Share Settings. So if you have some files for a cache-only share on disk1, for example, they would be included in reads of the share.

 

I understand what you mean, but cache-no would mean, if say it was Movies, all files going to Movies would just get written directly to array.  Cache-only would mean all those files would stay on the cache, much like appdata... so both of those aren't what I want.

 

Maybe I didn't explain it right, what I meant was to basically have an always there folder called /Movies or /Television in /mnt/cache/ that is always there, that way I can have a docker write to it all the time.  When mover moves, it takes the entire contents of those files waiting to move to the array, top folder included, so at the end of the day those /Movies and /Television folders, regardless of being empty of not, I would like to still exist.  That way they match up with their exacts on the array, but I can avoid getting the spin up that I seem to be having by Radarr/Couch/whatever is creating.

 

Point docker to final dir of /mnt/cache/Movies/X = no spin up

Point docker to final dir of /mnt/user/Movies = spin up

 

Hopefully I clarified this enough.

TL;DR = if those folder that get moved to the array, the top level folder (/Movies and /Television in this case), to have that very top level never move from /mnt/cache/

Mover just works with user shares, not folders within user shares. Actually, a user share is a top level folder, but there is no way to get its subfolders treated differently by mover. If you want mover to process something differently than something else, you will have to put it in a different share.

 

Another possibility would be to replace mover with your own script.

 

Ok, got it.  Was just curious if it was even a possibility.

Thanks for the responses guys

Link to comment

You can do what you want, sort of...

 

Tell couch to put the files in /mnt/cache/Movies/whatever.  A spinup of the array disks will never happen.

 

Movies will be a useCache=yes share.  Mover comes along daily and moves the files to the array.

 

Downsides

 

- Because you're writing directly to the cache drive and not to the user share, if there is a conflict in file names, Mover will probably bitch at you and not move the file.

- If there is a conflict in the file names, and someone wants to play the movie in question, then unRaid will pick the one stored on the lowest numbered disk to play (ie: the one not on the cache drive)

- Since mover will kill the movies folder on the cache drive after it moves, you have to have a post-processing script that will recreate it whenever a movie is downloaded so that CP doesn't error out because the folder doesn't exist

 

* conflicts will happen whenever CP decides to upgrade the quality, etc

 

All in all anything is possible, but IMHO the aggravation isn't worth it.

Link to comment

You can do what you want, sort of...

 

Tell couch to put the files in /mnt/cache/Movies/whatever.  A spinup of the array disks will never happen.

 

Movies will be a useCache=yes share.  Mover comes along daily and moves the files to the array.

 

Downsides

 

- Because you're writing directly to the cache drive and not to the user share, if there is a conflict in file names, Mover will probably bitch at you and not move the file.

- If there is a conflict in the file names, and someone wants to play the movie in question, then unRaid will pick the one stored on the lowest numbered disk to play (ie: the one not on the cache drive)

- Since mover will kill the movies folder on the cache drive after it moves, you have to have a post-processing script that will recreate it whenever a movie is downloaded so that CP doesn't error out because the folder doesn't exist

 

* conflicts will happen whenever CP decides to upgrade the quality, etc

 

All in all anything is possible, but IMHO the aggravation isn't worth it.

 

Good idea on the ppscript route, hadn't thought of that, I think that may be the best option for this.

Thanks!

Link to comment
  • 2 weeks later...

As an update... I appear to still be getting these spinups, only on a much worse scale now.  With nothing else about cache_dirs, the user shares, or anything else changing since.

From what I can see, cache_dirs being active, enabled, with the settings attached to this post, now results in even full spin ups browsing through MC on a putty session, as well as even through an ftp connection - which both of those never once ever did.

 

Is there a method to find a log of these spin ups and whats accessing this? System log doesn't give details, only thing I know of is the Open Files plugin, but thats not refreshed automatically, so I'll never catch the spin up as it occurs or what's causing it.

 

If there is any discrepancy someone can point out in either of these settings which make cause something like, whether it be docker or cache_dirs or the shares, please by all means, I'd appreciate the helpful info.  I was at my wits end when I created this thread, at this point it's making me lose sleep.

 

I've completely disabled plex now, as well as any notifications to it or kodi, so at this point it's just the NAS and the drives, with nothing accessing it.

 

As an update, plex and kodi both completely off and not receiving any notifications.

Sonarr can dl however many episodes, nothing spins.  Radarr downloads (based off those docker paths below), SPIN UP OCCURS.  How in the holy hell. 1 disk spun, disk 3.  0 clue as to why. Either a symptom of nothing existing for the movie prior, therefore Radarr had to create the directory that didn't exist first (again, Movies share is Use Cache: Yes).....

 

xSJyKal.jpg

IiFqZoq.jpg

WaS973v.jpg

fHHUVHN.jpg

d1bJG38.jpg

Link to comment

Access to excluded folders will cause spin ups.  I would default to not including or excluding any folders and let cache_dirs cache all shares.  Then tune it if you find it is working better.

 

I've found that if your system does not have enough memory, cache_dirs will not be able to cache enough and you will see spin ups.

Link to comment

Access to excluded folders will cause spin ups.  I would default to not including or excluding any folders and let cache_dirs cache all shares.  Then tune it if you find it is working better.

 

I've found that if your system does not have enough memory, cache_dirs will not be able to cache enough and you will see spin ups.

 

Memory size allocated 7.834 GB

installed 8 GB (max. capacity 32 GB)

 

I have no issue with running cache_dirs for all shares, I'll gladly switch over to that now.

Link to comment

Well it just happened again, this time spinning up every drive in the array.

Caught what I could on the Open Files plugin.

 

DCD2yKd.jpg

uJWvQux.jpg

 

And as I said in the previous post, Movies is set to Use Cache: Yes.

Plex is turned off.  No notifications or update libraries are going out to anything.  This is unraid just doing this all on its own.

I've also changed the cache_dirs settings to not include or exclude any, so it's caching all shares.

the Sin.City.(2005) folder and contents are on the cache in /mnt/cache/Movies, awaiting mover... but clearly unraid has other plans or something is still very wrong, as like I said, every drive spun up.

 

Any other thoughts from anyone would be incredibly helpful.  More than willing to change any settings, I've tried everything I can think of at this point.

 

If it helps at all, my current permissions settings for Sonarr and Radar:

File & folder chmod mask: 0777

Chown user: 99 (nobody)

Chown group: 100 (users)

 

All shares and folders are also set to 0777/0777 99/100

Link to comment

As a side note, if this were in fact a "not enough memory" issue and cache_dirs wasn't fully caching shares because of it (I can't imagine it being since allocation is barely used at all for cache_dirs), what alternative could I even do otherwise to figure out whats causing these spin ups?

 

Is the Open Files plugin my best bet at trying to figure out the culprit? iotools/ionotify?

Something is causing this that I almost never saw occur in the past, but things don't just change without some sort of reason, and that reason is what I can't figure out.

I don't feel like the amount of files I have in directories is a lot, especially compared to other users (at most, it's just regular video file, matching nfo and matching poster/episode image)

Link to comment

I have also seen random spinups with docker applications that don't seem like they should be spinning an HDD in the array until the mover runs. I possibly don't quite understand how cache-dirs works entirely, but I thought the point was to keep directories in memory and that HDD's wouldn't spin up until the mover runs. So following the advice on this thread I should place my mount points for things like sonarr and couchpotato to /mnt/cache/media/movies or /mnt/cache/media/tv/ would solve these HDD spinups at download?

Link to comment

I have also seen random spinups with docker applications that don't seem like they should be spinning an HDD in the array until the mover runs. I possibly don't quite understand how cache-dirs works entirely, but I thought the point was to keep directories in memory and that HDD's wouldn't spin up until the mover runs. So following the advice on this thread I should place my mount points for things like sonarr and couchpotato to /mnt/cache/media/movies or /mnt/cache/media/tv/ would solve these HDD spinups at download?

 

What he said!

I'd really love to know whats causing my "rogue spinups" as I'll start calling them.  If there's a method to all this madness in finding a way to see what could be doing it, I'm not familiar with how to properly use something like ionotify/iotools

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.