@DZMM Thanks for looking into it. Unfortunately this isn't what I was looking for. I'm fine with it writing to the cache at all times, that's the intent, I only want it to read from the first available location in the chain. But thanks to your hint with the policies I actually found the solution.
It's the "category.search=ff" setting. By adding this I can ensure that the merged directory will follow the order of the sources in the mergerfs command. I just tested it and it works exactly as I wanted it to.
If a file with the same name exists in all four merged sources it will first display the one on the cache (/mnt/user/media), then the array (/mnt/user/media_array), then the regular Gdrive (/mnt/disks/media_remote) and finally on the Team Share (/mnt/disks/media_team). As soon as I delete the file on the first path in the chain, it will show the next available one.
My mergerfs command now looks like this:
mergerfs /mnt/user/media_cache:/mnt/user/media_array:/mnt/disks/media_remote:/mnt/disks/media_team /mnt/disks/media -o rw,async_read=false,use_ino,allow_other,func.getattr=newest,category.action=all,category.create=ff,category.search=ff,cache.files=partial,dropcacheonclose=true
I renamed media to media_cache to make it easier to see which source it is. The only added entry is "category.search=ff" which does exactly what I'm looking for.