jonp Posted June 18, 2015 Share Posted June 18, 2015 One of the biggest limitations of managing game installations in Windows is the inability to move games from one storage location to another. Instead, we are forced to uninstall and reinstall games from scratch, increasing the time and effort required to manage our storage needs. In this video, we show you how to more efficiently manage your game library through user shares, enabling you to easily move games between high-performance and high-capacity storage without needing to reinstall them. We specifically highlight the proper procedures to configure this for both Steam and Origin. 7 1 Quote Link to comment
scottc Posted June 18, 2015 Share Posted June 18, 2015 Excellent Video Jon. I had not even though of doing it like that. Quote Link to comment
sane Posted June 19, 2015 Share Posted June 19, 2015 Some thoughts that occur, on the general subject. [*]Can you do something similar with Windows Applications, instead of games? Shifting rarely used large apps out of fast storage seems potentially useful. [*]Maybe I have caching on the brain after my suggested UNRAID improvement, but this seems like a handrolic caching approach to having current programs available on the fastest storage - eg its read caching for VMs. Another reason a real caching solution would be nice would be to make this transparent. [*]On a slightly more developed front, you can see a tiered structure of caching/backup that stretches to the separate desktop, with files cached to local SSD, tiered to UNRAID SSD and thence to UNRAID array. With a tolerable write-back structure your working desktop gets semi-backed-up to the array in real time - with old files migrating to only being on the UNRAID share, until needed. [*]The same concept, but with Android phones/tablets allows apps/data to be essentially infinite. SMB connections with Android are possible, not sure about install to a transparent share/fake SD card... Just thought I'd capture those whilst they were prompted by viewing your video. Quote Link to comment
jonp Posted June 19, 2015 Author Share Posted June 19, 2015 Can you do something similar with Windows Applications, instead of games? Shifting rarely used large apps out of fast storage seems potentially useful. Well, to be fair, games are applications in the eyes of Windows, but I understand what you're asking. The reality is that some will work, some won't, and some will come with limitations (similar to what I mentioned about Origin and resuming downloads near the end of the video). I focused on games in this video because they are probably one of the best use-case examples for this given their sheer size and the frequency of new ones being downloaded. Maybe I have caching on the brain after my suggested UNRAID improvement, but this seems like a handrolic caching approach to having current programs available on the fastest storage - eg its read caching for VMs. Another reason a real caching solution would be nice would be to make this transparent. What does handrolic mean? I think what you're trying to say in a more sophisticated tongue is that this seems like a rather manual process to handle storage tiering, and I absolutely agree that it is very manual. Longer term, we can look into ways to automate read-caching, but that is a much larger project. Tom commented on this in your thread about caching with respect to how bcache works and it's current limitations which prevent us from implementing. On a slightly more developed front, you can see a tiered structure of caching/backup that stretches to the separate desktop, with files cached to local SSD, tiered to UNRAID SSD and thence to UNRAID array. With a tolerable write-back structure your working desktop gets semi-backed-up to the array in real time - with old files migrating to only being on the UNRAID share, until needed. How would you get the data from the physical desktop PC to unRAID in the manner you are speaking. Also, what's different from this and using something like Crashplan or Sync? The same concept, but with Android phones/tablets allows apps/data to be essentially infinite. SMB connections with Android are possible, not sure about install to a transparent share/fake SD card... Get what you're saying here, but what happens when you're mobile and now to launch these applications you have to connect to a VPN or expose your SMB shares to the Internet? Interesting idea, but not sure if its really practical. Quote Link to comment
MyKroFt Posted June 19, 2015 Share Posted June 19, 2015 What is your suggestion for Blizzard games - as they detect and will no longer run/install from a network share.... Almost up to 65G of blizz data..... SSD is getting tight.... Thanks Myk Quote Link to comment
JonathanM Posted June 19, 2015 Share Posted June 19, 2015 What is your suggestion for Blizzard games - as they detect and will no longer run/install from a network share... Myk Can they detect a symbolic link? I've fooled a couple windows apps into using locations they normally won't with links. Quote Link to comment
jonp Posted June 19, 2015 Author Share Posted June 19, 2015 What is your suggestion for Blizzard games - as they detect and will no longer run/install from a network share... Myk Can they detect a symbolic link? I've fooled a couple windows apps into using locations they normally won't with links. That is exactly what I was going to suggest. The same procedure I showed for Origin could be attempted with Blizzard, although I haven't actually tested that one myself yet. If you try it, please report back with your results!! Quote Link to comment
Frank1940 Posted June 19, 2015 Share Posted June 19, 2015 jonp ---- You have put together several very interesting and informative videos recently. I have just search the main website (not the forum) for these videos. I did not find them there. I think you should make a dedicated section for these videos where that section is prominently displayed so that all users and potential customers will not miss seeing them. A two minute video can present a feature or how-to-do-this with much more clarity and impact then ten pages in a users manual! 2 Quote Link to comment
JonathanM Posted June 19, 2015 Share Posted June 19, 2015 What is your suggestion for Blizzard games - as they detect and will no longer run/install from a network share.... Almost up to 65G of blizz data..... SSD is getting tight.... Thanks Myk Also, keep in mind that the demo video is showing a KVM windows install that is running on the same hardware as the storage array, so latency and throughput are pretty much the best they possibly can be for a network link, since the traffic is really never leaving the box. If you are talking about running your data from one physical machine to another you may experience pauses and other latency that makes the experience a little less than desirable. It's definitely going to act much differently than your local SSD. Some games may suffer more than others, really depends how they are written and how much RAM your gaming box has. Quote Link to comment
MyKroFt Posted June 19, 2015 Share Posted June 19, 2015 What is your suggestion for Blizzard games - as they detect and will no longer run/install from a network share... Myk Can they detect a symbolic link? I've fooled a couple windows apps into using locations they normally won't with links. That is exactly what I was going to suggest. The same procedure I showed for Origin could be attempted with Blizzard, although I haven't actually tested that one myself yet. If you try it, please report back with your results!! Nope - sees thru the sym link - am trying a vhd thru the sym link so - so far so good, will update when its finished moving files over and I test from the launcher Myk Quote Link to comment
jonp Posted June 19, 2015 Author Share Posted June 19, 2015 Vhd through the symlink? Not sure I understand you. Quote Link to comment
jonp Posted June 19, 2015 Author Share Posted June 19, 2015 jonp ---- You have put together several very interesting and informative videos recently. I have just search the main website (not the forum) for these videos. I did not find them there. I think you should make a dedicated section for these videos where that section is prominently displayed so that all users and potential customers will not miss seeing them. A two minute video can present a feature or how-to-do-this with much more clarity and impact then ten pages in a users manual! Thanks Frank! The four videos that I classify as "guides" are actually now listed on the getting started page of the website under support. I'm working on numerous updates to the website to weave in the video content, but these exercises take some time. Quote Link to comment
MyKroFt Posted June 19, 2015 Share Posted June 19, 2015 Vhd through the symlink? Not sure I understand you. \ virtual hard disk - i will tru it \\tower\Games\Blizzard.vhd as well to see if the sym link is not required. Myk Quote Link to comment
jonp Posted June 19, 2015 Author Share Posted June 19, 2015 Totally lost. If you symlink to the vhd, how do you tell Blizzard to install a game to the vhd? Quote Link to comment
JonathanM Posted June 19, 2015 Share Posted June 19, 2015 Totally lost. If you symlink to the vhd, how do you tell Blizzard to install a game to the vhd? Maybe this will help? http://www.howtogeek.com/51174/mount-and-unmount-a-vhd-file-in-windows-explorer-via-a-right-click/ Quote Link to comment
jonp Posted June 19, 2015 Author Share Posted June 19, 2015 Totally lost. If you symlink to the vhd, how do you tell Blizzard to install a game to the vhd? Maybe this will help? http://www.howtogeek.com/51174/mount-and-unmount-a-vhd-file-in-windows-explorer-via-a-right-click/ Definitely. So then I'm guessing the VHD was created in windows but directed to be stored on a user share over SMB. Now mykroft is going to mount the vhd from inside his VM and then install the game to it that way. Is that right? Quote Link to comment
sane Posted June 19, 2015 Share Posted June 19, 2015 Well, to be fair, games are applications in the eyes of Windows, but I understand what you're asking. The reality is that some will work, some won't, and some will come with limitations (similar to what I mentioned about Origin and resuming downloads near the end of the video). I focused on games in this video because they are probably one of the best use-case examples for this given their sheer size and the frequency of new ones being downloaded. True enough, just I got bored of games a while back, whereas apps and VMs is a more interesting issue to me. VM on the SSD and apps on the array seems viable. What does handrolic mean? Sorry, technical term from the UK going back to WWII - you have 'hydraulic' lift, say, but when it's not working, or you have just lashed something together, you might have a 'handrolic' solution - eg a hand crank. Means doing it by hand, with more effort, since the automated mechanism isn't working/doesn't exist. I think what you're trying to say in a more sophisticated tongue is that this seems like a rather manual process to handle storage tiering, and I absolutely agree that it is very manual. Longer term, we can look into ways to automate read-caching, but that is a much larger project. Tom commented on this in your thread about caching with respect to how bcache works and it's current limitations which prevent us from implementing. Yep, just I got queried there about why you might ever need read caching - and this is effectively read-caching - so I feel justified in saying there's a benefit. How would you get the data from the physical desktop PC to unRAID in the manner you are speaking. Also, what's different from this and using something like Crashplan or Sync? Thought comes from Windows having caching of slow hard disk with fast SSD. It's not a big jump to see caching of NAS shared directory with fast SSD on the desktop, and thence if there were caching between the SSD and array ending up with a tiered structure. Get what you're saying here, but what happens when you're mobile and now to launch these applications you have to connect to a VPN or expose your SMB shares to the Internet? Interesting idea, but not sure if its really practical. You need the tunnel, but again, its not really that distant from the cloud type solutions out there. This is more spitballing, but you can kind of see how it could work, with caveats. Quote Link to comment
jonp Posted June 19, 2015 Author Share Posted June 19, 2015 Well, to be fair, games are applications in the eyes of Windows, but I understand what you're asking. The reality is that some will work, some won't, and some will come with limitations (similar to what I mentioned about Origin and resuming downloads near the end of the video). I focused on games in this video because they are probably one of the best use-case examples for this given their sheer size and the frequency of new ones being downloaded. True enough, just I got bored of games a while back, whereas apps and VMs is a more interesting issue to me. VM on the SSD and apps on the array seems viable. What does handrolic mean? Sorry, technical term from the UK going back to WWII - you have 'hydraulic' lift, say, but when it's not working, or you have just lashed something together, you might have a 'handrolic' solution - eg a hand crank. Means doing it by hand, with more effort, since the automated mechanism isn't working/doesn't exist. I think what you're trying to say in a more sophisticated tongue is that this seems like a rather manual process to handle storage tiering, and I absolutely agree that it is very manual. Longer term, we can look into ways to automate read-caching, but that is a much larger project. Tom commented on this in your thread about caching with respect to how bcache works and it's current limitations which prevent us from implementing. Yep, just I got queried there about why you might ever need read caching - and this is effectively read-caching - so I feel justified in saying there's a benefit. How would you get the data from the physical desktop PC to unRAID in the manner you are speaking. Also, what's different from this and using something like Crashplan or Sync? Thought comes from Windows having caching of slow hard disk with fast SSD. It's not a big jump to see caching of NAS shared directory with fast SSD on the desktop, and thence if there were caching between the SSD and array ending up with a tiered structure. Get what you're saying here, but what happens when you're mobile and now to launch these applications you have to connect to a VPN or expose your SMB shares to the Internet? Interesting idea, but not sure if its really practical. You need the tunnel, but again, its not really that distant from the cloud type solutions out there. This is more spitballing, but you can kind of see how it could work, with caveats. Definitely food for thought. I think you have some interesting ideas for caching, but to really get a block level caching function implemented would be a monumental amount of effort due a number of factors (some of which TomM mentioned in the roadmap thread). Shorter term objectives can be to simply eliminate the need to use the command line like I did in the video in order to move games around. That's much more doable in less time. Quote Link to comment
MyKroFt Posted June 19, 2015 Share Posted June 19, 2015 Totally lost. If you symlink to the vhd, how do you tell Blizzard to install a game to the vhd? Maybe this will help? http://www.howtogeek.com/51174/mount-and-unmount-a-vhd-file-in-windows-explorer-via-a-right-click/ Definitely. So then I'm guessing the VHD was created in windows but directed to be stored on a user share over SMB. Now mykroft is going to mount the vhd from inside his VM and then install the game to it that way. Is that right? Thanks what i am trying Myk Quote Link to comment
kal Posted June 19, 2015 Share Posted June 19, 2015 I only really see this as a plausible solution when the disks are kept local (as you demonstrate with the virtual games machine, with passthroughed gpu) - mapping across the network I can't imagine being a very performant route..? And gamers are all about the fps and load times.. Have a look at Steam Mover BTW, its widely used afaik Quote Link to comment
Frank1940 Posted June 19, 2015 Share Posted June 19, 2015 jonp ---- You have put together several very interesting and informative videos recently. I have just search the main website (not the forum) for these videos. I did not find them there. I think you should make a dedicated section for these videos where that section is prominently displayed so that all users and potential customers will not miss seeing them. A two minute video can present a feature or how-to-do-this with much more clarity and impact then ten pages in a users manual! Thanks Frank! The four videos that I classify as "guides" are actually now listed on the getting started page of the website under support. I'm working on numerous updates to the website to weave in the video content, but these exercises take some time. jonp ----- I went looking at the 'Main' page of the Website was if I was a new user or a potential customer. These are my thoughts. I would move that 'Getting Started' topic under support and make it a feature topic (like 'Support') on the Main page. If there is some 'rule' about the number of topics on that Main page, put either the 'Blog' or 'About' one under 'Support'. (If someone insists that this 'Getting Started' topic must be under 'Support', make the heading a link back to the 'Main' page one.) Consider using sub-headings under 'Video Guides' like 'Installation', 'Docker Containers','Virtual Machines', etc. Put these topics in the order that you think beginners are most likely to be looking for them. Consider putting the written Documentation (like the UnRAID 6 Manual) that you have prepared into a PDF format and make them available as downloads. It does not need to be super-fancy but a table of contents with page numbers and page numbers on the pages would be nice. I know that there are a lot of people who don't think that 'hard copy' is necessary but think of those people who prefer it! I currently have a two inch notebook filled with printouts of information on unRAID. There are probably ten to fifteen Post-it notes on pages that I need routinely. I write notes in the margins as reminders of the gotcha's or as a cross-reference to other information in my notebook. Plus, I can find things there much quicker there than I ever can on-line. (Think, two hundred pages of info vs 25,000,000,000 on the WWW!) Wow, this is rapidly getting off the topic. Frank Quote Link to comment
jonp Posted June 20, 2015 Author Share Posted June 20, 2015 I only really see this as a plausible solution when the disks are kept local (as you demonstrate with the virtual games machine, with passthroughed gpu) - mapping across the network I can't imagine being a very performant route..? It all depends on the connectivity and the storage devices you are using. I know there are folks out there who have a gaming PC separate from their NAS and store some of their games on the NAS. If using spinning disks, max network performance would be around 1gbps (or 125MB/s), which is actually not too far off from the peek speeds of a HDD. If using SSDs, doing this over the network is a waste as you'll be bottlenecked by the network. And gamers are all about the fps and load times.. Most definitely. The primary use case, as you indicated, is to do this where the gaming rig is just running as a VM with a GPU assigned to it. This way all IO operations are kept internal to the system, eliminating the copper network as a bottleneck. That said, FPS won't be impacted by your storage. FPS is all about GPU/CPU. Have a look at Steam Mover BTW, its widely used afaik Took a look at this earlier today. It's a fairly old tool and would essentially use the VM itself as a proxy for the move operations, adding unneeded overhead. In addition, as I showed in the video, I didn't need to use directory junctions for Steam. Instead, I just mapped a network drive. Lastly, Steam is only one of a few game app stores out there. Steam Mover wouldn't help me then with Origin, Blizzard, uPlay, or any others that I might be forgetting. Quote Link to comment
bonienl Posted June 21, 2015 Share Posted June 21, 2015 Your video guides is getting quite extensive - good job - I think they deserve a separate heading "Video Guides" under support on the LT web site. This will allow new users to quickly find them. Quote Link to comment
jonp Posted June 22, 2015 Author Share Posted June 22, 2015 Your video guides is getting quite extensive - good job - I think they deserve a separate heading "Video Guides" under support on the LT web site. This will allow new users to quickly find them. Been tossing the idea of a "lime tech TV" around. Could do a lot more than just guides here... Point is, I want to showcase the videos more prominently on the website. It's a goal for me to make that happen this week. Quote Link to comment
wirenut Posted June 22, 2015 Share Posted June 22, 2015 Stop tossing around the idea and catch it jonp. Great idea and great goal. Score. Can't wait to see it happen. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.