September 18, 201312 yr one for the Virtual server users. http://www.opendedup.org/ Havent tried it myself (currently downloading ) but looks promising.. SDFS is a Deduplication Based Filesystem SDFS is a filesystem for Windows and Linux designed to support the unique needs Virtual Environments and supports enhanced functionaliy for VMWare, Xen, and KVM SDFS Features include: Scalability - SDFS can dedup a Petabyte or more of data. Over 3TB per gig of memory at 128k chunk size. Speed - SDFS can perform deduplication/redup at line speed 350 MB/S+ VMWare support - Work with vms - can dedup at 4k block sized. This is required to dedup Virtual Machines effectively Flexible storage - deduplicated data can be stored locally, on the network across multiple nodes, or in the cloud. Inline and Batch Mode deduplication - The file system can dedup inline or periodically based on needs. This can be changed on the fly File and Folder Snapshot support - Support for file or folder level snapshots.
September 18, 201312 yr Looks interesting. I am looking for a cacheing file system. I want a local SSD on a Windows workstation to behave as a cache to unRaid. The Windows workstation will have a mid sized SSD for speed, that replicates to unRaid for capacity and backup. I really want the SSD to behave like a cache to the unRaid box. Have the entire unRaid appear virtually on the SSD, and be copied back and forth as necessary to optimize the workstation speed. Only current working files need to physically be on the SSD. The target application would be photo RAW image files that take gobs of space. We have millions on unRaid, but only need the current working files on the workstation. Doing everything on unRaid is much too slow. Doing everything locally on SSD is fine, for speed, but with limited space you need to manage the files that exist locally. We do this manually now with a 2tb Raid 0 (2 - 1tb spinners) array on the workstation. This is a bit faster, but SSD would be better yet, but the size limitations of SSD don't make this practical. We need a cacheing file system.
September 19, 201312 yr That is interesting. However, I'm not placing a high amount of importance on the deduplication of running VMs. I'd rather save this on the back end, for backups, off site backups and disaster recovery backups. To do this on live VMS? ... probably too much CPU overheard. .... Unless it's *really* clever.
September 20, 201312 yr I was all prepared to say this looks excellent but like all things dedupe the ram requirements are so high as to make it useless for home users..... and then i found this http://www.opendedup.org/administrators-guide#dsem Dedup Storage Engine Memory: The SDFS Filesystem itself uses about 2GB of RAM for internal processing and caching. For hash table caching and chunk storaged kernel memory is used. It is advisable to have enough memory to store the entire hashtable so that SDFS does not have to scan swap space or the file system to lookup hashes. For volumes that require high IOPs expand the memory edit the "-Xmx2g" within mount.sdfs or the startDSE.sh script to something better for your environment. For most environments -Xmx3g is sufficient. To calculate memory requirements keep in mind that each stored chunk takes up approximately 25 bytes of RAM. To calculate how much RAM you will need for a specific volume divide the volume size (in bytes) by the chunk size (in bytes) and multiply that times 25. Memory Requirements Calculation: (volume size/chunk size)*25 Data Whack this in excel and the number seem amazing to me... Disk 100 GB 102400 MB 104857600 KB 1.07374E+11 B Chunk 4 KB 4096 B Memory 655360000 B 640000 KB 625 MB 0.61 GB Can someone debunk this?
December 10, 201312 yr Yep, your math is right. I'm definitely not doing this since I have 5TB of work data I want to keep backed up, versioned, and deduped. I guess I'm sticking with crashplan until something better comes out.
December 11, 201312 yr Author It had potential, i did try it, its very quick to deploy mutiple versions of os, (10 servers take about 6mins, but as mentioned very hard on memory. I ended up removing it when it was taking 6Gb of memory out of 16Gb i have Sent from my GT-N7000 using Tapatalk
December 16, 201312 yr Do you remember how much data you had? I'm curious how accurate their algorithm is.
January 1, 201412 yr Author Sorry for delay in replying The windows 2008r2 was from a template so bog standard 40Gb partition. The 9 other servers when built where taking up small amounts of kb, approx 100Kb each. Sent from my GT-N7000 using Tapatalk
Archived
This topic is now archived and is closed to further replies.