October 27, 20178 yr My server is fairly busy, and I had a 1TB Evo 850 cache drive fail on me in less than a year. Admittedly it had spent several months having CCTV written to it at approx 5MB/s. CCTV is not written to the cache any more. Is this level of writes high, and expected? I am not downloading, I do have Crashplan running, but it is fully updated, and not much should be changing. I do have 5 windows 10 VMs running, but apart from HomeSeer home automation they aren't busy. Wondering if this is normal, and if not, how to pinpoint the culprit.
October 27, 20178 yr The number of read/writes as reported by the WebGUI is mostly meaningless, as similar devices can report very different values for the same activity, e.g., after a parity check. Check the SMART report and look at total_lbas_written, look again after one day usage and calculate how many bytes were written (multiply the difference by the sector size, usually 512 bytes)
October 27, 20178 yr Author Thanks JB. Here is current data for my new 1TB ssd which was put in about a month ago. 5TB written to it. If I assume 1TB was the inital build so 4TB per month. Samsung website say they are guaranteed for 5 years or 150TBW. So at this rate I can expect the SSD to run out of warranty in 3 years time. I'll check again in a couple of days to see if the difference matches up with the 4TB per month. Still seems high - works out at nearly 2MB per second continuous writing. Edited October 27, 20178 yr by al_uk
October 30, 20178 yr 850 EVO is a TLC SSD. TLC SSDs are a bad idea for an unRAID cache, they wear out far too fast.
December 28, 20178 yr Author To report back - the drive is nearly 3 months old now, and LBAs written is now 41808232224 which is 21TB In the last 21 hours it has written 0.45TB. I think this is way too high - but struggling to anrrow it down. Anyone any advice? Is there a logging version of iotop? I'd like to capture IO by process and destination for a 24 hour period. HellDiver, apologies I've only just seen your comment. Is there any particular make/model you'd recommend? Here are a few iotop screenshots. The Xeoma writes are (should be) to an Unassigned Device, not the cache. Hmm, I assume it writes directly to that device and not to the cache 1st.
December 29, 20178 yr do you have your plex and emby metadata on it and does your transcoding go through the ssd rather than ram? this might be an issue if those services are used heavily.
December 30, 20178 yr Author Plex and emby do not use a RAMDRIVE for transcoding based upon the last time I read a thread about it, which discouraged it. Having said that I do have 64GB and am using about half of it, so enough room for a ramdrive if necessary. The config directories for both are on the SSD. Has the recommendation changed? They're only used for about an hour per day in total, and probably half of that time the playback does not need transcoding. We use SageTV docker now but that records directly to the array. At this point I don't know whether it is something that is writing continuously at a lowish rate, or whether it is a scheduled process that kicks in and burst writes once a day for example.
December 30, 20178 yr Author I've dug into iotop a little and now have it running showing accumulating values. The following is a paste of the top few threads after running it for about 30 minutes. These entries started all at once incremented up to this value over the course of a few minutes, and then stopped. It happened the hour before as well. Cron is set to run hourly jobs at 47 minutes past the hour, so I don't think it is being triggered by that. How can I find out what it is? I am unsure whether this is showing a total write of 5.8GB or a total write of 5.8GB on each line ie. 60GB. TID PRIO USER DISK READ DISK WRITE> SWAPIN IO COMMAND 12604 be/4 root 11.92 M 5.77 G 0.00 % 0.05 % shfs /mnt/user -disks 15 51200000000 -o noatime,big_writes,allow_other -o remember=330 13043 be/4 root 7.54 M 5.77 G 0.00 % 0.03 % shfs /mnt/user -disks 15 51200000000 -o noatime,big_writes,allow_other -o remember=330 12714 be/4 root 1003.00 K 5.77 G 0.00 % 0.04 % shfs /mnt/user -disks 15 51200000000 -o noatime,big_writes,allow_other -o remember=330 12717 be/4 root 9.09 M 5.76 G 0.00 % 0.03 % shfs /mnt/user -disks 15 51200000000 -o noatime,big_writes,allow_other -o remember=330 12719 be/4 root 12.77 M 5.76 G 0.00 % 0.03 % shfs /mnt/user -disks 15 51200000000 -o noatime,big_writes,allow_other -o remember=330 12689 be/4 root 11.50 M 5.76 G 0.00 % 0.04 % shfs /mnt/user -disks 15 51200000000 -o noatime,big_writes,allow_other -o remember=330 13049 be/4 root 6.25 M 5.75 G 0.00 % 0.05 % shfs /mnt/user -disks 15 51200000000 -o noatime,big_writes,allow_other -o remember=330 12712 be/4 root 17.30 M 5.74 G 0.00 % 0.04 % shfs /mnt/user -disks 15 51200000000 -o noatime,big_writes,allow_other -o remember=330 13048 be/4 root 7.87 M 5.74 G 0.00 % 0.06 % shfs /mnt/user -disks 15 51200000000 -o noatime,big_writes,allow_other -o remember=330 12718 be/4 root 7.16 M 5.73 G 0.00 % 0.04 % shfs /mnt/user -disks 15 51200000000 -o noatime,big_writes,allow_other -o remember=330
December 30, 20178 yr Author Here's the latest, taken during the write which lasted around 10 minutes. This is now grouped into Processes not threads, so just one entry at the top. I think this is saying that it has written 33GB. The process finished at 57.3 GB. By comparison I can see the Xemoa process slowing incrementing consistently as it writes video files at around 4GB per hour. Total DISK READ : 915.70 K/s | Total DISK WRITE : 105.78 M/s Actual DISK READ: 904.71 K/s | Actual DISK WRITE: 10.94 M/s PID PRIO USER DISK READ DISK WRITE> SWAPIN IO COMMAND 13281 be/4 root 55.53 M 33.20 G 0.00 % 0.02 % shfs /mnt/user -disks 15 51200000000 -o noatime,big_writes,allow_other -o remember=330 1548 be/4 root 160.00 K 3.31 G 0.00 % 0.01 % xeoma -core -service -log -startdelay 5 10076 be/4 root 844.48 M 1259.86 M 0.00 % 0.03 % qemu-system-x86_64 -name guest=Ha2,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/~,vgamem_mb=16,max_outputs=1,bus=pci.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6 -msg timestamp=on 13187 be/4 root 2.80 M 423.52 M 0.00 % 0.04 % [btrfs-transacti] 31246 be/4 root 104.00 K 416.83 M 0.00 % 0.14 % qemu-system-x86_64 -name guest=Ha,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/d~ci,host=00:1b.0,id=hostdev0,bus=pci.0,addr=0x6 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x8 -msg timestamp=on 16438 be/0 root 20.58 M 269.14 M 0.00 % 0.03 % [loop0] 16461 be/4 root 432.00 K 180.48 M 0.00 % 0.03 % [btrfs-transacti] 8816 be/4 nobody 396.00 K 49.64 M 0.00 % 0.01 % mono-sgen --optimize=all /usr/lib/emby-server/bin/MediaBrowser.Server.Mono.exe -programdata /config -restartpath /usr/lib/emby-server/restart.sh 27912 be/4 root 904.00 K 40.93 M 0.00 % 0.01 % qemu-system-x86_64 -name guest=vDesk,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qem~tbus=1,hostaddr=2,id=hostdev0,bus=usb.0,port=1 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6 -msg timestamp=on 5840 be/4 nobody 4.17 M 19.66 M 0.00 % 0.00 % mono --debug NzbDrone.exe -nobrowser -data=/config 14997 be/4 root 828.00 K 16.11 M 0.00 % 0.03 % [kworker/u24:1] 3361 be/4 root 916.00 K 13.27 M 0.00 % 0.02 % [kworker/u24:4] 18707 be/4 root 740.00 K 7.42 M 0.00 % 0.07 % [kworker/u24:8] 15795 be/4 root 836.00 K 7.20 M 0.00 % 0.11 % [kworker/u24:0] 15637 be/4 root 396.00 K 6.51 M 0.00 % 0.00 % java -Dfile.encoding=UTF-8 -Dapp=CrashPlanService -DappBaseName=CrashPlan -Xms20m -Xmx1024m -Dsun.net.inetaddr.ttl=300 -Dn~alse -classpath /usr/local/crashplan/lib/com.backup42.desktop.jar:/usr/local/crashplan/lang com.backup42.service.CPService 18706 be/4 root 576.00 K 5.83 M 0.00 % 0.10 % [kworker/u24:3] 19970 be/4 root 128.00 K 4.45 M 0.00 % 0.03 % [kworker/u24:6] 19971 be/4 root 168.00 K 4.44 M 0.00 % 0.04 % [kworker/u24:7] 16472 be/4 root 0.00 B 4.02 M 0.00 % 0.00 % dockerd -p /var/run/docker.pid --storage-driver=btrfs --storage-driver=btrfs 19969 be/4 root 360.00 K 2.83 M 0.00 % 0.03 % [kworker/u24:5] 9861 be/4 aladmin 0.00 B 1660.00 K 0.00 % 0.00 % dropbox 22974 be/4 nobody 0.00 B 712.00 K 0.00 % 0.00 % java -Djava.awt.headless=true -Xmx768m -XX:+UseAdaptiveSizePolicy -XX:MaxGCPauseMillis=25 -XX:GCTimeRatio=24 -XX:+UseG1GC ~r:JARs/thetvdbapi-1.8.jar:JARs/tunerpreroll.jar:JARs/vecmath.jar:JARs/webui-3.3.jar sage.Sage 0 0 x sagetv Sage.properties 18708 be/4 root 76.00 K 656.00 K 0.00 % 0.03 % [kworker/u24:9] 24763 be/4 nobody 0.00 B 200.00 K 0.00 % 0.00 % Plex Media Server 15796 be/4 root 24.00 K 128.00 K 0.00 % 0.01 % [kworker/u24:2] 13186 be/4 root 0.00 B 48.00 K 0.00 % 0.00 % [btrfs-cleaner] 5310 be/4 nobody 84.00 K 36.00 K 0.00 % 0.00 % rslsync --nodaemon --config /config/sync.conf 18905 be/4 root 16.00 K 0.00 B 0.00 % 0.00 % syslog-ng -F -p /var/run/syslog-ng.pid --no-caps 11176 be/4 root 0.00 B 0.00 B 0.00 % 0.04 % emhttp 17729 be/4 root 584.00 K 0.00 B 0.00 % 0.00 % cadvisor -logtostderr 7593 be/4 root 13.88 M 0.00 B 0.00 % 0.00 % find /mnt/disk3/Backups -noleaf 22062 be/4 nobody 3.16 G 0.00 B 0.00 % 0.00 % mono Duplicati.Server.exe --webservice-interface=any --server-datafolder=/config 16025 be/4 root 128.00 K 0.00 B 0.00 % 0.00 % [xfsaild/sdj1] Edited December 30, 20178 yr by al_uk
December 31, 20178 yr Author Found the culprit - it was the Duplicati docker. Not sure why, because it was set to use HDD shares on the array not on the SSD cache. I'll recreate the docker.
January 11, 20188 yr Sorry, also just noticed this again. For SSD I try and stick to drives with high endurance ratings, and MLC flash. Which would mean Samsung 850 Pro, Sandisk Extreme Pro, Corsair XT, and some of ADATA's SSDs (I'm saying these because they're the ones I have in use that I know are reliable MLC SSDs). I'm currently using an Intel Pro SSD 5400s which is a TLC SSD, but it came in an Optiplex I stripped for parts, so it was effectively free so I don't care if it burns up in a few months. So far it's been fine. I'm only using it because it was on the top of the pile, and new, when I was building my server. I believe it has an endurance rating of 288TBW which is pretty high for a TLC SSD.
Archived
This topic is now archived and is closed to further replies.