February 3, 201610 yr This is a continuation of the issue I posted about last month. For some reason the last 3 days, whenever the mover runs (currently randomly every 2-ish hours) the system is now trying to move the entire TVShows share. Whenever this happens, the cpu pegs at 100%, nothing is accessible (no Web UI, no file shares work, etc.). I can SSH into the system but the Diagnostics won't complete, I can't even get "shutdown -r now" to work. About the only way to bring the system back up is to hard power it off. I've attached the diagnostics after the reboot. I'm hoping someone can see a setting/config I have wrong that's causing the mover to think it needs to move the TVShows share. This system was using 5.x for 3 years without issue. Once I upgraded to v6 this started happening. For the moment I've scheduled the mover to run Weekly so I can hopefully troubleshoot this and resolve it. medialibrary-diagnostics-20160203-1038.zip
February 3, 201610 yr Community Expert This is a continuation of the issue I posted about last month. For some reason the last 3 days, whenever the mover runs (currently randomly every 2-ish hours) the system is now trying to move the entire TVShows share. Whenever this happens, the cpu pegs at 100%, nothing is accessible (no Web UI, no file shares work, etc.). I can SSH into the system but the Diagnostics won't complete, I can't even get "shutdown -r now" to work. About the only way to bring the system back up is to hard power it off. According to your share settings the TVShows share is configured to use the cache drive. Therefore if there are any files on the cache drive that belong to that share it would be perfectly normal for mover to want to move them to the array. Not quite sure what behaviour you are expecting?
February 3, 201610 yr Author The TVShows share is not located on the cache drive, it wouldn't fit. Yes I have the share set to use the cache drive, as is the Movies share. The issue is, when the mover starts, it's trying to move TVShows/. It's not moving a show on the cache into/onto the TVShows share, it's trying to migrate the entire 8TB+ TVShows share - where it's moving it I have no idea. When it does this, the entire system locks up for 48+ hours and is unusable. Well I'm guessing that's what it's trying to do based on what I've seen in the syslog just before the lockups happen. Here's a log capture from last month when it happened ---=== tail of syslog begins here ===--- root@MediaLibrary:~# tail -35 /var/log/syslog Jan 2 18:03:36 MediaLibrary avahi-daemon[1555]: server.c: Packet too short or invalid while reading response record. (Maybe a UTF-8 problem?) Jan 2 18:05:46 MediaLibrary kernel: md: sync done. time=43545sec Jan 2 18:05:46 MediaLibrary kernel: md: recovery thread sync completion status: 0 Jan 2 18:20:47 MediaLibrary kernel: mdcmd (102): spindown 0 Jan 2 19:39:53 MediaLibrary kernel: sdc: sdc1 Jan 2 19:39:59 MediaLibrary kernel: sdc: sdc1 Jan 2 20:00:01 MediaLibrary logger: mover started Jan 2 20:00:01 MediaLibrary logger: mover finished Jan 2 21:30:31 MediaLibrary avahi-daemon[1555]: server.c: Packet too short or invalid while reading response record. (Maybe a UTF-8 problem?) Jan 2 22:00:01 MediaLibrary logger: mover started Jan 2 22:00:01 MediaLibrary logger: mover finished Jan 2 22:31:24 MediaLibrary kernel: mdcmd (103): spindown 0 Jan 2 22:31:25 MediaLibrary kernel: mdcmd (104): spindown 3 Jan 2 22:34:02 MediaLibrary kernel: mdcmd (105): spindown 1 Jan 2 22:34:04 MediaLibrary kernel: mdcmd (106): spindown 4 Jan 2 22:34:13 MediaLibrary kernel: mdcmd (107): spindown 2 Jan 2 23:34:39 MediaLibrary sshd[22350]: Accepted none for root from 10.15.98.122 port 56608 ssh2 Jan 2 23:45:57 MediaLibrary kernel: mdcmd (108): spindown 4 Jan 2 23:45:59 MediaLibrary kernel: mdcmd (109): spindown 7 Jan 2 23:46:03 MediaLibrary emhttp: shcmd (121): /usr/sbin/hdparm -y /dev/sda &> /dev/null Jan 2 23:46:05 MediaLibrary kernel: mdcmd (110): spindown 0 Jan 2 23:46:06 MediaLibrary kernel: mdcmd (111): spindown 2 Jan 2 23:49:10 MediaLibrary sshd[25586]: Accepted none for root from 10.15.98.122 port 57095 ssh2 Jan 3 00:00:01 MediaLibrary logger: mover started Jan 3 00:00:01 MediaLibrary logger: moving "TVShows" Jan 3 00:03:00 MediaLibrary sshd[25586]: Received disconnect from 10.15.98.122: 11: disconnected by user Jan 3 00:03:07 MediaLibrary sshd[22350]: Received disconnect from 10.15.98.122: 11: disconnected by user Jan 3 01:35:42 MediaLibrary sshd[16819]: Accepted none for root from 10.15.98.122 port 60914 ssh2 Jan 3 01:40:06 MediaLibrary sshd[16819]: Received disconnect from 10.15.98.122: 11: disconnected by user Jan 3 04:40:01 MediaLibrary apcupsd[9212]: apcupsd exiting, signal 15 Jan 3 04:40:01 MediaLibrary apcupsd[9212]: apcupsd shutdown succeeded Jan 3 04:40:04 MediaLibrary apcupsd[24946]: apcupsd 3.14.13 (02 February 2015) slackware startup succeeded Jan 3 04:40:04 MediaLibrary apcupsd[24946]: NIS server startup succeeded Jan 3 05:12:56 MediaLibrary kernel: sdc: sdc1 Jan 3 11:22:15 MediaLibrary sshd[20327]: Accepted none for root from 10.15.98.122 port 62767 ssh2 ---=== tail of syslog ends here ===--- The only other thing that could be useful is I have a monthly parity check scheduled and this behavior seems to have started when the parity check started.
February 3, 201610 yr It does seem like something in TVShows is hanging mover (AKA rsync)... First question: is there a TVShows folder on your cache drive right now? If not... something weird is happening, if so... then I would assume it's trying to move that folder and hanging, not the whole share. Is there a way to increase the verbosity of mover and pipe the output to a log or something? I think that would help diagnose what is actually causing this to hang.
February 3, 201610 yr Community Expert Mover is a fairly simple script that uses rsync to move folders from /mnt/cache to /mnt/user0. In other words it only moves files from the cache drive, and I haven't seen anything in the logs you have posted to make me think mover is involved in your issue except possibly as just another thing running and using the drives in normal ways.
February 3, 201610 yr Author Excellent, thanks for the replies everyone, I appreciate the assistance. There were two files on the cache drive under the TVShows path. They were new as of this morning, but I think I'm following the logic of checking on the cache drive being clear. So, I manually ran the mover and it moved all of the TVShows files (and the folder itself) from the cache drive to the share. At this moment, the cache drive is empty. I verified that by SSH'ing in and using 'ls -la' on /mnt/cache I've re-enabled the mover so we can see if it hangs the system with the same syslog message about TVShows or not. I want the mover to execute via cron vs. manually so give me around an hour and I'll report back what happens. Thanks again
February 3, 201610 yr Author Update: The mover ran and didn't try to move the TVShows folder and so far the system hasn't hung up.
February 4, 201610 yr Author Something still isn't right. The system isn't at 100% CPU, the GUI is responsive and i can SSH in, but the mover has been running for 8.5 hours trying to move a 1GB file. I've attached another diagnostics file in case that helps anyone track down what's going on. Here's a quick snippet of the syslog Feb 4 00:00:01 MediaLibrary logger: mover started Feb 4 00:00:01 MediaLibrary logger: moving "TVShows" Feb 4 00:00:01 MediaLibrary logger: ./TVShows/Chicago P.D/Season 03/Chicago.P.D.-S03E13-Hit.Me-HDTV-720p.mkv Feb 4 00:00:01 MediaLibrary logger: .d..t...... ./ Feb 4 00:15:32 MediaLibrary kernel: mdcmd (61): spindown 0 Feb 4 00:15:32 MediaLibrary kernel: mdcmd (62): spindown 1 Feb 4 00:38:59 MediaLibrary kernel: mdcmd (63): spindown 7 Feb 4 02:18:43 MediaLibrary kernel: mdcmd (64): spindown 7 Feb 4 08:01:27 MediaLibrary avahi-daemon[17364]: server.c: Packet too short or invalid while reading response record. (Maybe a UTF-8 problem?) Feb 4 08:23:23 MediaLibrary kernel: mdcmd (65): spindown 0 Feb 4 08:23:23 MediaLibrary kernel: mdcmd (66): spindown 4 Feb 4 08:24:55 MediaLibrary sshd[20086]: Accepted none for root from 10.15.98.122 port 63304 ssh2 medialibrary-diagnostics-20160204-0828.zip
February 4, 201610 yr Community Expert On the Main page, are there any counts in the Errors column for any drives?
Archived
This topic is now archived and is closed to further replies.