Redxd

G'day all, I've had this issue every since I tried out Portainer a few years back. For some reason ever since then my network interface name for all of my dockers shows has the UID instead of the name that's set in the docker-compose file which is 'eth1.2' I remember deep diving into the docker config files on /boot/ back when it happened but could never find the fix. I think this also might be causing my dockers to not auto start after a reboot/dirty shutdown due to the UID changing. I have to compose down them all then bring them back up for them to all work.

I've always done what you've shown there in my docker-compose.yml file and it worked previously. I always had weird issues where once you set it, and the image was built you could never change it no matter how many times you changed the URL itself. I've now just created two dockers from scratch with the label set correctly and the images haven't shown. Unsure exactly what's happening.

I just changed my docker disk format from BTRFS to XFS due to heaps of errors in my unraid log showing up. Quick google showed a rebuild of the disk was the easiest way to resolve (ignore) it. I stopped the docker service, chose an XFS disk format instead and started it again. I have a user made bridged docker network where all of my dockers have access to so they are able to send logs to my log collector even if they don't have an external IP. Everything was all good up until I tried to start a PostgreSQL docker when there was already one started for another web service. Then I got the error Intrigued why changing only the format of the disk caused this I jumped back to the old disk to confirm I wasn't going crazy and both of the PostgreSQL dockers started up perfectly fine with no port conflicts. They are both being composed using the exact same docker stack, although each in a different stack. I then did a docker inspect on the same docker, composed in both the BTRFS and XFS disk formats, and here is the main difference (minus different ID's etc). The BTRFS 'Ports' section is empty, but the XFS section has the following. "Ports": { "5432/tcp": [ { "HostIp": "0.0.0.0", "HostPort": "5432" }, { "HostIp": "::", "HostPort": "5432" } ] }, Am I running into a bug? Or have I been using a bug with BTRFS and XFS is the correct way it should be?

I'd also been keen to see the imfile module included in the default OS so I can onforward logs like /var/log/ipmifan which generates temp readings from my motherboards IPMI and adjust the fans accordingly.

It's exactly what I did!

With Youtube URL's now changed, has anyone figured out how to download a full channel when 'Usernames' are no longer being accepted? {"level":"error","msg":"There was an error downloading the video: ERROR: u'' is not a valid URL. Set --default-search \"ytsearch\" (or run youtube-dl \"ytsearch:\" ) to search YouTube\n[youtube:user] destinws2: Downloading channel page\n[youtube:playlist] UU6107grRI4m0o2-emgoDnAA: Downloading webpage\n[download] Downloading playlist: Uploads from SmarterEveryDay\n[youtube:playlist] UU6107grRI4m0o2-emgoDnAA: Downloading page #1\nERROR: u'content_html'\n","time":"2020-07-27T13:50:46Z"} {"level":"info","msg":"returning response: {Error ERROR_DOWNLOADING There was an error downloading the target Download: DownloadVideo: ERROR: u'' is not a valid URL. Set --default-search \"ytsearch\" (or run youtube-dl \"ytsearch:\" ) to search YouTube\n[youtube:user] destinws2: Downloading channel page\n[youtube:playlist] UU6107grRI4m0o2-emgoDnAA: Downloading webpage\n[download] Downloading playlist: Uploads from SmarterEveryDay\n[youtube:playlist] UU6107grRI4m0o2-emgoDnAA: Downloading page #1\nERROR: u'content_html'\n}","time":"2020-07-27T13:50:46Z"}

I second this request. I have an X11 that has FAN1-7 and also FANAB

I've just updated the docker as well and i'm getting the same error message. Docker doesn't even start. usermod: no changes groupmod: GID '100' already exists {"level":"info","msg":"Initiating db","time":"2020-01-14T07:23:08Z"} {"level":"error","msg":"database.load: could not read file: json: cannot unmarshal string into Go struct field DownloadTarget.CheckingInterval of type int","time":"2020-01-14T07:23:08Z"} {"level":"fatal","msg":"\u003cnil\u003e","time":"2020-01-14T07:23:08Z"} I also removed the image completely and re-downloaded it just in case, still same issue. I checked the permissions on the json config files and they are exactly the same as what's set in the docker config.

I've just got back from holidays and the docker is working pretty good now. One thing I have started to notice is that every once and a while (maybe when it does it's check for new videos) the docker will consume 100% CPU on my server (As shown on the dashboard) and slow down everything else. This has happened before though while pre-clearing disks and it wasn't the CPU that was maxing it, it was the IO wait time increasing. It goes straight back to normal when i kill the auto-yt docker. What information can I provide to help diagnose it?

I just tried manually within the docker using the same command but changing the 'bestvideo[ext=mp4]' to just 'best'. This downloaded normal like any other time but it included video. I think when choosing the 'bestvideo' option it's just doing video and it may also be missing the audio part and then needing to join the two?

I'm able to get the whole channel to download now by choose the quality as MP4, it doesn't work when set to 'Any'. With MP4 selected the videos download in the background but my guess is the 'channel' dosn't get added to the bottom of the gui until all videos are downloaded. I have noticed that all the videos it does download don't have any audio. Nothing in the logs about it. {"level":"info","msg":"received a request to add a target","time":"2019-12-20T11:02:27Z"} {"level":"info","msg":"executing youtube-dl command: /usr/local/bin/youtube-dl -j --playlist-end 1 https://www.youtube.com/user/destinws2","time":"2019-12-20T11:02:27Z"} {"level":"info","msg":"DOWNLOAD: {https://www.youtube.com/user/destinws2 Video And Audio Uploads from SmarterEveryDay mp4 [] 12-20-2019 11:02:35 Channel %(uploader)s [%(channel_id)s]/%(upload_date)s %(title)s_[%(id)s].%(ext)s}","time":"2019-12-20T11:02:35Z"} {"level":"info","msg":"downloading video","time":"2019-12-20T11:02:35Z"} {"level":"info","msg":"{ youtube-dl --ignore-errors bestvideo[ext=mp4] downloads/%(uploader)s [%(channel_id)s]/%(upload_date)s %(title)s_[%(id)s].%(ext)s https://www.youtube.com/user/destinws2}","time":"2019-12-20T11:02:35Z"} {"level":"info","msg":"executing youtube-dl command: /usr/local/bin/youtube-dl --ignore-errors -f bestvideo[ext=mp4] -o downloads/%(uploader)s [%(channel_id)s]/%(upload_date)s %(title)s_[%(id)s].%(ext)s https://www.youtube.com/user/destinws2","time":"2019-12-20T11:02:35Z"} {"level":"info","msg":"DOWNLOAD: {https://www.youtube.com/user/destinws2 Video And Audio Uploads from SmarterEveryDay mp4 [] 12-20-2019 11:02:35 Channel %(uploader)s [%(channel_id)s]/%(upload_date)s %(title)s_[%(id)s].%(ext)s}","time":"2019-12-20T11:02:35Z"} {"level":"info","msg":"downloading video","time":"2019-12-20T11:02:35Z"} {"level":"info","msg":"{ youtube-dl --ignore-errors bestvideo[ext=mp4] downloads/%(uploader)s [%(channel_id)s]/%(upload_date)s %(title)s_[%(id)s].%(ext)s https://www.youtube.com/user/destinws2}","time":"2019-12-20T11:02:35Z"} {"level":"info","msg":"executing youtube-dl command: /usr/local/bin/youtube-dl --ignore-errors -f bestvideo[ext=mp4] -o downloads/%(uploader)s [%(channel_id)s]/%(upload_date)s %(title)s_[%(id)s].%(ext)s https://www.youtube.com/user/destinws2","time":"2019-12-20T11:02:35Z"}

I have tried a few different custom paths in troubleshooting this. With a /channels/ folder (created and not created) No channels folder so just / /Downloads/ but then in the logs it must already prefix Downloads show it was showing up as downloads/downloads/

i've tried multiple scenarios with the custom output path. Even the default it dosn't work.

This is the docker settings I have set. Downloads location pointing to my array and config file pointing to my cache directory.

No matter what i set the download folder path to be, or what customised path I set, I'm always getting the same error messages in the logs. I've tried a two different youtube channels but get the same error. {"level":"info","msg":"received a request to add a target","time":"2019-12-14T08:09:58Z"} {"level":"info","msg":"executing youtube-dl command: /usr/local/bin/youtube-dl -j --playlist-end 1 https://youtube.com/users/destinws2","time":"2019-12-14T08:09:58Z"} {"level":"error","msg":"From GetMetadata(): unexpected end of JSON input","time":"2019-12-14T08:10:02Z"} {"level":"info","msg":"adding {https://youtube.com/users/destinws2 Video And Audio any [] 12-14-2019 08:10:02 Channel /%(uploader)s_[UC2LVhJH_9cT2XKp0VAfsKOQ]/%(title)s_[%(id)s].%(ext)s%(uploader)s/video/%(title)s.%(ext)s} to DB","time":"2019-12-14T08:10:02Z"} {"level":"info","msg":"DOWNLOAD: {https://youtube.com/users/destinws2 Video And Audio any [] 12-14-2019 08:10:02 Channel /%(uploader)s_[UC2LVhJH_9cT2XKp0VAfsKOQ]/%(title)s_[%(id)s].%(ext)s%(uploader)s/video/%(title)s.%(ext)s}","time":"2019-12-14T08:10:02Z"} {"level":"info","msg":"downloading video","time":"2019-12-14T08:10:02Z"} {"level":"info","msg":"{ youtube-dl --ignore-errors bestvideo[ext=any] downloads/%(uploader)s_[UC2LVhJH_9cT2XKp0VAfsKOQ]/%(title)s_[%(id)s].%(ext)s%(uploader)s/video/%(title)s.%(ext)s https://youtube.com/users/destinws2}","time":"2019-12-14T08:10:02Z"} {"level":"info","msg":"executing youtube-dl command: /usr/local/bin/youtube-dl --ignore-errors -f bestvideo[ext=any] -o downloads/%(uploader)s_[UC2LVhJH_9cT2XKp0VAfsKOQ]/%(title)s_[%(id)s].%(ext)s%(uploader)s/video/%(title)s.%(ext)s https://youtube.com/users/destinws2","time":"2019-12-14T08:10:02Z"} {"level":"error","msg":"exit status 1/usr/local/bin/youtube-dl --ignore-errors -f bestvideo[ext=any] -o downloads/%(uploader)s_[UC2LVhJH_9cT2XKp0VAfsKOQ]/%(title)s_[%(id)s].%(ext)s%(uploader)s/video/%(title)s.%(ext)s https://youtube.com/users/destinws2","time":"2019-12-14T08:10:05Z"} {"level":"info","msg":"updating latest downloaded video id","time":"2019-12-14T08:10:05Z"} {"level":"info","msg":"latest downloaded video id updated successfully","time":"2019-12-14T08:10:05Z"} {"level":"info","msg":"updating download history","time":"2019-12-14T08:10:05Z"} {"level":"info","msg":"[{https://youtube.com/users/destinws2 Video And Audio any [] 12-14-2019 08:10:02 Channel /%(uploader)s_[UC2LVhJH_9cT2XKp0VAfsKOQ]/%(title)s_[%(id)s].%(ext)s%(uploader)s/video/%(title)s.%(ext)s}]","time":"2019-12-14T08:10:05Z"} {"level":"info","msg":"returning response: {Error ERROR_DOWNLOADING There was an error downloading the target Download: DownloadVideo: exit status 1}","time":"2019-12-14T08:10:05Z"} {"level":"info","msg":"got this data: {channels}","time":"2019-12-14T08:10:06Z"} {"level":"info","msg":"getting all channels from channels.json","time":"2019-12-14T08:10:06Z"} {"level":"info","msg":"successfully read all channels","time":"2019-12-14T08:10:06Z"} {"level":"info","msg":"[{https://youtube.com/users/destinws2 Video And Audio any [] 12-14-2019 08:10:02 Channel /%(uploader)s_[UC2LVhJH_9cT2XKp0VAfsKOQ]/%(title)s_[%(id)s].%(ext)s%(uploader)s/video/%(title)s.%(ext)s}]","time":"2019-12-14T08:10:06Z"} {"level":"info","msg":"got this data: {channels}","time":"2019-12-14T08:10:43Z"} {"level":"info","msg":"getting all channels from channels.json","time":"2019-12-14T08:10:43Z"} {"level":"info","msg":"successfully read all channels","time":"2019-12-14T08:10:43Z"} {"level":"info","msg":"[{https://youtube.com/users/destinws2 Video And Audio any [] 12-14-2019 08:10:02 Channel /%(uploader)s_[UC2LVhJH_9cT2XKp0VAfsKOQ]/%(title)s_[%(id)s].%(ext)s%(uploader)s/video/%(title)s.%(ext)s}]","time":"2019-12-14T08:10:43Z"}

For this drive and the other 12TB that successfully worked I was seeing easy 120MB/s. The drives were plugged into the same SAS card as all the other drives in the array. all the speeds I was seeing both read and write seems normal. It's just this one drive grinded everything to a halt once writing zeros. I've currently got it booted off a trial unraid instance on my main PC doing its preclear.

See attached tower-diagnostics-20191201-1919.zip

I just shucked two 12TB WD White's and only one seems to have preclear'd without any issues. I've tried the old Preclear plugin, then the preclear docker when I saw notes that the plugin wasn't working well with 6.7 and then the most up to date plugin once updates were made for the MBR issues others were receiving. This one 12TB drive keeps on slowing down my unraid to a crawl. Hogs all I/O. When using the docker if left on for a while it takes minutes to run the docker stop command. The pre-read seems to run fine but it's not until it starts writing zero's that 1-2 minutes in the dashboard shows all CPU's maxed out. iotop shows the dd process writing the zero's at a high % and another process labeled [kworker/u16:7+flush-8:32]. see attached screenshot. Logs don't show much. I'm unable to find how to turn debugging on based on what gfjardim said a few posts back. Here are the preclear plugin logs for the drive itself Dec 01 19:10:01 preclear_disk_8CJLTRWE_10041: Zeroing: dd if=/dev/zero of=/dev/sdc bs=2097152 seek=26138902528 count=11973999722496 conv=notrunc iflag=count_bytes,nocache,fullblock oflag=seek_bytes Dec 01 19:10:01 preclear_disk_8CJLTRWE_10041: Zeroing: dd pid [12894] Dec 01 19:12:50 preclear_disk_8CJLTRWE_10041: SIGTERM received, exiting... Dec 01 20:45:44 preclear_disk_8CJLTRWE_10404: Command: /usr/local/emhttp/plugins/preclear.disk/script/preclear_disk.sh --notify 2 --frequency 2 --cycles 1 --no-prompt /dev/sdc Dec 01 20:45:44 preclear_disk_8CJLTRWE_10404: Preclear Disk Version: 1.0.6 Dec 01 20:45:45 preclear_disk_8CJLTRWE_10404: S.M.A.R.T. info type: default Dec 01 20:45:45 preclear_disk_8CJLTRWE_10404: S.M.A.R.T. attrs type: default Dec 01 20:45:45 preclear_disk_8CJLTRWE_10404: Disk size: 12000138625024 Dec 01 20:45:45 preclear_disk_8CJLTRWE_10404: Disk blocks: 2929721344 Dec 01 20:45:45 preclear_disk_8CJLTRWE_10404: Blocks (512 byte): 23437770752 Dec 01 20:45:45 preclear_disk_8CJLTRWE_10404: Block size: 4096 Dec 01 20:45:45 preclear_disk_8CJLTRWE_10404: Start sector: 0 Dec 01 20:45:47 preclear_disk_8CJLTRWE_10404: Pre-Read: dd if=/dev/sdc of=/dev/null bs=2097152 skip=0 count=12000138625024 conv=notrunc iflag=nocache,count_bytes,skip_bytes Dec 01 20:47:40 preclear_disk_8CJLTRWE_10404: SIGTERM received, exiting... Dec 01 20:49:38 preclear_disk_8CJLTRWE_32581: Command: /usr/local/emhttp/plugins/preclear.disk/script/preclear_disk.sh --load-file /tmp/.preclear/8CJLTRWE.resume /dev/sdc Dec 01 20:49:38 preclear_disk_8CJLTRWE_32581: Preclear Disk Version: 1.0.6 Dec 01 20:49:38 preclear_disk_8CJLTRWE_32581: Restoring previous instance of preclear Dec 01 20:49:38 preclear_disk_8CJLTRWE_32581: S.M.A.R.T. info type: default Dec 01 20:49:38 preclear_disk_8CJLTRWE_32581: S.M.A.R.T. attrs type: default Dec 01 20:49:38 preclear_disk_8CJLTRWE_32581: Disk size: 12000138625024 Dec 01 20:49:38 preclear_disk_8CJLTRWE_32581: Disk blocks: 2929721344 Dec 01 20:49:38 preclear_disk_8CJLTRWE_32581: Blocks (512 byte): 23437770752 Dec 01 20:49:38 preclear_disk_8CJLTRWE_32581: Block size: 4096 Dec 01 20:49:38 preclear_disk_8CJLTRWE_32581: Start sector: 0 Dec 01 20:49:40 preclear_disk_8CJLTRWE_32581: Continuing disk read from byte 26449281024 Dec 01 20:49:40 preclear_disk_8CJLTRWE_32581: Pre-Read: dd if=/dev/sdc of=/dev/null bs=2097152 skip=26449281024 count=11973689344000 conv=notrunc iflag=nocache,count_bytes,skip_bytes Dec 01 21:11:45 preclear_disk_8CJLTRWE_32581: SIGTERM received, exiting... Dec 01 21:11:56 preclear_disk_8CJLTRWE_25713: Command: /usr/local/emhttp/plugins/preclear.disk/script/preclear_disk.sh --cycles 1 --skip-preread --no-prompt /dev/sdc Dec 01 21:11:56 preclear_disk_8CJLTRWE_25713: Preclear Disk Version: 1.0.6 Dec 01 21:11:56 preclear_disk_8CJLTRWE_25713: S.M.A.R.T. info type: default Dec 01 21:11:56 preclear_disk_8CJLTRWE_25713: S.M.A.R.T. attrs type: default Dec 01 21:11:56 preclear_disk_8CJLTRWE_25713: Disk size: 12000138625024 Dec 01 21:11:56 preclear_disk_8CJLTRWE_25713: Disk blocks: 2929721344 Dec 01 21:11:56 preclear_disk_8CJLTRWE_25713: Blocks (512 byte): 23437770752 Dec 01 21:11:56 preclear_disk_8CJLTRWE_25713: Block size: 4096 Dec 01 21:11:56 preclear_disk_8CJLTRWE_25713: Start sector: 0 Dec 01 21:11:58 preclear_disk_8CJLTRWE_25713: Zeroing: emptying the MBR. Dec 01 21:11:58 preclear_disk_8CJLTRWE_25713: Zeroing: dd if=/dev/zero of=/dev/sdc bs=2097152 seek=2097152 count=12000136527872 conv=notrunc iflag=count_bytes,nocache,fullblock oflag=seek_bytes Dec 01 21:11:58 preclear_disk_8CJLTRWE_25713: Zeroing: dd pid [28276] Dec 01 21:15:12 preclear_disk_8CJLTRWE_25713: SIGTERM received, exiting... And disk log information Nov 21 18:52:32 Tower kernel: sd 1:0:1:0: [sdc] 23437770752 512-byte logical blocks: (12.0 TB/10.9 TiB) Nov 21 18:52:32 Tower kernel: sd 1:0:1:0: [sdc] 4096-byte physical blocks Nov 21 18:52:32 Tower kernel: sd 1:0:1:0: [sdc] Write Protect is off Nov 21 18:52:32 Tower kernel: sd 1:0:1:0: [sdc] Mode Sense: 73 00 00 08 Nov 21 18:52:32 Tower kernel: sd 1:0:1:0: [sdc] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA Nov 21 18:52:32 Tower kernel: sd 1:0:1:0: [sdc] Attached SCSI disk Nov 21 18:52:58 Tower emhttpd: WDC_WD120EMAZ-11BLFA0_8CJLTRWE (sdc) 512 23437770752 Nov 28 21:35:27 Tower emhttpd: WDC_WD120EMAZ-11BLFA0_8CJLTRWE (sdc) 512 23437770752 Nov 28 21:35:48 Tower emhttpd: WDC_WD120EMAZ-11BLFA0_8CJLTRWE (sdc) 512 23437770752 Nov 28 21:39:09 Tower emhttpd: WDC_WD120EMAZ-11BLFA0_8CJLTRWE (sdc) 512 23437770752 Nov 28 21:39:42 Tower emhttpd: WDC_WD120EMAZ-11BLFA0_8CJLTRWE (sdc) 512 23437770752 Nov 30 09:05:32 Tower emhttpd: WDC_WD120EMAZ-11BLFA0_8CJLTRWE (sdc) 512 23437770752 Nov 30 09:07:45 Tower emhttpd: WDC_WD120EMAZ-11BLFA0_8CJLTRWE (sdc) 512 23437770752 Nov 30 09:07:57 Tower emhttpd: WDC_WD120EMAZ-11BLFA0_8CJLTRWE (sdc) 512 23437770752 Dec 1 19:34:59 Tower preclear_disk_8CJLTRWE[16263]: Command: /usr/local/emhttp/plugins/preclear.disk/script/preclear_disk.sh --cycles 1 --skip-preread --no-prompt /dev/sdc Dec 1 19:35:03 Tower preclear_disk_8CJLTRWE[16263]: Zeroing: dd if=/dev/zero of=/dev/sdc bs=2097152 seek=2097152 count=12000136527872 conv=notrunc iflag=count_bytes,nocache,fullblock oflag=seek_bytes Dec 1 19:39:59 Tower preclear_disk_8CJLTRWE[10041]: Command: /usr/local/emhttp/plugins/preclear.disk/script/preclear_disk.sh --load-file /tmp/.preclear/8CJLTRWE.resume /dev/sdc Dec 1 19:40:01 Tower preclear_disk_8CJLTRWE[10041]: Zeroing: dd if=/dev/zero of=/dev/sdc bs=2097152 seek=26138902528 count=11973999722496 conv=notrunc iflag=count_bytes,nocache,fullblock oflag=seek_bytes Dec 1 21:15:44 Tower preclear_disk_8CJLTRWE[10404]: Command: /usr/local/emhttp/plugins/preclear.disk/script/preclear_disk.sh --notify 2 --frequency 2 --cycles 1 --no-prompt /dev/sdc Dec 1 21:15:47 Tower preclear_disk_8CJLTRWE[10404]: Pre-Read: dd if=/dev/sdc of=/dev/null bs=2097152 skip=0 count=12000138625024 conv=notrunc iflag=nocache,count_bytes,skip_bytes Dec 1 21:19:38 Tower preclear_disk_8CJLTRWE[32581]: Command: /usr/local/emhttp/plugins/preclear.disk/script/preclear_disk.sh --load-file /tmp/.preclear/8CJLTRWE.resume /dev/sdc Dec 1 21:19:40 Tower preclear_disk_8CJLTRWE[32581]: Pre-Read: dd if=/dev/sdc of=/dev/null bs=2097152 skip=26449281024 count=11973689344000 conv=notrunc iflag=nocache,count_bytes,skip_bytes Dec 1 21:41:56 Tower preclear_disk_8CJLTRWE[25713]: Command: /usr/local/emhttp/plugins/preclear.disk/script/preclear_disk.sh --cycles 1 --skip-preread --no-prompt /dev/sdc Dec 1 21:41:58 Tower preclear_disk_8CJLTRWE[25713]: Zeroing: dd if=/dev/zero of=/dev/sdc bs=2097152 seek=2097152 count=12000136527872 conv=notrunc iflag=count_bytes,nocache,fullblock oflag=seek_bytes