[Support] cheesemarathons repo

[kavinci]

I am running into an issue with Minio and not sure how to resolve it. I am unable to upload more than a few hundred KB files. So far the largest successful file was 112KB. Anything bigger keeps failing. I think it has something to do with these errors that keeps repeating in the logs
 

API: SYSTEM
Time: Sat Jun 04 2022 02:22:17 GMT-0700 (Pacific Daylight Time)
DeploymentID: 3fc1b408-8f2b-462c-87fe-d859dda27ccd
RequestID: 
RemoteHost: 
UserAgent: 
Error: Storage resources are insufficient for the write operation .minio.sys/buckets/.usage-cache.bin (cmd.InsufficientWriteQuorum)

Backtrace: 
0: internal/logger/logger.go:278:logger.LogIf()
1: cmd/erasure-single-drive.go:3076:cmd.(*erasureSingle).nsScanner.func1()

API: SYSTEM
Time: Sat Jun 04 2022 02:25:18 GMT-0700 (Pacific Daylight Time)
DeploymentID: 3fc1b408-8f2b-462c-87fe-d859dda27ccd
RequestID: 
RemoteHost: 
UserAgent: 
Error: srcVolume: .minio.sys/tmp, srcPath: 52346e68-2ad4-4da9-bc05-534b035c485e, dstVolume: .minio.sys/buckets:, dstPath: public/.usage-cache.bin - error file not found (*errors.errorString)

Backtrace: 
0: internal/logger/logger.go:278:logger.LogIf()
1: cmd/xl-storage.go:2071:cmd.(*xlStorage).RenameData.func1()
2: cmd/xl-storage.go:2331:cmd.(*xlStorage).RenameData()
3: cmd/xl-storage-disk-id-check.go:356:cmd.(*xlStorageDiskIDCheck).RenameData()
4: cmd/erasure-object.go:707:cmd.renameData.func1()
5: internal/sync/errgroup/errgroup.go:123:errgroup.(*Group).Go.func1()

I have tried looking at permissions for the share and rebuilding the container a few times with no luck. My /data is pointed to /mnt/user/object_storage/ which is a just a share I created, nothing special. Anyone know how to fix this?

[cheesemarathon]

4 hours ago, kavinci said:

I am running into an issue with Minio and not sure how to resolve it. I am unable to upload more than a few hundred KB files. So far the largest successful file was 112KB. Anything bigger keeps failing. I think it has something to do with these errors that keeps repeating in the logs
 

API: SYSTEM
Time: Sat Jun 04 2022 02:22:17 GMT-0700 (Pacific Daylight Time)
DeploymentID: 3fc1b408-8f2b-462c-87fe-d859dda27ccd
RequestID: 
RemoteHost: 
UserAgent: 
Error: Storage resources are insufficient for the write operation .minio.sys/buckets/.usage-cache.bin (cmd.InsufficientWriteQuorum)

Backtrace: 
0: internal/logger/logger.go:278:logger.LogIf()
1: cmd/erasure-single-drive.go:3076:cmd.(*erasureSingle).nsScanner.func1()

API: SYSTEM
Time: Sat Jun 04 2022 02:25:18 GMT-0700 (Pacific Daylight Time)
DeploymentID: 3fc1b408-8f2b-462c-87fe-d859dda27ccd
RequestID: 
RemoteHost: 
UserAgent: 
Error: srcVolume: .minio.sys/tmp, srcPath: 52346e68-2ad4-4da9-bc05-534b035c485e, dstVolume: .minio.sys/buckets:, dstPath: public/.usage-cache.bin - error file not found (*errors.errorString)

Backtrace: 
0: internal/logger/logger.go:278:logger.LogIf()
1: cmd/xl-storage.go:2071:cmd.(*xlStorage).RenameData.func1()
2: cmd/xl-storage.go:2331:cmd.(*xlStorage).RenameData()
3: cmd/xl-storage-disk-id-check.go:356:cmd.(*xlStorageDiskIDCheck).RenameData()
4: cmd/erasure-object.go:707:cmd.renameData.func1()
5: internal/sync/errgroup/errgroup.go:123:errgroup.(*Group).Go.func1()

I have tried looking at permissions for the share and rebuilding the container a few times with no luck. My /data is pointed to /mnt/user/object_storage/ which is a just a share I created, nothing special. Anyone know how to fix this?

 Are you accessing mimio via a reverse proxy? You might have a max file upload size set there. Try uploading via the local IP address.

[kavinci]

On 6/3/2022 at 11:38 PM, cheesemarathon said:

 Are you accessing mimio via a reverse proxy? You might have a max file upload size set there. Try uploading via the local IP address.

No, I am hitting it directly from the IP of the unraid server.

[mykal.codes]

On 6/6/2022 at 8:12 AM, kavinci said:

No, I am hitting it directly from the IP of the unraid server.

I am also running into the exact same problem with my MINIO instance.

 

Did you ever find a solution? I'm going to be opening a ticket on the Minio github after work as well so if you didn't figure it out let me know and I'll reach out when I figure out what's going on.  

 

Thanks!

[kavinci]

57 minutes ago, mykal.codes said:

I am also running into the exact same problem with my MINIO instance.

 

Did you ever find a solution? I'm going to be opening a ticket on the Minio github after work as well so if you didn't figure it out let me know and I'll reach out when I figure out what's going on.  

 

Thanks!

I did not find a solution yet. I did however spin up Minio on my windows machine using Docker Desktop and WSL engine. That instance worked without any issues. That instance also didn't have any errors in the logs. My current assumption is that it is either the CheeseMarathon version or something to do with Unraid itself. 

I should also note that I got the CM version to work one time and one time only. The data config was pointed to the appdata location minio was installed to in unraid. This lead me to think that the container does not have sufficient privilege for the location I want so I tried giving creds to the root user in that directory which didn't work.

Based on the error codes it almost acts like there is not enough space for the file which is where the share config comes in. It might be reporting to the container the drive space and importing based on that report. I run a waterfall setup and share space across drives for most shares. I tried changing the config for the share I wanted to use and that didn't work either.

[cheesemarathon]

On 6/7/2022 at 11:47 PM, kavinci said:

I did not find a solution yet. I did however spin up Minio on my windows machine using Docker Desktop and WSL engine. That instance worked without any issues. That instance also didn't have any errors in the logs. My current assumption is that it is either the CheeseMarathon version or something to do with Unraid itself. 

I should also note that I got the CM version to work one time and one time only. The data config was pointed to the appdata location minio was installed to in unraid. This lead me to think that the container does not have sufficient privilege for the location I want so I tried giving creds to the root user in that directory which didn't work.

Based on the error codes it almost acts like there is not enough space for the file which is where the share config comes in. It might be reporting to the container the drive space and importing based on that report. I run a waterfall setup and share space across drives for most shares. I tried changing the config for the share I wanted to use and that didn't work either.

I run backups to my Minio instance every night without issue. So here is my minio config and my share config that i backup to.

 

 

As for and issue with my version, the only thing I have done is created the template. It uses the official https://hub.docker.com/r/minio/minio/ container image. So the issue lies somewhere with its use specifically on unRAID.

 

Failing spotting any issues with the above, I would look at raising an issue on the Minio github. I would stick as much of your logs as you can in a pastebin for the guys over there.

Edited June 13, 2022 by cheesemarathon

[mykal.codes]

1 hour ago, cheesemarathon said:

I run backups to my Minio instance every night without issue. So here is my minio config and my share config that i backup to.

 

 

As for and issue with my version, the only thing I have done is created the template. It uses the official https://hub.docker.com/r/minio/minio/ container image. So the issue lies somewhere with its use specifically on unRAID.

 

Failing spotting any issues with the above, I would look at raising an issue on the Minio github. I would stick as much of your logs as you can in a pastebin for the guys over there.

I did end up opening an issue with them on GitHub- the reason they gave when they closed the issue was that I'm not mounting to an XFS filesystem directly (I guess Unraid uses SHFS as a mounting mechanism for the Minio share- tbh that part of things is a bit over my head). 

 

That said, I'm using Minio with almost the the exact same settings that you're using above (different ports) so I'm a bit confused as to why I'm still having issues. 

Edited June 13, 2022 by mykal.codes
clarifying share information.

[obi-WAN-kenobi]

How do you make Ghost Container Work? 
IP:Container port gives this error: ERR_SSL_PROTOCOL_ERROR
And made public via traefik it does not connect (Host Error)

Also there are barely any files in the AppData folder. Not sure if this is the default or theres an issue somewhere

[cheesemarathon]

16 hours ago, obi-WAN-kenobi said:

How do you make Ghost Container Work? 
IP:Container port gives this error: ERR_SSL_PROTOCOL_ERROR
And made public via traefik it does not connect (Host Error)

Also there are barely any files in the AppData folder. Not sure if this is the default or theres an issue somewhere

 

Have you read the instructions at https://squishedmooo.com/how-to-setup-a-blog-with-ghost/

[UnusMundus]

Hey I've been running into an annoying issue with my Ghost installation.

 

in /ghost/public/cards.min.css. Everytime I change something in the file, the docker container regenerates a new file so I can't edit the css for some widgets in my installation because as fast as I restart the container it just reverts back to a completely unedited cards.min.css.

 

It also reverts the whole appdata folder back to a psecific user ownership and read/write status, and if I set owner to nobody and change the permissions that too resets on restart of the container.

 

Any idea what's going on?

[cheesemarathon]

42 minutes ago, UnusMundus said:

Hey I've been running into an annoying issue with my Ghost installation.

 

in /ghost/public/cards.min.css. Everytime I change something in the file, the docker container regenerates a new file so I can't edit the css for some widgets in my installation because as fast as I restart the container it just reverts back to a completely unedited cards.min.css.

 

It also reverts the whole appdata folder back to a psecific user ownership and read/write status, and if I set owner to nobody and change the permissions that too resets on restart of the container.

 

Any idea what's going on?

I imagine it’s overwriting your changes because those files are not meant to be edited by the user. All UI changes should be set in a theme.

 

As for changes to the ownership and permissions those will be set by the docker container which is built and provided by ghost. Is there a reason you need them to be different.

 

Id suggest reading the ghost docs and asking for further help over at the Ghost forum. I only made the unRAID template not the whole docker image so I’m far from an expert. Hopefully they will be able to clear things up further.

[kavinci]

On 6/13/2022 at 9:02 AM, mykal.codes said:

I did end up opening an issue with them on GitHub- the reason they gave when they closed the issue was that I'm not mounting to an XFS filesystem directly (I guess Unraid uses SHFS as a mounting mechanism for the Minio share- tbh that part of things is a bit over my head). 

 

That said, I'm using Minio with almost the the exact same settings that you're using above (different ports) so I'm a bit confused as to why I'm still having issues. 

 

I had a feeling it was something like this. I think it has to do with the pool configuration. I am running a waterfall across all drives and I think the management and reporting mechanism that might be the issue. I tried setting it but there must be something going on under the hood. The error I got kinda acts like there is but isn't enough space on the drive for larger files. It must be a side effect of this. What is CheeseMarathon's pool configuration?

[wunyee]

On 6/14/2022 at 12:02 AM, mykal.codes said:

I did end up opening an issue with them on GitHub- the reason they gave when they closed the issue was that I'm not mounting to an XFS filesystem directly (I guess Unraid uses SHFS as a mounting mechanism for the Minio share- tbh that part of things is a bit over my head). 

 

That said, I'm using Minio with almost the the exact same settings that you're using above (different ports) so I'm a bit confused as to why I'm still having issues. 

 

I also had the same issue ,did you find any solution?

[mykal.codes]

On 7/8/2022 at 1:48 AM, wunyee said:

 

I also had the same issue ,did you find any solution?

 

I did not find a solution. I ended up installing it on my RPi4 Kubernetes cluster, which uses an NFS share on my Unraid server; no matter how much I tinkered I couldn't get the docker container to work on Unraid. 

 

I think it probably has something to do with how my drives are setup not meshing with the docker image.  

[troian]

i solved it with vdisk

 

[cheesemarathon]

17 hours ago, troian said:

i solved it with vdisk

 

Does this fix the issues people have been having with Minio?

[leejohnholroyd]

I was having trouble with the diyHUE docker where all in app settings were lost on reboot.  The default container storage location is no longer correct and doesnt store anything.  The container mapping should now be /opt/hue-emulator/config

Confirmed by the diyHUE team that they changed to settings location a couple of versions back. 

[cheesemarathon]

On 8/28/2022 at 7:31 PM, leejohnholroyd said:

I was having trouble with the diyHUE docker where all in app settings were lost on reboot.  The default container storage location is no longer correct and doesnt store anything.  The container mapping should now be /opt/hue-emulator/config

Confirmed by the diyHUE team that they changed to settings location a couple of versions back. 

I have just pushed this fix to the template. Thank you for letting me know about the change!

Edited August 30, 2022 by cheesemarathon

[shawnngtq]

Using unraid 6.10.3, with latest minio, facing this error
 

API: SYSTEM()
Time: 09:50:54 UTC 09/02/2022
DeploymentID: 37851195-54a1-4973-924d-f412403d98ea
Error: write /data/.minio.sys/tmp/e6ec6910-320c-4b7a-9e3b-207675712ad7/xl.meta: invalid argument (*fs.PathError)
       6: internal/logger/logger.go:259:logger.LogIf()
       5: cmd/storage-errors.go:171:cmd.osErrToFileErr()
       4: cmd/xl-storage.go:2380:cmd.(*xlStorage).RenameData()
       3: cmd/xl-storage-disk-id-check.go:373:cmd.(*xlStorageDiskIDCheck).RenameData()
       2: cmd/erasure-object.go:721:cmd.renameData.func1()
       1: internal/sync/errgroup/errgroup.go:123:errgroup.(*Group).Go.func1()
Waiting for all MinIO sub-systems to be initialized.. possible cause (Unable to initialize config system: migrateConfigToMinioSys: Storage resources are insufficient for the write operation .minio.sys/config/config.json)
Exiting on signal: TERMINATED
ERROR Server startup canceled upon user request: Initializing sub-systems stopped gracefully context canceled

 

[myusuf3]

On 9/2/2022 at 6:04 AM, shawnngtq said:

Using unraid 6.10.3, with latest minio, facing this error
 

API: SYSTEM()
Time: 09:50:54 UTC 09/02/2022
DeploymentID: 37851195-54a1-4973-924d-f412403d98ea
Error: write /data/.minio.sys/tmp/e6ec6910-320c-4b7a-9e3b-207675712ad7/xl.meta: invalid argument (*fs.PathError)
       6: internal/logger/logger.go:259:logger.LogIf()
       5: cmd/storage-errors.go:171:cmd.osErrToFileErr()
       4: cmd/xl-storage.go:2380:cmd.(*xlStorage).RenameData()
       3: cmd/xl-storage-disk-id-check.go:373:cmd.(*xlStorageDiskIDCheck).RenameData()
       2: cmd/erasure-object.go:721:cmd.renameData.func1()
       1: internal/sync/errgroup/errgroup.go:123:errgroup.(*Group).Go.func1()
Waiting for all MinIO sub-systems to be initialized.. possible cause (Unable to initialize config system: migrateConfigToMinioSys: Storage resources are insufficient for the write operation .minio.sys/config/config.json)
Exiting on signal: TERMINATED
ERROR Server startup canceled upon user request: Initializing sub-systems stopped gracefully context canceled

 

 

 

 I am running into the same thing. Did you managed to fix it?

[shawnngtq]

No, need @cheesemarathon's help on this. Believe this is some filesystem type issue?

[sreknob]

@shawnngtq @myusuf3 @troian @mykal.codes @wunyee @kavinci @cheesemarathon

 

For the minio filesystem error, I resolved this by using directIO rather than shfs, which removes the fuse layer from the equation.

The only downside is that you need to use a single disk. Rest of settings are default.

/mnt/disk1/minio

 

 

So far uploaded a couple of GBs without issue

[cheesemarathon]

1 hour ago, sreknob said:

@shawnngtq @myusuf3 @troian @mykal.codes @wunyee @kavinci @cheesemarathon

 

For the minio filesystem error, I resolved this by using directIO rather than shfs, which removes the fuse layer from the equation.

The only downside is that you need to use a single disk. Rest of settings are default.

/mnt/disk1/minio

 

 

So far uploaded a couple of GBs without issue

Thanks for the heads up! It’s a shame that this issue still exists and that I can’t replicate it but at least we have a viable fix now.

[shawnngtq]

On 9/29/2022 at 12:02 AM, sreknob said:

@shawnngtq @myusuf3 @troian @mykal.codes @wunyee @kavinci @cheesemarathon

 

For the minio filesystem error, I resolved this by using directIO rather than shfs, which removes the fuse layer from the equation.

The only downside is that you need to use a single disk. Rest of settings are default.

/mnt/disk1/minio

 

 

So far uploaded a couple of GBs without issue

 

I verified this works. Thanks @sreknob!

 

That said, I worry the day using directIO will blow up unexpectedly, or worse, data corruption. Hopefully we will have actual fix that uses shfs 🤞

Edited October 1, 2022 by shawnngtq

[TheSnotRocket]

MinIO:

 

Curious on my /data path.

Some of ya'll are pointing to a single disk - but I have about 70tb across about 10 disks in my array.

 

When I try to start the container I'm getting: ERROR Unable to use the drive /data: Drive /data: found backend type fs, expected xl or xl-single: Invalid arguments specified

 

What would my solution be?  Am I SOL?

Edited November 1, 2022 by TheSnotRocket