On 5/27/2024 at 9:46 PM, VRx said:

Today's image have amazon cloud driver added.

Thanks, is that a different driver than the S3 driver that was around for some years now? Thats fine, too, just need to look into how to use that, as I configured for the S3 driver so far.

On 4/19/2024 at 11:15 AM, VRx said:

which version did You use?
postgresql vs sqlite3 and 11 or 13 maybe 15 ?

can confirm the Error from the French User.

I installed a fresh Unraid yesterday and just installed Bacula Server (default sqlite version) 

if you change the admin password to something else you will endup in a broken webpage with 

 

"Error code: 100 Message: Problem with connection to remote host. cURL error 0: ."

 

after login.

 

If you delete the docker and the remaining files in appdata and install it fresh again with the default settings (no password change) everything works well.

 

 

On 5/29/2024 at 10:10 PM, murkus said:

is that a different driver than the S3 driver that was around for some years now?

Yes it is different, I can find manual if You need, I found it previous for testing, but I'm not using cloud backup.

On 6/21/2024 at 7:18 PM, MikePiko said:

if you change the admin password to something else you will endup in a broken webpage with 

I must test it one more time. I thought I had already improved it

On 6/27/2024 at 10:30 PM, VRx said:

Yes it is different, I can find manual if You need, I found it previous for testing, but I'm not using cloud backup.

I must test it one more time. I thought I had already improved it

Thanks, I have updated to you updated version. Since I am not usin AWS S3, but my own minio instance I have set the HostName accordingly for the Cloud resource. It seems the driver is ignoring the HostName field. Can you confirm this?

`Invalid endpoint: https://s3..amazonaws.com Unknown error during program execvp`

 

To work for me it would need use as URL the Cloud > HostName I have configured.

I also played with the aws driver script in the plugins folder. It became clear that it is using the python script "aws" to access s3. I have my ca-certificates.crt mapped into the container (because I am using my internal CA to sign internal certificates), but python isn't using it. I found descriptions how this can be achieved, but it would be something that would better be initialized by the container (as pip isn't installed by default):

pip config set global.cert /etc/ssl/certs/ca-certificates.crt
conda config --set ssl_verify /etc/ssl/certs/ca-certificates.crt

Could you kindly add something like this, so that python will use the certificates at this path for SSL verification?
/etc/ssl/certs/ca-certificates.crt

Bottom line: there are at least 2 item now working atm:
(1) hand-over of the variables for the driver script (i.e. including the cloud > HostName)
(2) SSL verification using own CA certificates

Edited July 5, 20242 yr by murkus
added a lot more info

15 hours ago, murkus said:

`Invalid endpoint: https://s3..amazonaws.com Unknown error during program execvp`

Here is docs which I used with testing configuration.
https://docs.baculasystems.com/BEDedicatedBackupSolutions/StorageBackend/cloud/cloud-backup.html#cloud-accounts

You must do like a ceph endpoint in the exapmle.
I've found somewere that for other cloud storage than amazon You must use: UriStyle = Path
Default is Virtualhost.

 

 

3 hours ago, VRx said:

Here is docs which I used with testing configuration.
https://docs.baculasystems.com/BEDedicatedBackupSolutions/StorageBackend/cloud/cloud-backup.html#cloud-accounts

You must do like a ceph endpoint in the exapmle.
I've found somewere that for other cloud storage than amazon You must use: UriStyle = Path
Default is Virtualhost.

 

 

Thanks for your quick response. I was already using Path, not Virtualhost. The trick is to use BlobEndpoint in addition to HostName. Then it will actually use that endpoint.

 

If you get this:

`[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1006) Child exited with code 1`

(1) map the host file /etc/ssl/certs/ca-certificates.crt to the same location in the container
(2) set the container variable AWS_CA_BUNDLE to this exact path

Thanks for your continued effort, highly appreciated!

 

Edited July 6, 20241 yr by murkus
updated

So far I was successfully making backup copies to minIO S3 using copy jobs. Each volume is a folder on minIO and has an individual file called part (numbered) for each job in the volume. Works fine. The part files are all around 5GB large.

 

Now I was for the firs time making copies of volumes to a different pool on minIO. The upload of the part files failed consistently, but the error doesn't seem to make sense:

 

Quote

part.2     state=done    retry=1/10 size=193.0 GB duration=4994s msg=/opt/bacula/plugins/aws_cloud_driver upload minio01-vol-8388 part.2. upload failed: - to s3://bacula-tier3-01/minio01-vol-8388/part.2 An error occurred (InvalidArgument) when calling the UploadPart operation: Part number must be an integer between 1 and 10000, inclusive Child exited with code 2
bacula-sd JobId 23057: minio01-vol-8388/part.1     state=done     size=297 B duration=3s

 

the part number 2 clearly is between 1 and 10000. The problem would probably something else, but I have no idea what it is.

 

the part files that failed upload are large, around 200-400GB.

 

Any ideas?

 

Edited July 24, 20241 yr by murkus

On 7/24/2024 at 1:17 PM, murkus said:

So far I was successfully making backup copies to minIO S3 using copy jobs. Each volume is a folder on minIO and has an individual file called part (numbered) for each job in the volume. Works fine. The part files are all around 5GB large.

 

Now I was for the firs time making copies of volumes to a different pool on minIO. The upload of the part files failed consistently, but the error doesn't seem to make sense:

 

 

the part number 2 clearly is between 1 and 10000. The problem would probably something else, but I have no idea what it is.

 

the part files that failed upload are large, around 200-400GB.

 

Any ideas?

 

 

This seems to be a problem of object limits with minIO or the aws cli tool used by the Amazon cloud driver.

I was able to solve this by setting reasonable limits for the cloud device resource:

MaximumPartSize = 1000000000

MaximumFileSize = 1000000000

 

This will create volume folders in cache and S3 of size 10GB containing 13 parts with max 1GB size.

 

 

@murkus

I don't know if it is stiil relevant but I found some important info in bacula docs:
1. Maximum Part Size

This directive allows specification of the maximum size for each part of any volume written by the current device. Smaller part sizes will reduce restore costs, but will cause additional but small overhead to handle multiple parts. The maximum number of parts permitted per Cloud Volume is 524,288. The maximum size of any given part is approximately 17.5TB.
2. it is useful to set the Maximum File Size to a value smaller than or equal to the configured Maximum Part Size

BUT You have to remember that Bacula is not small app, and is was not designed to work inside container, this is my small project to backup content of virtual machines on homelab.
Maybe it will never work inside container like it could on physical server.

Hello,
i have to create a fresh install because i dropped my database.
Now i have the problem i start with a clean bacula directory and an empty database but bacula refuses to initialize the db:

==> Creating Storage daemon config...

==> Creating File daemon config...

==> Creating Bacula Director config...

==> Checking Bacularis config...

==> Checking DB...

==> It looks like You run bacula-server before. Configuration files are set, but database is empty. Make sure everything going right.

===== You can ommit this check by setting ENV 'check=0', but this will cause importing clean database, clean schema without history of backups done in the past

I set check=0 via -e parameter of container, still the same..

Edited June 30, 20251 yr by pk1057

@pk1057 try to backup configuration files (move to different place on disk), and try to run container without any configuration files.
If it go correctly You could stop Container, and move configuration files back on place, should work without problems (but one thing: user/password/name/host for database should be the same)

btw. there is posibility to restore database from bacula backup, but this is more difficult, and can take some time

Completely emptied the folder, still the same problem.

I ended up starting the container with --entrypoint=/bin/bash and had to modify the original startup script so that the check variable was no longer checked (the count seemed wrong to me, but I was too tired to check).

After that, I had a clean database.

I used BScan to read the last backup archive and restore the original database dump from tape. Now I could revert to the old version, and everything is working as intended again.

@pk1057 thx for info, I will check entrypoint

Are there any plans for an updatet Version with newer Bacularis or do we have to do it ourself ?
If we have to do it ourself, where can i find the docker file ?

I don't know if it makes sense.

This is a hobby project developed in my free time.

Bacularis delivers the application in Docker, including the entire Bacula stack, including the Postgresql database engine.

https://bacularis.app/doc/brief/installation.html#install-using-docker

You don't need to run the database in a separate container.

I don't think this is a good solution, as containers for different applications use their own database servers, which creates a significant resource overhead for multiple applications. I also believe that Docker wasn't designed to package entire application stacks into a single container.

This makes diagnostics and maintenance more difficult, but that's just my opinion, and I know that many users find this convenient.
Docker file is here https://github.com/vrx-666/bacula-server/tree/master if You want to play...

I totally agree with your opinion.
A container should be as lightweight as possible.