July 8, 201015 yr just fyi, I noticed how virtualbox uses two "info" files: I use exactly the same system (text file with the latest version numbers for stable and beta) for commercial applications I sell.
July 9, 201015 yr Why have a central website for the *metadata* about the packages which you need people to submit and upload..and not have them put the actual package file there too? This will only lead to out of date / broken links being presented by the central website (without constant review). Whereas if the package is stored there you at least know it's available even if it's not maintained / original author has moved on from unraid / authors webspace is no longer valid. You're both centralising and decentralising at the same time, surely not the best approach? You would also need some sort of hash check provided by the centralised metadata to ensure the package you download from the 3rd party site has not been maliciously tampered with. You would likely still need this even if you were hosting the package centrally but you at least have more of a guarantee it has not been arbitrarily changed.
July 9, 201015 yr Author Part of this is management, the other part is cost. Space and traffic cost money. Are members willing to continually contribute monetarily? Would Limetech fund the effort?
July 9, 201015 yr Why have a central website for the *metadata* about the packages which you need people to submit and upload..and not have them put the actual package file there too? Of course the packages can be hosted there too... nothing says they can't. As a dev, however, I prefer to host it where I choose. Plus this method allows a dev to simply change his info file on his own host, and not have to log in to the master config server each time he wants to push out an update. Alternatively, the master config server can fetch and retain a copy of each package, in case the dev's site goes dark. This will only lead to out of date / broken links being presented by the central website (without constant review). Whereas if the package is stored there you at least know it's available even if it's not maintained / original author has moved on from unraid / authors webspace is no longer valid. No, the nightly build will check that each package file is still there, and the hash matches. If not, that package is removed from the master config. You're both centralising and decentralising at the same time, surely not the best approach? The point is flexibility, and allowing devs to host their stuff wherever they want. You would also need some sort of hash check provided by the centralised metadata to ensure the package you download from the 3rd party site has not been maliciously tampered with. You would likely still need this even if you were hosting the package centrally but you at least have more of a guarantee it has not been arbitrarily changed. Read further up in this tread... I include hash checking. Part of this is management, the other part is cost. Space and traffic cost money. Are members willing to continually contribute monetarily? Would Limetech fund the effort? I have a server closet, co-hosted with a colleague with unlimited bandwidth and storage, served by a T3. The T3 is shared with other hosts, so speed is not blazing fast, but it is pretty much unlimited capacity. I'll volunteer to write the php backend. We need a mirror. We need to design the file formats, file names, and layouts of the data. We need the package manager frontend.
July 9, 201015 yr No, the nightly build will check that each package file is still there, and the hash matches. If not, that package is removed from the master config. Which means we end up with a package vanishing completely overnight. I see your points but I've used too many distributions / Operating systems that rely on this methodology (gentoo and freebsd ports spring to mind) for their package management and the number of times a package hasn't installed because the source file is no longer available for whatever reason is high. Keeping it centrally means it's should always be there - unless of course the author specifically asks it to be taken down for whatever reason or the central site is down - but then you wouldn't get the metadata either. I'm clearly in the minority with this opinion, so I won't interfere with the debate further but I'd also happily donate disk space / bandwidth / server resource. I think with the number of tech savvy people on the forum that part of it will be an abundance of riches.
July 9, 201015 yr Which means we end up with a package vanishing completely overnight. Again, no. A missing package file or unreachable server can be handled with an e-mail to the developer, but the package is not necessarily immediately take out of the master config. It can stay there for any period of time we want, and regularly check to see if the dev's files are accessible. And there is no reason the package management client can't accept a URL pasted into it to go and get a .info file and act on it on an ad-hoc basis even if that package is not in the master config.
July 9, 201015 yr Again, no. I can only work with the info you gave on the previous page, which said it would be removed! : No, the nightly build will check that each package file is still there, and the hash matches. If not, that package is removed from the master config. A missing package file or unreachable server can be handled with an e-mail to the developer, but the package is not necessarily immediately take out of the master config. It can stay there for any period of time we want, and regularly check to see if the dev's files are accessible. But in the meantime clients are still being told it's there - but failing when trying to fetch it. Shrug..this can be made as complicated as you like. Or you can have a simple web resource with everything in one place. Again it's clear we fundamentally disagree but it seems the workarounds and clauses you're putting in are making this much more complicated than it needs to be. Does slackware actually support 'remote' package repositories natively? I know from my experience of using it 10 odd years ago it didn't. But I'm not so sure these days. I suspect we'd be having a very different conversation if unraid was built around RedHat or Debian.
July 9, 201015 yr Author If we use google code for a plugin project or a developer repository (I have both) then it diminishes the responsibility of the developer and the site going dark. What if we go with the decentralized packages and if it doesn;t work, then we cache and centralize them. I like the idea of a centralized information repository access point. I'm concerned of the burden in the storage and added management for both the repository and the developer. Furthermore, if the package management client can access any URL, info file adhoc, then decentralized or centralized works at the client side.
July 9, 201015 yr What if we go with the decentralized packages and if it doesn;t work, then we cache and centralize them. Or do both and offer a list of mirrors in the central metadata - dev site (if available) first and then the locally cached copy second? This could also allow for scaling out the number of mirrors to distribute burden on hosting if it becomes a concern.
July 9, 201015 yr Author Again, no. I can only work with the info you gave on the previous page, which said it would be removed! : No, the nightly build will check that each package file is still there, and the hash matches. If not, that package is removed from the master config. I thought the conditions were. 1. The file is still there. 2. The hash matches. If the hash does not match, do you want to allow it to be installed? If the file is not there, do you want it to be presented? Perhaps if the file is inaccessible for a predetermined period, then it is deleted from the master config. A missing package file or unreachable server can be handled with an e-mail to the developer, but the package is not necessarily immediately take out of the master config. It can stay there for any period of time we want, and regularly check to see if the dev's files are accessible. But in the meantime clients are still being told it's there - but failing when trying to fetch it. Shrug..this can be made as complicated as you like. Or you can have a simple web resource with everything in one place. Again it's clear we fundamentally disagree but it seems the workarounds and clauses you're putting in are making this much more complicated than it needs to be. I don't think it's that complicated. PHP can open a resource anywhere. so if we see this does not work, then we change the repository to cache the files. Even that can be automated at the repository end. Does slackware actually support 'remote' package repositories natively? I know from my experience of using it 10 odd years ago it didn't. But I'm not so sure these days. I suspect we'd be having a very different conversation if unraid was built around RedHat or Debian. No, but wget and installpkg can do the basics as long as you have an url to access. RPM is much more complicated vs a slackware package. I have not even gotten into Debian packages, (but I will this weekend for my XBMC/unRAID experiment).
July 9, 201015 yr Author What if we go with the decentralized packages and if it doesn;t work, then we cache and centralize them. Or do both and offer a list of mirrors in the central metadata - dev site (if available) first and then the locally cached copy second? This could also allow for scaling out the number of mirrors to distribute burden on hosting if it becomes a concern. If the file format is set to handle multiple URLS like mirrors then the basic code will be there. it's just determining of it's cached on the master configuration server.
July 9, 201015 yr I can only work with the info you gave on the previous page, which said it would be removed! : I did, but I spoke imprecisely. it *can* be removed or retained, depending on what logic you want to use. Stick an "unavailable" icon in front of it, but leave it in the list? Keep it for 48 hours, then delete it? Actually handling a package that can't be downloaded would be up to the package management client, when it tries to pull a package, if it fails the client needs to appraise the user of that and handle it. If the hash does not match, do you want to allow it to be installed? Should be up to the user.... the client has to handle it.
July 9, 201015 yr it's just determining of it's cached on the master configuration server. Do it by default. The master server by definition of containing the metadata knows where to find the package. Easy cronjob to cache every package each night. I thought the conditions were. 1. The file is still there. 2. The hash matches. If the hash does not match, do you want to allow it to be installed? If the file is not there, do you want it to be presented? Hash does not match -> User queried. We may be seeing a lag between author updating plugin and the master hash being updated. A side effect of not being a definitive resource means we can't know. It could be malicious or it could be by design. If the file is not there no it shouldn't be presented / it doesn't really matter if it is as the client will error either way. But that's my entire issue as then the package would disappear. If centrally located (or cached which would be the compromise between both our approaches) this becomes less of a problem. RPM is much more complicated vs a slackware package. Perhaps though it's just learning curve and tools exist to automate. My point being rpm (more yum) and apt do all the client side functionality for us and have an easy way to set up remote repositories on a server. It makes the client side activities of installation, handling updates, hash checking and dependency tracking taken as a given. If nothing like that exists for slackware then it's not worth including in the discussion. Thanks for the update, I've had a quick look previously for something like them for slackware but it's nice to know I didn't miss something obvious!
July 9, 201015 yr Should be up to the user.... the client has to handle it. I think we just agreed on something
July 9, 201015 yr Perhaps though it's just learning curve and tools exist to automate. My point being rpm (more yum) and apt do all the client side functionality for us and have an easy way to set up remote repositories on a server. This is not rocket science. For the perspective of a dev: - Create a package.... host is somewhere or plan to host it on the master config server - Create a .info file.... host is somewhere or plan to host it on the master config server - Log into the master config server - Provide URL to the .info file (or upload it if you want it hosted on the master config server) and indicate if he wants the master config server to host the package file. - Server will do some checks, and tell you your .info file is valid and added to the master config. That's all the dev has to do. If the dev wants the master config server to host the .info file, he will have to re-upload the .info file whenever he makes a change. If the dev hosts the .info and package file himself. he never has to log into the master config server again, as the server will detect the .info change, and update everything. Of course, if he wants, he can log in to tell the server to update his .info file he just changed, or wait till the nightly job finds it.
July 9, 201015 yr I'll add my few comments for discussion. Once location for master list of plug-ins available. No argument... A single list that can be presented to a user in whatever fashion we like would be great. As mentioned, it is best as a parse-able file (with "title", "description", "version", "path to actual package", "files in the package", "dependencies", etc) As far as presentation to users, that will evolve over time. The program(s) that perform that task just needs to be able to read the master-list. One repository, vs. distributed This comes down to control and access. Many developers, myself included, wish to retain control so it is easy to deploy fixes, new versions, etc. We may already have version control in place (As I do, under google.code and SVN). To move it for the sake of having it in one physical location does not add any value to unRAID users. The programs downloading and installing could care less about where the links point. (I have no issue with a mirror of a repository, in case a owner moves on and loses interest) as stated, the use of google.code makes availability over time less of an issue. I have some experience in writing the code used by unMENU to update itself. I needed to be able to check and update individual files in an application, using checksums to validate their contents. (a single Slackware "installpkg" would not work well for its plug-ins and scripts as I have no need to download and re-install the entire unMENU set of files if only one changes. In fact, that would overwrite local changes and be the wrong methodology for it.) I could put together a single "Slackware .tgz" file to install and run unmenu_install itself. That might be possible now that Tom has added an "extras" folder where packages in it are automatically installed upon reboot. I cannot see each unMENU plug-in as a separate package though... although I can see a complete replacement of its "package-manager" files with the new scheme. Actually, under unRAID 5.0, I can see that unMENU pages might be converted into plug-ins for unRAID. (If not the code itself, the features) unMENU's whole purpose was to investigate improvements to the existing interface. It may actually have well served its purpose. In any case, unMENU has given me insight into what else will be needed by ANY package manager scheme. The need to assist the user in local configuration of the downloaded package. The configuration fields, default values, field-labels, and descriptions I added for individual packages in umMENU 1.3 are a big step in that direction. To be even better the values entered should have some basic validations applied. In unMENU, currently, they do not. Those configurable fields must be a part of any scheme developed. Joe L.
July 9, 201015 yr Perhaps though it's just learning curve and tools exist to automate. My point being rpm (more yum) and apt do all the client side functionality for us and have an easy way to set up remote repositories on a server. This is not rocket science. For the perspective of a dev: - Create a package.... host is somewhere or plan to host it on the master config server - Create a .info file.... host is somewhere or plan to host it on the master config server - Log into the master config server - Provide URL to the .info file (or upload it if you want it hosted on the master config server) and indicate if he wants the master config server to host the package file. - Server will do some checks, and tell you your .info file is valid and added to the master config. That's all the dev has to do. If the dev wants the master config server to host the .info file, he will have to re-upload the .info file whenever he makes a change. If the dev hosts the .info and package file himself. he never has to log into the master config server again, as the server will detect the .info change, and update everything. Of course, if he wants, he can log in to tell the server to update his .info file he just changed, or wait till the nightly job finds it. Sounds workable to me. Joe L.
July 9, 201015 yr Perhaps though it's just learning curve and tools exist to automate. My point being rpm (more yum) and apt do all the client side functionality for us and have an easy way to set up remote repositories on a server. This is not rocket science. No of course not, but if a mechanism similar to rpm/yum and apt already exist in slackware it would be re-inventing the wheel somewhat. As I queried and WeeboTech confirmed there isn't and it's not!
July 9, 201015 yr I cannot see each unMENU plug-in as a separate package though I agree. I was using the term generically, and not necessarily as a Slackware .tgz package file. I was envisioning the .info file containing the necessary information as to how to do the update. And nothing prevents any application from having it's own updating or "phone-home" system built into it. I do this with a firefox extension I wrote... it can be updated via the regular plugin system, but it also phones home to get some configuration and data updates that happen frequently... much like adaware.
July 9, 201015 yr The need to assist the user in local configuration of the downloaded package. Of course. Look at how Firefox add-ins are handled. The interface has a way to install, remove, and run the configuration program for each add-in. So the unraid plugin manager has to be able to invoke the configuration for a plugin. This should be much easier than with unmenu since you will have a real web server and php. So the plugin manager has to: - pull down the master .conf file from a host (or mirror) - have its own configuration options (such as default master.conf location) - show you available plugins - show you the plugins you have installed - check for updates to plugins - add, disable, and delete plugins - run a plugin's configuration script - allow the user to manually enter a URL to (or upload) a .info file in order to manually add a plugin that is not in the master .config file.
July 9, 201015 yr Perhaps though it's just learning curve and tools exist to automate. My point being rpm (more yum) and apt do all the client side functionality for us and have an easy way to set up remote repositories on a server. This is not rocket science. No of course not, but if a mechanism similar to rpm/yum and apt already exist in slackware it would be re-inventing the wheel somewhat. As I queried and WeeboTech confirmed there isn't and it's not! There is. Slackware's Package Management Tool is called pkgtool and slackpkg. It supports nearly all that's needed for this effort.
July 9, 201015 yr There is. Slackware's Package Management Tool is called pkgtool. It supports nearly all that's needed for this effort. No it doesn't. It is not amenable to novice end users. Please take a look at the NMT Community apps system... it's far and away a better system than the Slackware package tools.
July 9, 201015 yr I didn't know you wanted to build the Cadillac model first. I figured get something working first and then make it nicer as needed.
July 9, 201015 yr Author The need to assist the user in local configuration of the downloaded package. Of course. Look at how Firefox add-ins are handled. The interface has a way to install, remove, and run the configuration program for each add-in. So the unraid plugin manager has to be able to invoke the configuration for a plugin. This should be much easier than with unmenu since you will have a real web server and php. So the plugin manager has to: - pull down the master .conf file from a host (or mirror) - have its own configuration options (such as default master.conf location) - show you available plugins - show you the plugins you have installed - check for updates to plugins - add, disable, and delete plugins - run a plugin's configuration script - allow the user to manually enter a URL to (or upload) a .info file in order to manually add a plugin that is not in the master .config file. This is all sounding like the android market. Is there a buck to be made? <GRIN>
July 9, 201015 yr There is. Slackware's Package Management Tool is called pkgtool and slackpkg. It supports nearly all that's needed for this effort. Taken outwith the context of this topic as I don't want to fan the fire - thanks for this info, very interesting. Off for a google now
Archived
This topic is now archived and is closed to further replies.