MrChunky Posted October 20, 2020 Author Share Posted October 20, 2020 Its the right one, but it definitely doesn't have those fields as seen in the XML config, must be something on your side. Quote Link to comment
RealActorRob Posted October 20, 2020 Share Posted October 20, 2020 (edited) 18 minutes ago, MrChunky said: Its the right one, but it definitely doesn't have those fields as seen in the XML config, must be something on your side. Ok, can't imagine how they got there. Solved, at this point, as I have an edited copy that works. Edited October 20, 2020 by RealActorRob Quote Link to comment
Greyberry Posted January 8, 2021 Share Posted January 8, 2021 (edited) Hi, I am not able to get this template running: logs: 2021-01-08T20:07:01.846Z INFO Operator email {"Address": "censored"} 2021-01-08T20:07:01.846Z INFO Operator wallet {"Address": "censored"} Error: Error starting master database on storagenode: group: --- stat config/storage/blobs: no such file or directory --- stat config/storage/temp: no such file or directory --- stat config/storage/garbage: no such file or directory --- stat config/storage/trash: no such file or directory this is my xml: <Networking> <Mode>bridge</Mode> <Publish> <Port> <HostPort>28967</HostPort> <ContainerPort>28967</ContainerPort> <Protocol>tcp</Protocol> </Port> <Port> <HostPort>14002</HostPort> <ContainerPort>14002</ContainerPort> <Protocol>tcp</Protocol> </Port> </Publish> </Networking> <Data/> <Environment> <Variable> <Value>**censored**</Value> <Name>WALLET</Name> <Mode/> </Variable> <Variable> <Value>**censored**</Value> <Name>EMAIL</Name> <Mode/> </Variable> <Variable> <Value>**censored**:28967</Value> <Name>ADDRESS</Name> <Mode/> </Variable> <Variable> <Value>2TB</Value> <Name>STORAGE</Name> <Mode/> </Variable> <Variable> <Value/> <Name>BANDWIDTH</Name> <Mode/> </Variable> </Environment> <Labels/> <Config Name="TCP Port" Target="28967" Default="28967" Mode="tcp" Description="Default Storj V3 node port." Type="Port" Display="always" Required="true" Mask="false">28967</Config> <Config Name="Wallet Address" Target="WALLET" Default="" Mode="" Description="Your Payout address here." Type="Variable" Display="always" Required="true" Mask="false">**censored**</Config> <Config Name="Email address" Target="EMAIL" Default="" Mode="" Description="Email address used to sign Storj V3 node. (recommended)" Type="Variable" Display="always" Required="true" Mask="false">**censored**</Config> <Config Name="Internet Address" Target="ADDRESS" Default="domain.ddns.net:28967" Mode="" Description="Your ISP IP address:28967 (static IP) or dynamic DNS address here." Type="Variable" Display="always" Required="true" Mask="false">**censored**:28967</Config> <Config Name="Allocated Storage" Target="STORAGE" Default="2TB" Mode="" Description="A minimum of 500GB with no maximum of available space per node. Preferred minimum of 8TB and maximum of 24TB of available space per node." Type="Variable" Display="always" Required="true" Mask="false">2TB</Config> <Config Name="Dashboard Port" Target="14002" Default="14002" Mode="tcp" Description="The port to access web dashboard" Type="Port" Display="always" Required="true" Mask="false">14002</Config> <Config Name="Bandwidth" Target="BANDWIDTH" Default="" Mode="" Description="(OPTIONAL) Amount of bandwidth used, per month, to Storj network. According to Storj documentation, minimum is 2TB, recommended is 16+ TB, preferred is unlimited (e.g. 100000TB)." Type="Variable" Display="advanced" Required="false" Mask="false"/> what am I doing wrong? //edit: these are my "extra parameters" --mount type=bind,source="/mnt/user/appdata/storj/identity/storagenode/",destination=/app/identity --mount type=bind,source="/mnt/user/storj/",destination=/app/config I created the identitiy on Windows and copied the files over into the appdata folder. I also created the "storj" share. Edited January 8, 2021 by Greyberry Quote Link to comment
NVrEnough Posted January 10, 2021 Share Posted January 10, 2021 I have an almost identical setup with the exception of my "identity" not being located in "appdata" - instead I created a separate user share: --mount type=bind,source="/mnt/user/storjid/identity/storagenode/",destination=/app/identity --mount type=bind,source="/mnt/user/storj/",destination=/app/config On 1/8/2021 at 2:25 PM, Greyberry said: Error: Error starting master database on storagenode: group: --- stat config/storage/blobs: no such file or directory --- stat config/storage/temp: no such file or directory --- stat config/storage/garbage: no such file or directory --- stat config/storage/trash: no such file or directory Your errors are subfolders normally created at runtime from your extra parameter statement: On 1/8/2021 at 2:25 PM, Greyberry said: --mount type=bind,source="/mnt/user/storj/",destination=/app/config Could there be a permission error with your "/mnt/user/storj" user share causing this issue? Quote Link to comment
MrChunky Posted January 10, 2021 Author Share Posted January 10, 2021 I agree that nesting storage and identity folders this way is probably resulting in this error. Quote Link to comment
KrisMin Posted January 14, 2021 Share Posted January 14, 2021 Could I in theory change the "Allocated Storage": of the live node by taking it down for a sec adjusting the parameter and bring it back up? Or is the only option creating another node? Quote Link to comment
KrisMin Posted January 14, 2021 Share Posted January 14, 2021 On 1/10/2021 at 5:30 AM, NVrEnough said: I have an almost identical setup with the exception of my "identity" not being located in "appdata" - instead I created a separate user share: --mount type=bind,source="/mnt/user/storjid/identity/storagenode/",destination=/app/identity --mount type=bind,source="/mnt/user/storj/",destination=/app/config Your errors are subfolders normally created at runtime from your extra parameter statement: Could there be a permission error with your "/mnt/user/storj" user share causing this issue? I had a same issue. Apparently when running the first time -e SETUP=true --mount type=bind,source="/mnt/user/storj/<identityfolder>/",destination=/app/identity --mount type=bind,source="/mnt/user/storj/<datafolder>/",destination=/app/config is needed. 1 Quote Link to comment
NVrEnough Posted January 16, 2021 Share Posted January 16, 2021 On 1/14/2021 at 6:15 AM, KrisMin said: Could I in theory change the "Allocated Storage": of the live node by taking it down for a sec adjusting the parameter and bring it back up? Or is the only option creating another node? Its safest to just start another node. You could check Storj forum resize search for some help.... Quote Link to comment
benhall847 Posted January 17, 2021 Share Posted January 17, 2021 Hey guys! Thank you for putting this together! I'm trying to get a brand new node running. I'm having issues just getting this docker container to start though. I've created my identity and signed it via a windows machine then I moved all my identity files to /mnt/user/appdata/storj/identity/storagenode/ I updated the 'extra parameters' accordingly My data is to live in /mnt/user/storj/ I also updated the 'extra parameters' accordingly Here is the "extra parameters" I have & an image of my docker template Extra Parameters: --mount type=bind,source="/mnt/user/appdata/storj/identity/storagenode/",destination=/app/identity --mount type=bind,source="/mnt/user/storj/",destination=/app/config I'm unable to start the docker, it immediately turns off. My system logs are showing this error Is there something else I need to configure I'm missing here? Or am doing something wrong? Thank you guys for any support you can provide! Quote Link to comment
NVrEnough Posted January 17, 2021 Share Posted January 17, 2021 7 hours ago, benhall847 said: Extra Parameters: --mount type=bind,source="/mnt/user/appdata/storj/identity/storagenode/",destination=/app/identity --mount type=bind,source="/mnt/user/storj/",destination=/app/config Try moving your "identity" to the same drive your data is stored on. I first tried "appdata" as well and ran into an error. My extra parameters are: --stop-timeout 300 --mount type=bind,source="/mnt/user/storjid/identity/storagenode/",destination=/app/identity --mount type=bind,source="/mnt/user/storj/",destination=/app/config You didn't mention if you remembered to port forward TCP PORT 28967 in your router to your Storj container. Quote Link to comment
MrChunky Posted January 17, 2021 Author Share Posted January 17, 2021 (edited) 10 hours ago, benhall847 said: Hey guys! Thank you for putting this together! I'm trying to get a brand new node running. I'm having issues just getting this docker container to start though. I've created my identity and signed it via a windows machine then I moved all my identity files to /mnt/user/appdata/storj/identity/storagenode/ I updated the 'extra parameters' accordingly My data is to live in /mnt/user/storj/ I also updated the 'extra parameters' accordingly Here is the "extra parameters" I have & an image of my docker template Extra Parameters: --mount type=bind,source="/mnt/user/appdata/storj/identity/storagenode/",destination=/app/identity --mount type=bind,source="/mnt/user/storj/",destination=/app/config I'm unable to start the docker, it immediately turns off. My system logs are showing this error Is there something else I need to configure I'm missing here? Or am doing something wrong? Thank you guys for any support you can provide! Do not nest the identity and storage folders. e.g. the path should be /mnt/user/storj/storjidentity for the identity and /mnt/user/storj/storage for the storage folder. Here storj is a new share you should create. Make sure the port forwarding is done properly, as it looks like your issue is a network issue from the error that you showed. Edited January 17, 2021 by MrChunky Quote Link to comment
MrChunky Posted January 17, 2021 Author Share Posted January 17, 2021 2 hours ago, NVrEnough said: Try moving your "identity" to the same drive your data is stored on. I first tried "appdata" as well and ran into an error. I am not sure what you mean by this. You should not move the identity to the same folder as the data folder. And especially you should not use drives when operating on the unraid system, but rather user shares. Quote Link to comment
geeksheikh Posted January 17, 2021 Share Posted January 17, 2021 (edited) Anyone have any recommendations on optimizing this container. I'm ready to lend about 80TB across 4 containers...but...I have a few thoughts / concerns / questions. How to minimize pressure across provided disks (i've specified sepcific disks for each container and storj is the only thing that his them. I've disabled cache, any other recommendations to increase disk longevity? I also have the share set to "Fill-Up" to ensure minimum disk spin for new data acquisition. I just started the sync about 3 days ago but it looks like I'm averaging about 3GB/day of new data acquisition. I am allowing 6 cores to the docker container and 24TB with 1Gb/s up/down and unlimited bandwidth...any ideas on how to speed this up? Should I expect ingress to speed up as I gain rep on the network? Edited January 17, 2021 by srfnmnk Quote Link to comment
MrChunky Posted January 17, 2021 Author Share Posted January 17, 2021 (edited) 9 minutes ago, srfnmnk said: Anyone have any recommendations on optimizing this container. I'm ready to lend about 80TB across 4 containers...but...I have a few thoughts / concerns / questions. How to minimize pressure across provided disks (i've specified sepcific disks for each container and storj is the only thing that his them. I've disabled cache, any other recommendations to increase disk longevity? I just started the sync about 3 days ago but it looks like I'm averaging about 3GB/day of new data acquisition. I am allowing 6 cores to the docker container and 24TB with 1Gb/s up/down and unlimited bandwidth...any ideas on how to speed this up? Should I expect ingress to speed up as I gain rep on the network? 1. Not sure what you mean exactly by pressure on the disks. If you are talking about reducing the read/write cycles, then enabling cache is probably a good idea. It works fine for me and the amount of cache disk used is pretty stable with mover running daily. 2. The nodes you created will be vetted first, afterwards the traffic should increase significantly. You can read more here. In general you should check out the storj forum directly for such questions, there is a lot of information there. Edited January 17, 2021 by MrChunky 1 Quote Link to comment
geeksheikh Posted January 17, 2021 Share Posted January 17, 2021 Thanks @MrChunky -- I will do that. Also, for all of you asking good questions about how to set this up, I also had a bit of a challenge, I'll create a video when I set up my next container and provide a link so others can see how I get it running on Unraid with this Docker. One thing I've been doing differently is to actually complete the authorization from within the docker container. The authorize allows you to specify an identity dir location -- then you'll see that it actually creates a prefix_path/storagenode/<6 id files> here. This is what ultimately allowed me to figure out how to organize the prefixes and id files. I'll try to get the video out today or tomorrow but no promises. Quote Link to comment
geeksheikh Posted January 17, 2021 Share Posted January 17, 2021 46 minutes ago, MrChunky said: then enabling cache is probably a good idea Ah yes, i guess this does make good sense. whoops. Thanks. Is there a good way to set up auto-update for this one container (I don't want to auto-update all my containers) Quote Link to comment
benhall847 Posted January 17, 2021 Share Posted January 17, 2021 10 hours ago, MrChunky said: Do not nest the identity and storage folders. e.g. the path should be /mnt/user/storj/storjidentity for the identity and /mnt/user/storj/storage for the storage folder. Here storj is a new share you should create. Make sure the port forwarding is done properly, as it looks like your issue is a network issue from the error that you showed. Ok, I've updated my parameters as such --mount type=bind,source="/mnt/user/storj/identity",destination=/app/identity --mount type=bind,source="/mnt/user/storj/storage",destination=/app/config I confirmed I've port forwarded correctly. I'm still unable to start the container. It just immediately turns off. Is there some storj logs I should check? Quote Link to comment
geeksheikh Posted January 18, 2021 Share Posted January 18, 2021 2 hours ago, benhall847 said: It just immediately turns off. First time start up needs to use - `-e SETUP=true` It will not start then either, but then you need to go back to the extra parameters and remove that again and start it back up. This is to provision the initial requirements. 1 1 Quote Link to comment
benhall847 Posted January 18, 2021 Share Posted January 18, 2021 47 minutes ago, srfnmnk said: First time start up needs to use - `-e SETUP=true` It will not start then either, but then you need to go back to the extra parameters and remove that again and start it back up. This is to provision the initial requirements. THANK YOU!!! Much appreciated Quote Link to comment
NVrEnough Posted January 18, 2021 Share Posted January 18, 2021 20 hours ago, MrChunky said: I am not sure what you mean by this. You should not move the identity to the same folder as the data folder. And especially you should not use drives when operating on the unraid system, but rather user shares. Good point, I should have been clearer. By "drive" I assumed one would automatically use an Unraid "share" - but that may not always be the case. My setup has separate unraid "shares" for storj identity and data. No other "shares" are located on the "drives" containing storj data. Sorry for the confusion. After further research many small node operators don't recommend having local parity/redundancy as the network does that already. I may try something with unassigned devices, if possible, just not sure yet. I also couldn't get the container to start/validate/initialize...whatever. Then I moved my identity share off my appdata cache pool share thinking the "bind mount point" was causing an issue... guess it was the "setup=true" command that I NEVER used that was really the problem... Somehow it still worked. Guess it was dumb luck. Quote Link to comment
KrisMin Posted January 18, 2021 Share Posted January 18, 2021 Just to clarify what I wrote earlier When you run with a: -e SETUP=true for the first time, you will see from the log that the setup was done and container exits. After that you need to remove this parameter and start again. If you mounted your identity and data folders correctly, then you should be good to go and your front end dashboard should now be available from "your-unraid-pc-ip:14002". Happy hddmining! On the sidenote: fresh miners data accumulation rates are extremely slow right now. Hopefully accelerates at least 10 times when the node gets vetted. Quote Link to comment
geeksheikh Posted January 18, 2021 Share Posted January 18, 2021 (edited) Hey folks. After digging into this, the combination of authorizing your identity and getting the keys and signed certs in the proper location is quite challenging. I created a video to help clarify some of the confusing parts. I'm not sure if the video helps or makes it worse but if you're stuck, perhaps you can see the way I got it to work and get over the hump. YT VIDEO P.S. I realize you don't have to authorize from within the container but there are challenges authorizing outside of the container as well (i.e. unraid only has a root user (typically) and you probably don't want the identity binary creating hidden files in your /root/.local. Signing it on another ubuntu machine is probably the easiest but if you don't have that this is maybe the easiest way to go. Edited January 18, 2021 by srfnmnk Quote Link to comment
KrisMin Posted January 19, 2021 Share Posted January 19, 2021 (edited) Cool vid. I see you had some struggles, but got them sorted. Posted below your vid as well that: You can do all cli commands on the local terminal. Just point the container to the identity certs and data directories and its good to go. Also there is no need to turn off the cash disc. The space warning is misleading because the node only sees your cache drive and not beyond. Your storage space should fill up just fine (eventually, because the start is really slow). And one more thing, in case some of you don't know yet. Running several Storj nodes from a same IP unfortunately does not get you a multiplied ingress. Storj throttles down big storage providers that way. They filter IP subnets and ingress data is divided between all the nodes in the subnet. Also this goes for a vetting process time. If you start two new nodes on a same day, they get vetted pretty much 2 times slower than a single node would. This is why I am currently holding back the start of my third node. Edited January 19, 2021 by KrisMin 1 Quote Link to comment
geeksheikh Posted January 20, 2021 Share Posted January 20, 2021 19 hours ago, KrisMin said: Cool vid. I see you had some struggles, but got them sorted. Posted below your vid as well that: You can do all cli commands on the local terminal. Just point the container to the identity certs and data directories and its good to go. Also there is no need to turn off the cash disc. The space warning is misleading because the node only sees your cache drive and not beyond. Your storage space should fill up just fine (eventually, because the start is really slow). And one more thing, in case some of you don't know yet. Running several Storj nodes from a same IP unfortunately does not get you a multiplied ingress. Storj throttles down big storage providers that way. They filter IP subnets and ingress data is divided between all the nodes in the subnet. Also this goes for a vetting process time. If you start two new nodes on a same day, they get vetted pretty much 2 times slower than a single node would. This is why I am currently holding back the start of my third node. Thanks for the pointers -- it was a struggle -- primarily because I didn't want to use the unraid server root to generate and authorize identities. I did know that multiple storage nodes could limit ingress/egress but I DID NOT realize it would double vetting time. Thanks for that info. My thought was that assuming there is sufficient heterogeneous data, storj would still send me enough to populate two nodes. I'll shut down the second and do some testing once the first one fills up (which may never happen (24TB could take a while lol). Why is 24TB the max? Is that still a recommendation or is a single, larger node ok? Quote Link to comment
KrisMin Posted January 21, 2021 Share Posted January 21, 2021 (edited) Of what I have read, there is no hard cap on the storage size per node and the number is just a strong suggestion from the dev team. Someone in the forum did some math with hes years of statistics data and came up with a conlcusion that a theoretical limit would be aound 40TB of per IP subnet if IP filters work like we think they do. That number is based on hes statistics that a ~5% of all data gets trashed and therefor a pool of >40TB would never fill up, because 40TB would be the line in which point ingress = trashed. Of course if in the future an average Ingress rate increases, then the max pool size would also be larger. On our case (running nodes on docker), I would keep a single node size less or equal to 24TB, because it's less risky. If one node gets a bad reputation for some reason, then you still have other good ones to compensate and you could easily increase their size if needed. I started two identical nodes running on the same machine and the same network. Somehow one of them got uptime warning (from one us. sattelite) and lowered its uptime score, even tho they are identical in terms of availability and uptime. How did that happen, I have no idea. Must be some kind of bug. If I see this happening again, then I'll investigate it some more. Hopefully it was just once. Edited January 21, 2021 by KrisMin Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.