Jump to content
Jcloud

[Support] QDirStat, Jcloud - cryptoCoin templates

321 posts in this topic Last Reply

Recommended Posts

17 hours ago, L0rdRaiden said:

In my json files for the storj containers I have 4004 for Storj2 and 4008 for Storj3. If you look at the storjshare status they are using those ports not 4000

 

What you define in the json file only affects the docker internally. What you are showing in the image above are the ports you have forwarded to the docker when you created it. To me it looks like you need to change the ports in the template configuration to what you want e.g. 4000-4003, 4004-4007 etc. (step 4 in what I posted above). If this is not fixing your problem, I would guess that there is some issue on your side unrelated to Storj...

Share this post


Link to post
12 hours ago, MrChunky said:

 

What you define in the json file only affects the docker internally. What you are showing in the image above are the ports you have forwarded to the docker when you created it. To me it looks like you need to change the ports in the template configuration to what you want e.g. 4000-4003, 4004-4007 etc. (step 4 in what I posted above). If this is not fixing your problem, I would guess that there is some issue on your side unrelated to Storj...

I did the setup with the bridges and I could replicate your result, thanks.

 

On the firewall do you open ports 4000, 4004, 4008, or the whole range from 4000 to 4011?

 

image.thumb.png.ee3b2748765002be360ec47fb239032f.png

 

Let see how it goes.

Edited by L0rdRaiden

Share this post


Link to post

What firewall?

 

Do you mean router?

 

You need to port forward all. 

Share this post


Link to post
4 hours ago, nuhll said:

What firewall?

 

Do you mean router?

 

You need to port forward all. 

 

Well I have pfsense, so I call it firewall.

Regarding the ports I have read the documentation and in principle if you are not using tunels just opening 4000 4004 4008 4012 and 4016 is enough.

Edited by L0rdRaiden

Share this post


Link to post

Ah, yeah, thats right, but i guess, at some time, that will infuence score, maybe?!

Share this post


Link to post
17 hours ago, L0rdRaiden said:

 

Well I have pfsense, so I call it firewall.

Regarding the ports I have read the documentation and in principle if you are not using tunels just opening 4000 4004 4008 4012 and 4016 is enough.

I have been looking at my firewall and it looks like the tunnel ports are indeed unused. Keeping the whole range open for now.

Share this post


Link to post

Wouldn't this docker implementation completely circumvent the limits storj imposed on the number of nodes being limited by the number of cores on one machine?

Edited by MrChunky

Share this post


Link to post

Im not sure, but it could be, atm i only have 2 nodes and 4 cores, so this shouldnt break that point. But when i see that other ppls have like 100+ nodes, i guess, its easy to go around that limit.... 

Share this post


Link to post

Okay guys, i added 10 more nodes, so you can fugg that cpu limit pretty easy with this solution.

 

Currently i have 11 nodes running (and you need different ports, i checked it). And they also get already some MBs.


Would you please add storjmonitor to your docker. Then this would be perfect. I tried running it extra, but i guess, it only works on LOCALHOST... so it needs to be run in docker.

 

https://github.com/calxibe/StorjMonitor

 

seems VERY easy:

Linux

  1. Clone/Copy this Repo to your local storage. Eg. git clone https://github.com/calxibe/StorjMonitor.git
  2. Change into the directory. cd StorjMonitor/
  3. Modify the install Script to executable. chmod +x storjMonitor-install.sh
  4. Run the Script! ./storjMonitor-install.sh
  5. Enter your API Key from the website into the line 10 in storjMonitor.js (var token = "YOUR-TOKEN-HERE").
  6. Execute the Monitor Script. ./storjMonitor.sh or via seperate screen screen -dmS StorjMonitor ./storjMonitor.sh

 

1. is the same thing as you probably alreayd do with storj

4. only need to be run on startup of docker

5. can be skipped, or lets say, edited in the config file directly

6. needs to be executed after the storj node startet

Edited by nuhll

Share this post


Link to post
3 hours ago, nuhll said:

Would you please add storjmonitor to your docker.

LOL It's not my docker, but I hear what you're saying.  I'll try and do something with-in the month, unless someone beats me to the punch. Life has been a distraction lately haven't had much time to work on projects.

 

3 hours ago, nuhll said:

seems VERY easy:

  HAHAHA! (emphasis on 'seems' mine) -- I'm sure your not wrong, once one know's how to run and slipstream containers. 

Share this post


Link to post

Sorry, i would help you, but i googled yesterday docker tutorial... and i dont find any usefull information on how to create dockers. 

 

I dont know where to start.

 

 

Share this post


Link to post
8 hours ago, nuhll said:

I dont know where to start.

 

I was the same way when I first started, now I'm like 10%, maybe 15%, "I know what I'm doing." ;) But, that's why I started this was to learn how to use, and work with these. 

There's actually I lot of good resources In our community forums -- I found them once but I don't think I bookmarked them. Doh! 

Third reading of your posted instructions, it does look easy. I think I can manage it - once I figure out a few of the steps that come before it. The project will definitely get some time this weekend. 

Share this post


Link to post
On 4/6/2018 at 7:08 PM, Jcloud said:

I was the same way when I first started, now I'm like 10%, maybe 15%, "I know what I'm doing." ;) But, that's why I started this was to learn how to use, and work with these. 

There's actually I lot of good resources In our community forums -- I found them once but I don't think I bookmarked them. Doh! 

Third reading of your posted instructions, it does look easy. I think I can manage it - once I figure out a few of the steps that come before it. The project will definitely get some time this weekend. 

 

I would really appreciate that too. If you are in too deep, you could possibly ask someone on the forums like aptalca, who has a lot of experience with dockers to look over it.

Share this post


Link to post
On 4/12/2018 at 3:22 AM, MrChunky said:

I would really appreciate that too. If you are in too deep, you could possibly ask someone on the forums like aptalca

I'm about waist deep at the moment. ;) It's on my general radar and I'll spend time and energy on it. I'm also on the Linuxserve.io discord so I've started making the hookups for asking for help in real-time. Really I just need to spend more time tinkering, and less time gaming. :DWhen I get the repo working probably have a few more options for working with multiple nodes and load them back up on updates. 

 

UPDATE:  I *think* I know what to do now, and starting to tinker with fake code. Then I'll move onto broken code, and finally working code. 

UPDATE2:  I know how to insert it into the container and make it work. Now to just figure out how to automate all of this - (I'm such a newb lol)

Also realized I borked my firewall configs, at some point, nuked settings - storjmonitor says I have open ports now.  . ..Woops. 

 

storjMonitorPrgrs.png

Edited by Jcloud

Share this post


Link to post
On 4/12/2018 at 11:52 AM, Jcloud said:

UPDATE:  I *think* I know what to do now, and starting to tinker with fake code. Then I'll move onto broken code, and finally working code. 

     I'm well into broken code. I know how to insert the monitor into the container; and I'll share that here since there are many interested parties. I know how to build the new container from repo, and it builds -- but doesn't run. \:(

I'm working on it, but for now I'm stumped.

 

For the In-term:  

How to Add StorjMonitor Into Existing Storj Container

Pre-requests:

  • Your Storjstat.com API-Key (this can be copy/paste from web page and terminal)
  • shell terminal 

STEP1  -  Get inside the container. Run the following command from your unRAID host:

docker exec -ti Storj /bin/sh

Your prompt should change to "/#"   ---   EXAMPLE:

root@HYDRA:/#
root@HYDRA:/# docker exec -ti Storj /bin/sh
/ #

 

STEP2  --  Inside container download pre-reqs for StorjMonitor and script itself. Run the following lines in order.

apk add --no-cache g++ gcc git make bash python
export MAKEFLAGS=-j8
git clone  https://github.com/calxibe/StorjMonitor.git

 

STEP3  --  Run the install for StorjMonitor.

cd /StorjMonitor
chmod +x storjMonitor-install.sh
./storjMonitor-install.sh

 

STEP4  --  Setup StorjMonitor for your API-Key.  

In this step, be sure to replace MONITOR_KEY with your API-Key string

sed -i "s/YOUR-TOKEN-HERE/MONITOR_KEY/" /StorjMonitor/storjMonitor.js;

 

EXAMPLE:

sed -i "s/YOUR-TOKEN-HERE/H8WvafR710FPK2zA46Td7STT24HsGB5xc9cZrODs/" /StorjMonitor/storjMonitor.js;

 

STEP5  --  Run time StorjMonitor

Prepare for running in background:

nohup ./storjMonitor.sh &

Start the script:

./storjMonitor.sh daemon -F &

Leave the container:

exit

END OF INSTRUCTIONS.

 

    Note: since this is adding to the container, and not part of the container's repo, any time an update is run these changes will be removed!  Personally I'm going to try turning off automatic updates for this Docker Container; use storjstats to check my user version level, and see just how often it actually changes.  And yes, optimal solution is a working repo of this just built-in, I think it will get there. ;) 

 

EDIT: I'm onto working code now; I have a running docker container of Storj and StorjMonitor! Trying to remove a few quarks and perhaps add a multiple-nodes feature. Afterwards its pushing it to the repo and documentation here.

Edited by Jcloud
Changed a command in step5, seems to work better.

Share this post


Link to post

I got into Storj because of this container and was able to get StorjMonitor running on my 3 nodes thanks to your well written instructions. 

 

Thanks for your hard work!

Edited by Tvguru

Share this post


Link to post

IT'S FINISHED! *For now.  @nuhll @L0rdRaiden @MrChunky  -- you asked for it, here's the initial release. ;)

 

Be on the look-out in Community Apps for a new container, StorjMonitor  (( https://hub.docker.com/r/zugz/r8mystorj/ )).   The container has the following features:

  • Base container you all know: Storj DAEMON + CLI tools.
  • Installed, storjstat.com monitoring script AND a place for users to put  API-key in the template on webui! 
  • Now supporting building and launching of multiple farming nodes, AND a place for users to put it in the template on webui! 
  • Redone webui template, from lessons learned in past two months; afield to put in your DDNS address (don't have to edit your configs after build anymore)
  • Changed default network type from Bridge to Host (seemed to work out better for most and/or preferred)  . . . Should be able to do either way, if done correctly.

To start the storjstat.com monitor script, once the container has started:    Auto-runs now (removed previously listed steps).

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

 

For users already running multiple nodes, using a single container, the extra node directories will be "Node_#"  -- which is what I was using before. If you are using something different like, "node#," you will want to rename your directory names. I think I could make this it's own custom field in the template  (DONE); honestly I've been trying keep the features down on initial release because of debugging errors, and my newbie coding coding skills -- I think I've hit all the major requests though.

 

EDIT: I've noticed the storjMonitor.sh doesn't seem to stay up for long if the interactive session to container is closed -- making this better is on my to-do list. 

EDIT2: I changed the nohup line based on my web searches, and trying it out now on personal system.

EDIT3:   Current "to-do" list

  • Link repo and docker hub, for updates.
  • Running storjMonitor on container startup. Done, just need to update the template. Should be in a few.
  • Not breaking (n+1) version(s).
Edited by Jcloud
Pushed an update; two additional features.

Share this post


Link to post

Looking good, I got the nodes up and running from the latest docker! How do you check that the storjmonitor is reporting properly? Everything is up, but I don't see any changes on StorjStat.com... Probably need to wait for someone to actually put things on the share :)

Edited by MrChunky

Share this post


Link to post

hey, thanks for the update.

 

dont wonder, ill try and report back asap, but im currently in egypt and come monday back.

 

thanks!!

 

edit:

am i right, i can just put files and config from old docker into new?

Edited by nuhll

Share this post


Link to post
7 hours ago, MrChunky said:

How do you check that the storjmonitor is reporting properly? Everything is up, but I don't see any changes on StorjStat.com..

Non-technical way:

storj.thumb.png.82fc4fe4a5850e408aa52a5ed03c5c64.png

 

Technical way, from your console:

docker exec NameOfYourContainer ps -a

You should see the process running there. As for the output it generates, that is gone, a by-product of making go in the background. 

7 hours ago, nuhll said:

edit:

am i right, i can just put files and config from old docker into new?

Yes, I've done NOTHING to Storj and StorjMonitor code, all I've done is shell-script code to install storjMonitor, handle the API-key, and start/stop multiple nodes in same Storj daemon.

Edited by Jcloud

Share this post


Link to post

Played around with a new docker. Thanks a bunch for getting this done.

 

1. Firstly, I think that bridge still works better. If I try to run multiples of the new docker (e.g. I want to share unassigned devices and main storage) the second docker tries to launch with the same IDs as the first in Host mode. Not a problem in Bridge mode.

 

2. Secondly the storagemonitor is not working for me. I get:

* starting daemon in foreground

{"level":"info","message":"attempting to start node with config at path /storj/config.json","timestamp":"2018-04-20T07:05:12.296Z"}

* starting node with config at /storj/config.json
{"level":"info","message":"got status query","timestamp":"2018-04-20T07:05:12.498Z"}
[07:05:12] 94fc2d886f6c664a20f55f8c357c189775f79a46 | Submit to Storjstat
[07:05:12] ERROR {"error":"Invalid farmer data (spaceUsedBytes)"}

{"level":"info","message":"got status query","timestamp":"2018-04-20T07:16:41.768Z"}
{"level":"info","message":"got status query","timestamp":"2018-04-20T07:16:56.028Z"}
{"level":"info","message":"got status query","timestamp":"2018-04-20T07:20:12.528Z"}
[07:20:12] 94fc2d886f6c664a20f55f8c357c189775f79a46 | Submit to Storjstat
[07:20:12] 94fc2d886f6c664a20f55f8c357c189775f79a46 | Error

{"level":"info","message":"got status query","timestamp":"2018-04-20T07:20:41.421Z"}
{"level":"info","message":"got status query","timestamp":"2018-04-20T07:35:12.628Z"}
[07:35:12] 94fc2d886f6c664a20f55f8c357c189775f79a46 | Submit to Storjstat
[07:35:12] 94fc2d886f6c664a20f55f8c357c189775f79a46 | Error

...

This node does however have stored data:

┌─────────────────────────────────────────────┬─────────┬──────────┬──────────┬─────────┬───────────────┬─────────┬──────────┬───────────┬──────────────┐
│ Node                                        │ Status  │ Uptime   │ Restarts │ Peers   │ Allocs        │ Delta   │ Port     │ Shared    │ Bridges      │
├─────────────────────────────────────────────┼─────────┼──────────┼──────────┼─────────┼───────────────┼─────────┼──────────┼───────────┼──────────────┤
│ 94fc2d886f6c664a20f55f8c357c189775f79a46    │ running │ 3h 40m … │ 0        │ 162     │ 42            │ 2ms     │ 4000     │ 13.5GB    │ connected    │
│   → /storj/share                            │         │          │          │         │ 4 received    │         │ (TCP)    │ (0%)      │              │
└─────────────────────────────────────────────┴─────────┴──────────┴──────────┴─────────┴───────────────┴─────────┴──────────┴───────────┴──────────────┘

All the numbers are blue in my storjstat website. Not sure what other details I can provide to diagnose this...

 

3. Another thing is that I tried to migrate multiple nodes to the new docker. The first node (that is in the root folder) gets the stored data identified properly in the terminal with "storjshare status" command. The extra nodes do not get existing data shown properly, although they connect fine. This could be peculiarity of my setup as I ran the nodes of unassigned devices in R/W Salve mode...

Edited by MrChunky

Share this post


Link to post
[13:55:53] nodeid | Submit to Storjstat
[13:55:53] nodeid | Submit to Storjstat
[13:55:53] nodeid | Submit to Storjstat
[13:55:53] nodeid | Submit to Storjstat
[13:55:54] nodeid | Error
[13:55:54] nodeid | Error
[13:55:54] nodeid | Error
[13:55:54] nodeid | Error

I'm having the same issue as MrChunky, plain error message. 

 

Do you guys know why this happens?

 

The nodes have data

 

 

Edited by phithor

Share this post


Link to post
8 hours ago, MrChunky said:

1. Firstly, I think that bridge still works better. If I try to run multiples of the new docker (e.g. I want to share unassigned devices and main storage) the second docker tries to launch with the same IDs as the first in Host mode. Not a problem in Bridge mode.

 

If the consensus is bridge mode is better, I don't have a problem changing it back. Sorry folks.  For a "online poll" of this give this post an upvote (not the like) and if there's a bunch I'll change it. 

 

8 hours ago, MrChunky said:

2. Secondly the storagemonitor is not working for me. I get:

Fuuuu...  Um, yeah.  Is that node running from a different container, or was this from consolidating multiple containers/paths into my new container?

8 hours ago, MrChunky said:

┌─────────────────────────────────────────────┬─────────┬──────────┬──────────┬─────────┬───────────────┬─────────┬──────────┬───────────┬──────────────┐ │ Node │ Status │ Uptime │ Restarts │ Peers │ Allocs │ Delta │ Port │ Shared │ Bridges │ ├─────────────────────────────────────────────┼─────────┼──────────┼──────────┼─────────┼───────────────┼─────────┼──────────┼───────────┼──────────────┤ │ 94fc2d886f6c664a20f55f8c357c189775f79a46 │ running │ 3h 40m … │ 0 │ 162 │ 42 │ 2ms │ 4000 │ 13.5GB │ connected │ │ → /storj/share │ │ │ │ │ 4 received │ │ (TCP) │ (0%) │ │ └─────────────────────────────────────────────┴─────────┴──────────┴──────────┴─────────┴───────────────┴─────────┴──────────┴───────────┴──────────────┘

 

8 hours ago, MrChunky said:

3. Another thing is that I tried to migrate multiple nodes to the new docker. The first node (that is in the root folder) gets the stored data identified properly in the terminal with "storjshare status" command. The extra nodes do not get existing data shown properly, although they connect fine. This could be peculiarity of my setup as I ran the nodes of unassigned devices in R/W Salve mode...

Are the file and directory permissions the same on root folder as it is with the sub folders? Perhaps that's what is going on?

 

@MrChunky @phithor Sorry to have broke stuff; we'll try and get this sorted out. Perhaps the config of being on a host versus bridge before is causing some breakage?  You could try changing the webui template from HOST to bridge, then at the bottom click on ADD "port, variable, directory" select PORT ... make it the same as your other container.

Edited by Jcloud

Share this post


Link to post

@Jcloud you're a badass...I forget if you sent me your ETH address or not. If not, please send.  I need to donate.

Edited by physikal

Share this post


Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.