Jump to content
Jcloud

[Support] QDirStat, Jcloud - cryptoCoin templates

321 posts in this topic Last Reply

Recommended Posts

7 hours ago, phithor said:

The error below showing at docker image start is caused by the StorjMonitor being loaded before the node has fully started up. It should be the last thing to be started and preferably at least 10 seconds after the last node start command to be sure.

Oh okay, I think I know what to change there in the Docker and entrypoint file. I'll make the change tonight and push it.

Share this post


Link to post

Till now:

(storj) 1 fresh new install with ur new docker - some reputation gained, but no files stored so far, so it makes sense that storjmonitor is not working

(storj1) 1 fresh install and then (stop) and copy old files over - i guess here it should report files stored, but didnt do 

9 other nodes with just "old files with new docker" 

 

none of them show any data till now, (for storj1-10)  im also concerned if they work correct, because i get many timeouts in log... also no increase in reputation.. but timeout goes down...?!

 

BTW, while were at logs... in all of my dockers the log directory get bigger and bigger, could u implement an simple startup script which deleted logs after 3 days (or something like this?) Currently i have about 10GB of logs in my 11 nodes xD

storj.png

 

storj1.png

Edited by nuhll

Share this post


Link to post

ITS WORKING. CALL ME GOD! xD

 

1.) Ur script doesnt seem to enter the API key correct. Make sure YOUR KEY HERE is in your original storjmonitor.js (its not in mine when i install from your template, its your key in there for me, every time)

 

2.) (Also it might be that ur scritp doenst set chmod +x? (executable?))? [doenst needed see last post]

 

3.) could u implement a "if share diretory = 0 bytes" dont open storjmonitor? (would reduce false positives) 

 

4.) please add a "delete /log/" after X days or after X mbs 

 

You can skip the rest until the last post last words...

 

It said var token = "H8WvafR710FPK2zA4ZTd7NTX24HsGB5xc9curODs"; (where does it get that from!?)

 

I just "vi"ed it and changed it with my real api key... and ...

 

/StorjMonitor # vi storjMonitor.js
/StorjMonitor # ./storjMonitor.js
/bin/sh: ./storjMonitor.js: Permission denied
/StorjMonitor # chmod +x storjMonitor.js
/StorjMonitor # ./storjMonitor.js
[09:33:01] 5e2ce75d9e5d679d125eb2a2228c987f2f236a61 | Submit to Storjstat
[09:33:02] 5e2ce75d9e5d679d125eb2a2228c987f2f236a61 | Success

 

It also appears on the website finally..

 

THANK YOU FOR YOUR HELP! AND YOUR SCRIPTS! AND YOUR TIME!

 

Now just one last question... when the docker gets updated, does this changes vanish? (the correct api key?)

 

edit:

 

It seems like its just manually working, in log still: 

* starting node with config at /storj/config.json
{"level":"info","message":"got status query","timestamp":"2018-04-26T09:45:24.250Z"}
[09:45:24] 6dd89bb44f8b72ed5cfe5be5f506161b8dac5b3f | Submit to Storjstat
[09:45:24] ERROR {"error":"Invalid farmer data (spaceUsedBytes)"}

 

:(:(

Edited by nuhll

Share this post


Link to post

I cant write more without writing in red, so new thread...

 

/ # ls -l
total 12
drwxr-xr-x    1 root     root           218 Apr 26 09:45 StorjMonitor
drwxr-xr-x    1 root     root           808 Jan  9 19:36 bin
drwxr-xr-x    5 root     root           340 Apr 26 09:45 dev
-rwxrwxrwx    1 root     root          2016 Apr 19 07:10 entrypoint
drwxr-xr-x    1 root     root           530 Apr 24 13:31 etc
drwxr-xr-x    1 root     root             0 Jan  9 19:36 home
drwxr-xr-x    1 root     root           222 Jan  9 19:36 lib
lrwxrwxrwx    1 root     root            12 Jan  9 19:36 linuxrc -> /bin/busybox
drwxr-xr-x    1 root     root            28 Jan  9 19:36 media
drwxr-xr-x    1 root     root             0 Jan  9 19:36 mnt
dr-xr-xr-x  580 root     root             0 Apr 26 09:45 proc
drwx------    1 root     root            56 Apr 26 09:43 root
drwxr-xr-x    1 root     root             0 Jan  9 19:36 run
drwxr-xr-x    1 root     root           766 Jan  9 19:36 sbin
drwxr-xr-x    1 root     root             0 Jan  9 19:36 srv
drwxrwxrwx    1 99       users           16 Apr 24 12:05 storj
dr-xr-xr-x   13 root     root             0 Apr 26 09:45 sys
drwxrwxrwt    1 root     root            64 Apr 19 05:31 tmp
drwxr-xr-x    1 root     root            46 Apr 19 05:31 usr
drwxr-xr-x    1 root     root            78 Jan  9 19:36 var
-rw-rw-rw-    1 root     root           152 Apr 19 06:25 versions
/ # cd storj
/storj # dir
/bin/sh: dir: not found
/storj # ls -l
total 8
-rw-rw-rw-    1 99       users         3062 Apr 24 14:01 config.json
-rw-rw-rw-    1 99       users         3062 Apr 24 14:01 config.json.bak
drwxrwxrwx    1 99       users          108 Apr 26 00:00 log
drwxrwxrwx    1 99       users           68 Apr 24 12:03 share
/storj #

 

Could it because of wrong rights? If so, you could add "set correct rights" at startup part?

 

For my part, i dont need everything setup in template, i would rather set the correct values in the conf, so i know its working all correctly... 

 

Also, for me, in your entrypoint file, but i could have no idea about it, it seems like you dont use variables for ports? How does it even work? xD 

 

sed -i "s/YOUR-TOKEN-HERE/${MONITORKEY}/" /StorjMonitor/storjMonitor.js;

mkdir -pv "${DATADIR}/share" "${DATADIR}/log";
if [ ! -f "${DATADIR}/config.json" ]; then
        storjshare create --storj "${WALLET_ADDRESS}" --storage "${DATADIR}/share" --size "${SHARE_SIZE}" --rpcport 4000 --rpcaddress "$ (later ports also no variable)

 

What i can confirm, if you install it with your docker, you dont set YOUR-TOKEn-HERE in storjmonitor.js, so your script cant find that point and all dockers created with this use wrong api key... if i set YOUR TOKEN HERE, the api key g ets replaced just fine.  (but still not working) - only manual.

 


I would suggest to publish this container first without variables, just config files which needs to be edited (like you can access atm storj conf, but not storjmonitor from outside of docker)

I also wonder why i cant use apt-get in the docker, shouldnt this all be installed via the install script?
I also cant "find" (inside the docker) the daemon.log which storjmonitor should create... 

 

EDIT:

okay some nodes seem to work and report their size correctly, i just edited the storjmonitor.js and replaced the key with mine. Ill try to find out if all work or what i have done other then on others (like if its needed to +x it)


Edit2:

GOOD NEWS!

 

It seems to10 from 11 nodes are working, it doesnt matter if u +x it. U just need to replace the key in the conf.

 

If you want to keep the variables, then you need to make sure that ur original storjhmonitor.js has actually YOUR_KEY_HERE in it. So the replacement works.


Edit3:

Seems like all 11 nodes are running fine now. Its just confusing because you get invalid farmer data and or error (and it works 15 min later on his own) (i guess there are a max 1 connection per 15 min setting on their server or something)

Edited by nuhll

Share this post


Link to post
4 hours ago, nuhll said:

GOOD NEWS!

 

 

@nuhllI'm still half a sleep, coffee needs to kick in, but glad it's working. I think you have my old code, or I uploaded an old version. A number of the points you brought up are issues I fought with before I posted.  I think I need to clean up my repo and stream line my updating process, and stop being lazy with the online documentation on github. 

 

For example you listed 

4 hours ago, nuhll said:

f [ ! -f "${DATADIR}/config.json" ]; then
        storjshare create --storj "${WALLET_ADDRESS}" --storage "${DATADIR}/share" --size "${SHARE_SIZE}" --rpcport 4000 --rpcaddress "$ (later ports also no variable)

 

And you're right it doesn't increment there, and I updated (at some point, again maybe you got my old version before I caught it  -- sorry ):

i=1;
j=4000;
while [ "$i" -le "${NODE_COUNT}" ]; do
         mkdir -pv "${DATADIR}/${NODE_DIR}$i/share" "${DATADIR}/${NODE_DIR}$i/log";
         if [ ! -f "${DATADIR}/${NODE_DIR}$i/config.json" ]; then
             j=$(( j + 1 ));
             storjshare create --storj "${WALLET_ADDRESS}" --storage "${DATADIR}/${NODE_DIR}$i/share" --size "${SHARE_SIZE}" --rpcport "$j" --rpcaddress "${RPCADDRESS}"  --logdir "${DATADIR}/${NODE_DIR}$i/log" --outfile "${DATADIR}/${NODE_DIR}$i/config.json" --noedit;
         fi;
      i=$(( i + 1 ));
done;

 

I didn't make the changes I said I would yesterday, my night walked away from me. Nuhll I'll go over your notes, my finished code, compare notes and try to clean house. I'll try and get on all of this ASAP. Sorry I'm such a noob software dev everyone -- thanks for putting up with my learning project. :)

 

 

 

 

 

Share this post


Link to post

It seems the only thing you need to change now is:

 

1.) Make sure YOUR KEY HERE is in your original storjmonitor.js so your entrypoint script can cahnge it to the correct api key, besides this, it seems to work just fine!


or change and dont look for YOUR KEY HERE, but for your api key "H8WvafR710FPK2zA4ZTd7NTX24HsGB5xc9curODs" and let it get changed to the real api key.

 

2.) let it delete the log folder (or the files in it) automatic (maybe just always on restart, or if its X days old.)

Edited by nuhll

Share this post


Link to post

Was there an update or something that isn't current on this docker? Seems like my peers are super low.

 

image.thumb.png.d0419bc1a04e30f1c5a2bd30c431682a.png

Share this post


Link to post
5 hours ago, physikal said:

Was there an update or something that isn't current on this docker? Seems like my peers are super low.

   Yeah I was noticing that too on mine; I need to get on the ball, I'm not sure what's going on with my Container anymore. However, I have noticed a significant change to the Docker and Entrypoint file on the Original -- I think this broke my container; that's the bad news. The good news is: I think contributor of push totally stole (lol) my storjstats code saw my idea; tweaked it; and storjstat.com script totally part of the official container now. 

 

   Also Nuhll code snip that I commented on, actually looks more like the merged code, but that is my template and screenshot is my repo. I'm confused. :/ It's broken.

Tonight's Plan (right after this post):

  1.  Revisit the Storj template, update it for the storjstat.com API.DONE Making the assumption these people are more competent with the work.
  2. Nuke my repo, pull from official, rewrite for multiple nodes DONE(as this is NOT in the official docker). 
  3.  I hear you Nuhll, I'll see about Docker logs, and check for mine. However here's a tip I pulled a year ago out of the forums:

 

Run the following command from unRAID host, to delete all logs from your docker image file.  (the short-term fix; or for CA user scripts plugin)

 logs=$(find /var/lib/docker/containers/ -name '*.log');for log in $logs; do cat /dev/null > $log;done

 

Edited by Jcloud

Share this post


Link to post
17 hours ago, nuhll said:

Now just one last question... when the docker gets updated, does this changes vanish? (the correct api key?)

 

Technically, yes. But if/when the template is back to working order the line-item for the API key is what inserts it back into the container -- so from what the user experiences it should act like, "No, the changes don't vanish."

 

Make sense, or did I make it more confusing?

Share this post


Link to post

The webui template for Storj (oreandawe/storjshare-cli) has been updated for the Storjstat.com monitor script, and as best as I can tell, is working.

The template and container/repo zugz/r8mystorj also updated; for the time being please consider it experimental -- I haven't tested it yet, but did notice the ticker for downloads +1 in the process of updating the container image. 

 

I still haven't managed to correctly configure auto-builds, for up-stream updates; I need to work on that this weekend (if time permits).

 

Share this post


Link to post

Hmm.. im confused some minutes ago all my dockers updated... and storj monitor is complete removed... :/

 

With the replacement i understand what you mean, but your script is broken at this point (atleast for me) the last version try to change YOUR KEY HERE to my key, but there was not YOUR KEY HERE in your original file, you understand? :D

 

Im also wondering i dont get any new MBs since i switched to your new template... i wonder if storj itself is working correctly.

 

And now after update, all storj monitor is missing.. 

Quote

 


root@Unraid-Server:~#  docker exec -ti Storj1 /bin/sh
/ # cd
bin/    etc/    lib/    mnt/    proc/   run/    srv/    sys/    usr/
dev/    home/   media/  opt/    root/   sbin/   storj/  tmp/    var/
/ #
/ # cd storj
/storj # dir
/bin/sh: dir: not found
/storj # ls -l
total 8
-rw-rw-rw-    1 99       users         3062 Apr 24 14:01 config.json
-rw-rw-rw-    1 99       users         3062 Apr 24 14:01 config.json.bak
drwxrwxrwx    1 99       users          108 Apr 27 00:00 log
drwxrwxrwx    1 99       users           68 Apr 24 12:01 share
/storj #
 

 

Edited by nuhll

Share this post


Link to post

 

For log deleting im using

 

Quote

 


echo ""
echo "<font color='red'><b>Before:</b></font>"
echo "====================================================================================================================================================================================="
du -ah /var/lib/docker/containers/ | grep -v "/$" | sort -rh | head -60 | grep .log
echo "====================================================================================================================================================================================="
echo "Cleaning Logs:"
logs=$(find /var/lib/docker/containers/ -name '*.log');for log in $logs; do cat /dev/null > $log;done
sleep 6
echo "...<font color='blue'>cleaning complete!</font>"

echo ""
echo "<font color='green'><b>After:</b></font>"
echo "====================================================================================================================================================================================="
du -ah /var/lib/docker/containers/ | grep -v "/$" | sort -rh | head -60 | grep .log
echo ""
 

 

 

Quote

Script location: /tmp/user.scripts/tmpScripts/Clean Docker Log/script
Note that closing this window will abort the execution of this script

Before:
=====================================================================================================================================================================================
8.0K /var/lib/docker/containers/b1f4af6c273bc4ceedc56e5c9cfa9cb085c20cd1598ee90983da1c2f58a604c8/b1f4af6c273bc4ceedc56e5c9cfa9cb085c20cd1598ee90983da1c2f58a604c8-json.log
4.0K /var/lib/docker/containers/f8a0c1d6badabee7df34bcfe12bfee05d12e9b24823e2076494fecc8bbf3e29f/f8a0c1d6badabee7df34bcfe12bfee05d12e9b24823e2076494fecc8bbf3e29f-json.log
=====================================================================================================================================================================================
Cleaning Logs:
...cleaning complete!

After:
=====================================================================================================================================================================================
20K /var/lib/docker/containers/e9ea68fe60617ac51270ab5985365f655038028f04abceaa757c093b52970d14/e9ea68fe60617ac51270ab5985365f655038028f04abceaa757c093b52970d14-json.log

 

 

But i also tried yoru script via ssh, still there are logs in the storj containers...?!

Quote

 


logs=$(find /var/lib/docker/containers/ -name '*.log');for log in $logs; do cat /dev/null > $log;done

 

 

As far as i understand these logs has to be deleted from within the container!?

Edited by nuhll

Share this post


Link to post

btw, i found https://ssdynamite.com/ there you can check your log files of storj if there is any problem... :) seems like all instances are working.

Edited by nuhll

Share this post


Link to post

I found out why its not working.

 

In template you name your storjmonitor key "MONITORKEY", but in entrypoint its:

if [ -n "${STORJ_MONITOR_API_KEY}" ]; then
    sleep 10;
        sed -i "s/YOUR-TOKEN-HERE/${STORJ_MONITOR_API_KEY}/" /opt/StorjMonitor/storjMonitor.js;
        cd /opt/StorjMonitor;
        (/opt/StorjMonitor/storjMonitor.sh &) &

 

i guess you need to change MONITORKEY to STORJ_MONITOR_API_KEY or remove the "if"


Edit:
Yes it works, you need to change MONITORKEY in template to STORJ_MONITOR_API_KEY...., then its filling in the correct API Key. Its because that IF checks if this key is there, and dont start storjmonitor when the key is not there... 


While you at it, you could add (or a working code)

 

logs=$(find /storj/log/ -name '*.log');for log in $logs; do cat /dev/null > $log;done

 

to the entrypoint script (dont work for me, i dont know why :()

Edited by nuhll

Share this post


Link to post
5 hours ago, nuhll said:

while you at it, you could add (or a working code)

 

logs=$(find /storj/log/ -name '*.log');for log in $logs; do cat /dev/null > $log;done

Is the container's /var/log directory filling up, or are you refering to /storj/storj1/log folder(s) that are increasing in space?

 

Share this post


Link to post
On 4/25/2018 at 3:35 AM, phithor said:

So tried to change the ownership of the storjfolder from 99(nobody) to root but it I still get the error. Using unassigned devices. Any wizards who know if there is anything else I should change? 

Hi Phithor, there seems to have been a SNAFU or FUBAR with with my  stuff the last few days. A this time I would recommend deleting the container, and its image, from unraid and download the container again; I'm hoping that will fix your issue, but I think that's the first place to start. Please note the Storj templates have changed to work with the up-stream variables, so it should look a little different from the old template -- might be best to go back to CA Apps and grab it from there.

 

My "Storj" docker template appears to be working again; work is a little slow so I'm going to try and remotely test my StorjMonitor repo/container.

 

Share this post


Link to post

Did you change this?

 

Quote

 


In template you name your storjmonitor key "MONITORKEY", but in entrypoint its:

if [ -n "${STORJ_MONITOR_API_KEY}" ]; then
    sleep 10;
        sed -i "s/YOUR-TOKEN-HERE/${STORJ_MONITOR_API_KEY}/" /opt/StorjMonitor/storjMonitor.js;
        cd /opt/StorjMonitor;
        (/opt/StorjMonitor/storjMonitor.sh &) &

 

i guess you need to change MONITORKEY to STORJ_MONITOR_API_KEY or remove the "if"


Edit:
Yes it works, you need to change MONITORKEY in template to STORJ_MONITOR_API_KEY...., then its filling in the correct API Key. Its because that IF checks if this key is there, and dont start storjmonitor when the key is not there... 
 

 

 

And yes im refering to the stroj/log folders

Edited by nuhll

Share this post


Link to post
2 minutes ago, nuhll said:

Did you change this?

Yes, MONITORKEY has been changed to STORJ_MONITOR_API_KEY

 

18 minutes ago, nuhll said:

And yes im refering to the stroj/log folders

Okay, now I know what you're looking at, and what to focus on. You're right, what I gave you before won't work for that, I thought you were talking about something else.

Share this post


Link to post

No problem!

 

Im glad u know what i mean now... i guess your log files are also getting big?

Share this post


Link to post

Okay, now i really now it works all.

 

My first stored MB since i tried to install storjmonitor... :)

 

Now we only need to fix the logs directory and then its perfect.

 

 

yeah.png

 

My suggestion for the deleting of the log file would be:

 

if [ -n "${DELETE_LOG}" ]; then
    sleep 10;
    find /storj/log -iname "*" -mtime +1 -delete

fi

 

And we just add a key DELETE_LOG to the template and when its there it deletes logs older then 1 day. I cant think why you dont want this, but if someone dont want it, he just removes the DELETE_LOG from the template and finish.

 

I testet 

    find /storj/log -iname "*" -mtime +1 -delete

and it works inside the storj container. Removed over 300mb in jsut one storj instance.. :D

 

We could also do a "if the dir is there" instead, but i guess, it doenst matter at the end.

 

Edited by nuhll

Share this post


Link to post
6 hours ago, nuhll said:

And we just add a key DELETE_LOG to the template and when its there it deletes logs older then 1 day. I

   Storj.io has a process of auditing the monthly usage, because of this, logs might be useful to some. Storj.io is also a slow, and in my opinion, inconsistent with their monthly payout process. I was thinking I might be better if the script killed logs older then a month or two.  My perspective, may be partially skewed as you're talking of cleaning GB  of data, while my user share is 19TB free space :D.   However the growth of this data space is real issue, I won't deny that. 

   I'm also going to take my time on log deletion, because we're talking about deleting files on unRAID host.  I don't want to nuke something important, community members or mine.

 

Thanks for the code snips, agree, I think an ON/OFF switch is a good idea. Perhaps a second variable to set how many days of logs to keep, instead of a hard-set number.

 

Quote

 guess your log files are also getting big?

My /storj/ folder totlal is only 37G, logs and data; log folder is only 407M.

 

root@HYDRA:/mnt/user/storj# du -h -s
37G     .
root@HYDRA:/mnt/user/storj# du ./*/log -h -s
0       ./Node_1/log
0       ./Node_10/log
0       ./Node_2/log
0       ./Node_3/log
0       ./Node_4/log
0       ./Node_5/log
0       ./Node_6/log
0       ./Node_7/log
0       ./Node_8/log
0       ./Node_9/log
root@HYDRA:/mnt/user/storj# du ./log -h -s
407M    ./log

 

Edited by Jcloud

Share this post


Link to post

Hey, in my mind its not deleting from unraid host.

 

Isnt entrypoint only run inside the docker and /storj/log is also only inside the docker?

 

My log dirs are different in size, but i think about 10gb in not a month, without my big node which i deleted and was 3 months old.

 

U can easy change +1 to +30 to delete 30 days old data. I dont really think the log files are improtant, because you could also set it to no logging at all. Log files are so easy to modify i really hope storj is not relayin on them for anything xD

Edited by nuhll

Share this post


Link to post
12 minutes ago, nuhll said:

snt entrypoint only run inside the docker and /storj/log is also only inside the docker?

You have a point, I've been "gun-shy" over deleting data, my brain is flagging it as a danger-zone, and with last few days -- my confidence is a bit torn up. 

 

A point of information:  While /storj/log is the path inside the container,  /storj (the parent directory) is mapped outside of the container  -  this is why your contract data, and logs, are not deleted whenever the container image is updated or deleted.  The /storj path is mapped in the template, and a requirement of template/container.


  hW8oF2Fl.png

Share this post


Link to post

Yeah, i know that. Thats why i could easy check how big my log dir is. 

 

But if you want to make it perfect you could make the +1 a variable, with standard 30.

 

This way you have 1 variable to enable delete and one variable where every user could say if delted after 1...2...3..or 60 days... 


Also, i dont care about 100gb more or less, like you, but just think about it at some time you forget the storj containers and its getting more and more... :) like in a year 1000gb...^^

Edited by nuhll

Share this post


Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.