[Support] Partition Pixel - Chia


Recommended Posts

5 hours ago, Warwick Price said:

Haven't seen mention of this error/issue in this thread, so maybe it's just me!

 

I have been running Chia on a windows machine for a few weeks and so already have a key. When I saw that we had a CA version I quickly loaded it. Followed the instructions and created my mnemonic.txt file and pasted by 24 words. Whenever I try and start the app I get the error that it can't find the file. I've checked the spelling.... I've tried moving it to another folder and changing the entry in the setup page and still the same issue.  FYI... I have windows set to always show file extensions.

 

Any ideas?

 

image.png.3f68656ab760a5bccd1430163571aaa2.png

is the mnemonic.txt file just plain no punctuation or anything like just the 24 words.   cat dog mouse farm chia and so on and so forth? 

Link to comment
Posted (edited)
1 hour ago, guy.davis said:

Yes, this is a good reason to have a plotting manager bundled into the Docker image.  In that case, the plot manager runs plotting in the background even if an Unraid (or other) terminal window is closed.  However, if entire Docker container is restarted/updated, then all plots are killed.

 

I'm working on this with Machinaris, which bundles Plotman + a basic WebUI along with everything in the main Docker image.  I hope to have Machinaris listed in the Unraid CA repo soon.  Any feedback is appreciated.

 

If you guys want to try this champions @guy.davis docker with plotman inside jump on his discord channel he has a walk through for template setup and it works well so far...or actually i just saw he has added it to get approved by CA so u can wait a little bit if you want and do it from there

Edited by burgess22
  • Like 1
  • Thanks 1
Link to comment
Posted (edited)

So I have swar plot manager running, had time to play with it today.

This setup isn't elegant and probably isn't following best practices but it works for me. Please be mindful of this, if you break something, lose plots, etc.. don't come cryin'

 

Open the docker console (click on the docker --> console) NOTE: not the unraid main console

We are going to pull swar's git and install its python dependencies.  Installing into /root/.chia in the docker which points to /mnt/user/appdata in unraid.

 

cd /root/.chia
git clone https://github.com/swar/Swar-Chia-Plot-Manager
cd Swar-Chia-Plot-Manager
/chia-blockchain/venv/bin/pip install -r requirements.txt
cp config.yaml.default config.yaml

 

You can now edit the config.ymal file using an editor supported within this docker OR from the appdata/chia/Swar-Chia-Plot-Manager folder within unraid (unraid console, krusader, a windows share if you set that up)

 

Here are some values to be used along with whatever else you set in the config...

chia_location: /chia-blockchain/venv/bin/chia

folder_path: /root/.chia/logs/plotting

  temporary_directory: /plotting
  destination_directory: /plots

 

now test if it's working

. /chia-blockchain/activate
python manager.py view

 

make sure the drives look like they have the correct space and used space values (if not then your probably mapping to a folder inside your docker image. /plotting and /plots are the mappings used during the default chia docker setup, that's why we used them here.  If you start a chia plotting process and these aren't right you will fill your docker to 100% usage!  If you're running "fix common problems" plugin, you will see warnings in the unraid GUI.  You'll have to clean up the mess you made buy deleting whatever incorrect folders you created in the docker.

 

CTRL+C to get out of view mode

If everything looks good lets start swar manager

 

python manager.py start

 

Now whenever you want to use the swar manager open the chia docker console and view (or replace which ever command you need.. start, restart). You need to activate the python virtual environment everytime before your manager.py command as stated by the swar documentation. This is the second line you see here..

 

cd /root/.chia/Swar-Chia-Plot-Manager/
. /chia-blockchain/activate
python manager.py view

 

Want to use the main unraid console instead of being stuck inside the docker console, heck even use tmux?  Do the following then repeat the commands directly above...

 

docker exec -it chia bash

 

Enjoy!
I look forward to any suggestions for improvement, I'm sure there are better methods. 

 

Edited by zer0ed
  • Like 5
  • Thanks 1
Link to comment

I've managed to solve my wallet not syncing issue by using the windows client in a VM and throwing a VPN client (PIA) to bypass all my current double NAT'd badness.

Chia docker is still doing the hard work of harvesting, but now everything is synced up and working as normal in the windows client (pointing to chia smb share)

 

I'm sure this can be done with docker networking, but I wanted a quick solution to tide me over.

Link to comment
Posted (edited)
1 hour ago, tjb_altf4 said:

I've managed to solve my wallet not syncing issue by using the windows client in a VM and throwing a VPN client (PIA) to bypass all my current double NAT'd badness.

Chia docker is still doing the hard work of harvesting, but now everything is synced up and working as normal in the windows client (pointing to chia smb share)

 

I'm sure this can be done with docker networking, but I wanted a quick solution to tide me over.

if you just wanted to run a container through a VPN  and you have a docker with a VPN already running its quite easy to route the traffic through that dockers VPN the master him self space invader one shows how to do it...ill find the clip. save you having to run the VM just for that 

 

Edited by burgess22
Link to comment
5 hours ago, burgess22 said:

is the mnemonic.txt file just plain no punctuation or anything like just the 24 words.   cat dog mouse farm chia and so on and so forth? 

correct. you just type the words with one space. No punctuation. Like:

dog fish bird house car

Link to comment

What is the general consensus on opening router port 8444 to your container for this? Do we feel the Chia software is secure enough?

 

I haven't opened the port on my router yet but I'm getting fewer peers as those who do.

 

Thoughts?

Link to comment
10 hours ago, Trinity said:

From the official chia docs: https://github.com/Chia-Network/chia-blockchain/wiki/FAQ#i-am-eeing-blocks-and-connections-but-my-node-says-not-synced
 


So It can be related, I think. I also have troubles to sync, but even after installing systemd it seems like its not syncing correctly.
I have no idea, why the whole docker is causing so much trouble, but I tried to run the GUI in a windows vm on unraid. This. Was. Horrible. The docker container was synced more than twice as fast as the one in the windows vm. So it doesn't seem to be an option. It must work with docker. Because if everything is to slow, one could have a bazillion plots, and would never win, because the reaction of a challenge is to slow...
But the docker feels very unstable. For me it syncs, then it stops, then it works, right now after fixing the time it's again not syncing... It's a nightmare. Oh and the wallet never ever synced on my end. Not sure, what is going on there.

My understanding of the official wiki is that your system clock must not be off by more than 5 minutes, whatever your timezone is, reminded that the clocks (both software and hardware) are managed as UTC by default in Linux.

Anyway, I see that many have problems syncing when running the container as a full node.

Personally I went the Raspberry way and everything is working like a charm so far. The RPi4 4GB boots from an old 120GB SATA M.2 SSD under Ubuntu 20.04.2 (in an Argon One M.2 case which is great btw) and runs the full node. In a first time, I ran the harvester on the RPi and it accessed the plots stored on the Unraid server through a SMB share. Performance was horrible, average was 5-7 seconds to respond, with peaks at more than 30 sec ! I tried NFS which was a bit better, but still too slow, often > 3 sec.

With this container running only the harvester, and connected to the farmer on the RPi, response times are under 0,1 sec at farmer level, it has a negligible footprint on the server and it's rock solid. I also create plots in the container as a bonus, to help my main plotting rig.

If others want to replicate this architecture, I posted a step-by-step a few posts above on how to setup the container as a harvester-only, and connect it to a remote farmer/full node.

An RPi is not a big investment, compared with the storage cost, and as chia was designed as a distributed and decentralised project, I think it's the way to go to have a resilient setup.

  • Like 1
Link to comment
15 minutes ago, adminmat said:

What is the general consensus on opening router port 8444 to your container for this? Do we feel the Chia software is secure enough?

 

I haven't opened the port on my router yet but I'm getting fewer peers as those who do.

 

Thoughts?

Without port 8444 forwarded to the full node, you'll have sync issues, as you will only have access to the peers which have their 8444 port accessible, and they are obviously a minority in the network....

Generally speaking, opening a port to a container is not such a big security risk, as any malicious code will only have access to the resources available to the container itself. But in the case of chia, the container must have access to your stored plots to harvest them. An attacker may thus be able to delete or alter your patiently created plots from within the container, which would be a disaster if you have hundreds or thousands of them ! And one of the ways to maximise one's netspace ownership could be the deletion of others' plots ...

That's one of the reasons why I decided to have a dedicated RPi to run the full-node, with the firewall allowing only the strictly required ports, both on the WAN and LAN sides. There's no zero risk architecture, but I think such a setup is much more robust than a fully centralised one where a single machine supports the full set of functions.

  • Like 1
Link to comment
On 5/13/2021 at 12:35 PM, guy.davis said:

 

First off, a big thanks @Partition Pixel for getting the official Docker image added to Unraid as an application! Plotting is good fun, but tricky to get going well.  

 

@Footlocker_ Yes, Plotman is great!  In fact, I am working on a single Docker image that includes this official docker image along with Plotman and a basic Web UI, all wrapped up as an Unraid application template. Right now, the Plotman CLI is invokable via the container CLI, but I hope to wire control into the Web UI shortly.  

 

The primary goal is to make plotting on Unraid as easy as the official Windows GUI application.  I've called it Machinaris and it's available for early beta testing now.  I'm using it for plotting on Unraid 6.9 and welcome anyone interested in trying the beta and helping out with the project.   

 

As you an see below, I'm just starting out too and still a member of 0 XCH gang.  Plot on!

 

Thanks,

Guy 

machinaris_home.png

machinaris_plotting.png

 

This. Looks. Amazing. Thank you for this!

 

Another neat addition to add (which imo is the most important, since not everyone uses unraid for plotting, but im assuming everyone (or at least, more people ) who uses this uses it for harvesting): https://github.com/stolk/chiaharvestgraph

  • Like 1
Link to comment
Posted (edited)

What about plotting performance? It's been quite bad for me. With a 3700X, 32GB RAM and 2x Sabrent NVMe 1TB I do about 15 plots per day, while a friend with a similar config and bare metal Ubuntu does 24.

 

I have disabled parity and am plotting on XFS-formatted unassigned devices, then moving finished plots to the array. I'm not sure if it's due to unraid or docker (and I'll be trying Ubuntu on the same machine as well), but I'm wondering what your numbers are. Using plotman, staggering plots with a maximum of 7 active plots and 1 or 2 plots in phase 1 for each SSD.

 

Thanks!

Edited by squeck
Link to comment
Posted (edited)
12 hours ago, Gnomuz said:

My understanding of the official wiki is that your system clock must not be off by more than 5 minutes, whatever your timezone is, reminded that the clocks (both software and hardware) are managed as UTC by default in Linux.

Anyway, I see that many have problems syncing when running the container as a full node.

Personally I went the Raspberry way and everything is working like a charm so far. The RPi4 4GB boots from an old 120GB SATA M.2 SSD under Ubuntu 20.04.2 (in an Argon One M.2 case which is great btw) and runs the full node. In a first time, I ran the harvester on the RPi and it accessed the plots stored on the Unraid server through a SMB share. Performance was horrible, average was 5-7 seconds to respond, with peaks at more than 30 sec ! I tried NFS which was a bit better, but still too slow, often > 3 sec.

With this container running only the harvester, and connected to the farmer on the RPi, response times are under 0,1 sec at farmer level, it has a negligible footprint on the server and it's rock solid. I also create plots in the container as a bonus, to help my main plotting rig.

If others want to replicate this architecture, I posted a step-by-step a few posts above on how to setup the container as a harvester-only, and connect it to a remote farmer/full node.

An RPi is not a big investment, compared with the storage cost, and as chia was designed as a distributed and decentralised project, I think it's the way to go to have a resilient setup.

This sounds like a really nice setup, but I still didn't fully understand, what you need to extend the farm. Which services do you need running for plotting? Can you plot via the harvesters or via the farmer, which is running together with the full node or does it only require the full node? How many farmers you need, how many harvesters? Right now I am storing the plots on the xfs array, since I don't have unassigned devices yet. I'd maybe invest in a pi4, wanted to buy one for a long time anyway but had no project to use it for in mind, so using it as chia node would be a good enough project. I wasn't aware that the pi has a M.2 slot, are there other options to store the system, like a big sd card or is it better to use a ssd?

So if I get you right:
- There is a pi4 with a m.2 sdd running the full node to accept challenges, syncing the blockchain and handle the wallet and master keys
- You have a special rig for plotting, which stores the plots on your unraid server in a share folder of your array (which is optional) - What services does that rig need to run to plot?
- You are running harvesters on your unraid system via this container, which accesses the plot folder and are connected to the pi, so the pi running the farmer accepting challenges and the harvesters on your unraid server are responding to those?
- You can use the harvesters to create plots from there as well (which is optional)? Or how are you plotting there?

So if I wouldn't have a special plotting rig, I can use the unraid system harvesters to do so or do I need to run something else?
 

Edited by Trinity
Link to comment
17 hours ago, burgess22 said:

If you guys want to try this champions @guy.davis docker with plotman inside jump on his discord channel he has a walk through for template setup and it works well so far...or actually i just saw he has added it to get approved by CA so u can wait a little bit if you want and do it from there

What is his discord?

Link to comment

Thank you for the container and all the information that comes with it! 

I've been useing the container for a little while now and I'm wondering if there is a way to automate the plot command on start-up and then every hour untill X amount are active.

Link to comment

Thank you so much for this container and for starting such a wonderful thread full of information. I am running into a situation where my Chia Node time is 2 hours off from real time. I've run the date command in console and it's returning the correct time. Because of this time discrepancy, I can't stay synced and therefore I can't do any farming. Has anyone run into this issue? Port 8444 is open and has been checked. Thanks for any help

Link to comment
Posted (edited)
7 minutes ago, mc_handler said:

Thank you so much for this container and for starting such a wonderful thread full of information. I am running into a situation where my Chia Node time is 2 hours off from real time. I've run the date command in console and it's returning the correct time. Because of this time discrepancy, I can't stay synced and therefore I can't do any farming. Has anyone run into this issue? Port 8444 is open and has been checked. Thanks for any help

Yes, around 1-2 pages back, some of us discovered the same issue. Including myself.
I tried to enter the container and to run: apt-get systemd
to set the correct time. Another option would be installing: apt-get tzdata.
There is also an open Pull request on the official chia docker: https://github.com/Chia-Network/chia-docker/pull/50
Not sure if it will solve the issues tbh. Installing it inside the container is only a quick solution, which will be lost when ever the container is rebuild. So not the best solution. Additionally to that, it didn't fix my sync status. I am now kind of more than one hour off and it never reaches the synced status. So it seems to be something different causing these issues. So you can try to get you clock right, but I can't promise it will solve your sync issues.

Edited by Trinity
Link to comment
On 5/7/2021 at 8:27 AM, burgess22 said:

Nice work boss...I nutted out how to build the docker from reddit the other day and then fix common problems popped up tonight and found an issue because you have put a template up...winning now I can plot away happily and and get auto updates in CA 🍻

 

 

one thing you might want to add is a how to start plotting once docker is up and running I assume from the console it will be 

venv/bin/chia plots create -b 5000 -r 2 -n 1 -t /plotting/plot1 -d /plots 

 

I've always started my from the terminal window using 

 

docker exec -it chia venv/bin/chia plots create -b 5000 -r 2 -n 1 -t /plotting/plot1 -d /plots

 

-b is amount of ram you want to give

-r is the amount of threads 

-n is the number of plots you want to queue

-t is temp dir

-d is the completed directory

 

and if you want to run in Parallel just run the command in another terminal window as many times as your rig will allow

Thanks for the details just one thing I'm not clear on is that with this command line how to you chose your plotting sizes ? Basically I saw in the UI the ability to choose different plotting sizes but with this command line tool does not seem to indicate it. I need to determine how many I can run on a SSD and not run out of space?

Link to comment
14 hours ago, Trinity said:

This sounds like a really nice setup, but I still didn't fully understand, what you need to extend the farm. Which services do you need running for plotting? Can you plot via the harvesters or via the farmer, which is running together with the full node or does it only require the full node? How many farmers you need, how many harvesters? Right now I am storing the plots on the xfs array, since I don't have unassigned devices yet. I'd maybe invest in a pi4, wanted to buy one for a long time anyway but had no project to use it for in mind, so using it as chia node would be a good enough project. I wasn't aware that the pi has a M.2 slot, are there other options to store the system, like a big sd card or is it better to use a ssd?

So if I get you right:
- There is a pi4 with a m.2 sdd running the full node to accept challenges, syncing the blockchain and handle the wallet and master keys
- You have a special rig for plotting, which stores the plots on your unraid server in a share folder of your array (which is optional) - What services does that rig need to run to plot?
- You are running harvesters on your unraid system via this container, which accesses the plot folder and are connected to the pi, so the pi running the farmer accepting challenges and the harvesters on your unraid server are responding to those?
- You can use the harvesters to create plots from there as well (which is optional)? Or how are you plotting there?

So if I wouldn't have a special plotting rig, I can use the unraid system harvesters to do so or do I need to run something else?
 

Hi,

Thanks for your interest in my post, I'll try to answer your numerous questions, but I think you already got the general idea of the architecture pretty well 😉

- The RPi doesn't have a M.2 slot out of the box. By default, the OS runs from a SD card. The drawback is that SD cards don't last so long when constantly written, e.g. by the OS and the chia processes writing logs or populating the chain database, they were not designed for this kind of purpose. And you certainly don't want to lose your chia full node because of a failing 10$ SD card, as the patiently created plots are still occupying space, but do not participate in any challenge... So, I decide running the RPi OS from a small SATA SSD was a much more resilient solution, and bought an Argon One M.2 case for the RPi. I had an old 120 GB SATA SSD which used to host the OS of my laptop before I upgraded it to a 512 GB SSD, so I used it. This case is great because it's full-metal and there are thermal pads between the CPU and RAM and the case. So, it's essentially a passive heat dissipation setup. There's a fan if temps go too high, but it never starts, unless you live in a desert I suppose. There are many other enclosures for RPis, but this one is my favorite so far.

 

- The RPi is started with the 'chia start farmer' command, which runs the following processes : chia_daemon, chia_full_node (blockchain syncing), chia_farmer (challenges management and communication with the harvester(s)) and chia_wallet. chia_harvester is also launched locally, but is totally useless, as no plots are stored on a storage accessible by the RPi. To get a view on this kind of distributed setup, have a look at https://github.com/Chia-Network/chia-blockchain/wiki/Farming-on-many-machines, that was my starting point. You can also use 'chia start farmer-no-wallet', and sync your wallet on another machine, I may do that in the future as I don't like having it on the machine exposed to the internet.

 

- The plotting rig doesn't need any chia service running on it, the plotting process can run offline. You just need to install chia on it, and you don't even need to have your private keys stored on it. You just run 'chia plots create (usual params)  -f <your farmer public key> -p <your pool public key>' , and that's all. The created plots will be farmed by the remote farmer once copied into the destination directory.

 

- I decided to store the plots on the xfs array with one parity drive. I know the general consensus is to store plots on non-protected storage, considering you can replot them. But I hate the idea of losing a bunch of plots. You store them on high-density storage, let's say 12TB drives, which can hold circa 120 plots each. Elite plotting rigs with enterprise-grade temporary SSDs create a plot in 4 hours or less. So recreating 120 plots is circa 500 hours or 20 days. When you see the current netspace growth rate of 15% a week or more, that's a big penalty I think. I you have 10 disks, "wasting" one as a parity drive to protect the other 9 sounds like a reasonable trade-off, provided you have a spare drive around to rebuild the array in case of a faulty drive. To sum up, two extra drives (1 parity + 1 spare) reasonably guarantee the continuity of your farming process and prevent the loss of existing plots, whatever the size of your array is. Of course with a single parity drive, you are not protected against two drives failing together, but as usual it's a trade-off between available size, resiliency and costs, nothing specific to chia ... And the strength of Unraid is you won't lose the plots on the healthy drives, unlike other raid5 solutions.

 

- As for the container, it runs only the harvester process ('chia start harvester'), which must be setup as per the link above, nothing difficult. From the container console, you can also optionally run a plotting process, if your Unraid server has a temporary unassigned SSD available (you can also use your cache SSD, but beware of space ...). You will run it just like on your plotting rig : 'chia plots create (relevant params) -f <farmer key> -p <pool key>'. The advantage is that the final copy from the temp dir to the dest dir is much quicker, as it's a local copy on the server from an attached SSD to the Unraid share (10 mins copy vs 20/30 mins over the network for me).

 

- So yes, you can imagine running your plotting process from a container on the Unraid server if you don't have a separate plotting rig. But then I wouldn't use this early container, and would rather wait for a more mature one which would integrate a plotting manager (plotman or Swar), because managing all that manually is a nightmare on the long run, unless you are a script maestro and have a lot of time to spend on it 😉

 

Happy farming !

  • Like 1
Link to comment
Posted (edited)
1 hour ago, dandiodati said:

Thanks for the details just one thing I'm not clear on is that with this command line how to you chose your plotting sizes ? Basically I saw in the UI the ability to choose different plotting sizes but with this command line tool does not seem to indicate it. I need to determine how many I can run on a SSD and not run out of space?

If you ever want to create something else than k-32 , the CLI tool also accepts the '-k 33' parameter for instance. If '-k' is not specified, it defaults to '-k 32' which is the current minimal plot size.

All options are documented here https://github.com/Chia-Network/chia-blockchain/wiki/CLI-Commands-Reference , and 'chia plots create -h' is a good reference also 😉

Edited by Gnomuz
  • Thanks 1
Link to comment
Posted (edited)
7 hours ago, Trinity said:

Yes, around 1-2 pages back, some of us discovered the same issue. Including myself.
I tried to enter the container and to run: apt-get systemd
to set the correct time. Another option would be installing: apt-get tzdata.
There is also an open Pull request on the official chia docker: https://github.com/Chia-Network/chia-docker/pull/50
Not sure if it will solve the issues tbh. Installing it inside the container is only a quick solution, which will be lost when ever the container is rebuild. So not the best solution. Additionally to that, it didn't fix my sync status. I am now kind of more than one hour off and it never reaches the synced status. So it seems to be something different causing these issues. So you can try to get you clock right, but I can't promise it will solve your sync issues.

Btw, the pull request is mine 😂. You could approve it, maybe it will draw attention from the repo maintainer ...

Edited by Gnomuz
Link to comment
1 hour ago, dandiodati said:

Thanks for the details just one thing I'm not clear on is that with this command line how to you chose your plotting sizes ? Basically I saw in the UI the ability to choose different plotting sizes but with this command line tool does not seem to indicate it. I need to determine how many I can run on a SSD and not run out of space?

it will default to the 32k one or what ever it is using about 300ish GB for a plot with final size being about 106GB

Link to comment

@Trinity You seem to have had exactly the same issue that I'm having. My farm is farming, my current blockchain status is "Full Node Synced", but my Wallet status is "Not Synced". Did you or anyone else find a solution in the end?

 

image.png.618552f87b98129c06a335c16370218a.png

image.png.520d4c7ba3cc76f1e1007cbadba8247b.png

image.png.52c673a47c9b2392bde0ce3958347364.png

 

Only thing I can think of is that I created a wallet on the Windows client, and added it to the docker via the mnemonic.txt method, rather than generating in the docker its self.

 

 

Link to comment
Posted (edited)

Hey @Partition Pixel I am having an issue with this docker failing to sync to the chia blockchain due to the time inside the docker being incorrect. Do you have any ideas on how I can solve this?

I tried changing the time with this but it wouldn't let me. As you can see the time is out by more than 10 hours

root@BigRig:~# date
Mon May 17 01:27:31 AEST 2021
root@BigRig:~# docker exec -it chia /bin/bash
root@1afebfbb2b67:/chia-blockchain# date
Sun May 16 15:27:37 Australia 2021



Thanks!

Edited by danthekilla
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.