Chia farming, plotting; array and unassigned devices

[TexasUnraid]

18 minutes ago, sota said:

and I'm kind of 2 minds on pools.

since I'm about to split out 8 of the 16 drives from the one machine (since it can't effectively use them) and stick them in another identical machine, I may split my plotting between personal and pools.  i dunno yet.

 

With how the network is growing, I am going 100% into pools but I can see why someone would solo mine as well. I just fear solo mining for a year only for the network to grow to the point it takes 10 years to win vs getting paid regularly from a pool.

 

I am guessing the network goes over 100-200Eib in short order, not sure where it will level off at though. I am basing most of my calculations around that number for the time being.

[sota]

I guess the way i'm reading it, I'll keep plotting solo for the time being, then when pools/portable plots become available, switch over to portable plots.  That way I can either keep them solo, or join them to a pool, or move them around as i see fit/desire.

Once i fill up my current space, I'll replot/replace the earlier solo-only ones.  I'm not going to wind up with an ever expanding array of disks, I don't think.

 

So for a period of time, I'll be solo and posslbly pooled at the same time.

Edited June 2, 2021 by sota

[TexasUnraid]

7 minutes ago, sota said:

I guess the way i'm reading it, I'll keep plotting solo for the time being, then when pools/portable plots become available, switch over to portable plots.  That way I can either keep them solo, or join them to a pool, or move them around as i see fit/desire.

Once i fill up my current space, I'll replot/replace the earlier solo-only ones.  I'm not going to wind up with an ever expanding array of disks, I don't think.

 

So for a period of time, I'll be solo and posslbly pooled at the same time.

 

Ah, ok yeah I think that is the plan for a lot of people. I will be doing the same thing. Why waste these plots until I need the space.

[sota]

I see one post over on reddit, that costco will take HDDs back up to a year, instead of 90 days since it's not a computer.

Next time I go over there with a buddy of mine, I should see about talking to the GM of that store. Find out what happens to such returns (do they get binned? manufacturer return? etc.) and if there isn't some way to get drives at pennies on the dollar.

I need space for things OTHER than chia, hence why I already HAVE like 60 4TB drives here.

[sota]

revamped my plotter setup. only running the three DL380G6 boxes now. 2x8 and 1x4 plot sets.  looking like 20 plots every 24 hours or so.

[TexasUnraid]

1 minute ago, sota said:

revamped my plotter setup. only running the three DL380G6 boxes now. 2x8 and 1x4 plot sets.  looking like 20 plots every 24 hours or so.

 

Not bad at all, most people are happy with 6-10 plots a day while burning SSD's.

 

With pools apparently being delayed I am debating joining hpool even as much as I hate the idea of joining a china pool.

 

I found another set of SAS HDD's, a lot more expensive but should arrive Tuesday, gonna wait for that to do much more.

 

unixsurplus on ebay Is screwing me, they clearly have the drives in stock and listed for sale at higher prices but are refusing to ship my order.

 

They also screwed me on shipping for some cables I ordered, $30 shipping for what cost maybe $6 to ship and I messaged them to let them correct it but just got "that is our shipping price" as a reply each time.

[sota]

I could get to 24/day, but i don't have enough disks to put into the one system.

[sota]

https://www.ebay.com/itm/373600276463

 

plotting engine?

[TexasUnraid]

32 minutes ago, sota said:

https://www.ebay.com/itm/373600276463

 

plotting engine?

lol, interesting ideal though. Good price for what it is.

 

They are actually not too far from me, I considered taking a road trip and picking up one of their supermicro systems at one point but decided to hold out for the larger 4u system.

 

I have always avoided HP/Dell etc servers, too propriety for me. I like supermicro simply because they are pretty universal in my experience. They do cost a bit more though due to that, feature for feature.

[sota]

I'm a die hard HP guy. my Production server is an HP, my unRAID box is an HP (as my sig shows  ), all my desktops and laptops are HP, even my mopier is an HP (just worked out that way on that one.)  I get the proprietary problem though: for example there's only ONE USB 3.0 card I've found that works in an HP DL380G6 and up boxes, without pissing off iLO such that it turbo boosts all the fans (PEXUSB3S44V). Also had a couple SATA HDDs that would I guess report invalid temperature data to iLO (it would report the drives as overheating almost immediately), also pissing it off and cranking up the fans.  Is that really an HP server fault though? Or more likely a drive manufacturer being silly/proprietary.  The fix was to get some HP branded ones, that actually didn't cost any more than the non-HP stickered units, and that problem was solved.  I balance those... quirks... with the fact that in the past 17 years of having HP servers for my production work, they've NEVER failed of their own volition.  The lightning strike a block over that blew out half of a ProLiant 1850R, fried a UPS, and melted down the DSL modem, doesn't really count.  

[TexasUnraid]

I can understand that as well and from a professional data center point of view I have no issues with HP or Dell for that matter.

 

My use cases are basically always home lab/cheapest poop I can find that will do the task. I always seemed to regret buying dell and HP years ago as using them with other old/cheap gear I came across always seemed to cause something to complain.

 

Supermicro on the other hand just seems to work with whatever I throw at it. So it kinda won out by default. I don't have time to research every cheap/old item I purchase to make sure it will work lol.

 

I will say that the H240 and HP expanders I have work fine, HP makes good stuff.

[TexasUnraid]

Well the replacement 300gb 10k drives showed up today so decided to make my server cry, hard lol.

 

It is loaded up with 25 drives right now, 10x 300gb sas running in a raid1 with 12 plots total as a stating test, going to be bandwidth limited on the backplane for those (sas 1 backplane).

 

The rest of the drives are a mix of old drives I had laying around. Running 2x plots on all drives 500gb and larger, single plot for smaller drives.

 

All plots started with 4 threads, 7gb memory, 64 buckets, 4 or 6 threads (switched at some point lol).

 

All told there are 34 plots running or set to queue in over the next few hours.

 

CPU is maxed during phase 1, drive IO is limited by the backplane for sure for the 10k drives, I have another backplane, might swap them in as that would net me an extra ~600-700mb/s bandwidth.

 

Memory should be fine with 224gb cutting it close but highly unlikely they will all need peak memory usage at the same time.

 

Putting a hurting on the system that is for sure, fans are screaming for mercy lol.

 

I know I could optimize this but honestly just want to get a baseline at this point with some "basic settings"

 

My main sever is a bit more powerful, I can move drives around and spread the load to optimize speed once I settle on a setup. My limit will be plotting drives though, just not enough of them,

[TexasUnraid]

I can tell I will need to change the timing around to maximize things for sure. Looks like around 16 plots maxes out the CPU, so if I staggered them to start more after the first ones passed phase one I am sure that would help a fair amount.

[sota]

CHIA1.11.20210603.2224.log 	6/3/2021	22:24:34
CHIA1.12.20210603.2241.log 	6/3/2021	22:41:12
CHIA10.11.20210603.2253.log 	6/3/2021	22:53:54
CHIA1.13.20210603.2257.log 	6/3/2021	22:57:54
CHIA10.12.20210603.2310.log 	6/3/2021	23:10:30
CHIA1.14.20210603.2314.log 	6/3/2021	23:14:33
CHIA10.13.20210603.2327.log 	6/3/2021	23:27:12
CHIA1.15.20210603.2331.log 	6/3/2021	23:31:13
CHIA10.14.20210603.2343.log 	6/3/2021	23:43:51
CHIA1.16.20210603.2347.log 	6/3/2021	23:47:53
CHIA10.15.20210604.1200.log 	6/4/2021	00:00:31
CHIA11.11.20210604.1202.log 	6/4/2021	00:02:50
CHIA1.17.20210604.1204.log 	6/4/2021	00:04:34
CHIA10.16.20210604.1217.log 	6/4/2021	00:17:11
CHIA11.12.20210604.1219.log 	6/4/2021	00:19:28
CHIA1.18.20210604.1221.log 	6/4/2021	00:21:14
CHIA10.17.20210604.1233.log 	6/4/2021	00:33:51
CHIA11.13.20210604.1236.log 	6/4/2021	00:36:08
CHIA10.18.20210604.1250.log 	6/4/2021	00:50:32
CHIA11.14.20210604.1252.log 	6/4/2021	00:52:48
CHIA10.11.20210604.1703.log 	6/4/2021	17:03:34
CHIA1.11.20210604.1731.log 	6/4/2021	17:32:00
CHIA11.11.20210604.1811.log 	6/4/2021	18:11:16
CHIA1.17.20210604.1926.log 	6/4/2021	19:26:30
CHIA11.12.20210604.1949.log 	6/4/2021	19:49:40
CHIA11.13.20210604.1210.log 	6/4/2021	20:10:23
CHIA10.12.20210604.1216.log 	6/4/2021	20:16:39
CHIA11.14.20210604.1224.log 	6/4/2021	20:24:50
CHIA10.13.20210604.1250.log 	6/4/2021	20:50:22
CHIA1.12.20210604.1251.log 	6/4/2021	20:51:24
CHIA10.14.20210604.2323.log 	6/4/2021	23:24:47

 

the drift between threads are hilarious.

Edited June 6, 2021 by sota

[TexasUnraid]

those outliers are interesting.

 

Well looks like I went a little overboard, learned that a 250gb drive is not enough after formatting to hold a single plot.

 

A 500gb drive is not enough for 2 plots IF started at the same time.

 

If staggered then 2 plots can be run on a single 500gb (at least so far, will see if it grows anymore, sitting around 400gb of usage right now).

 

Once phase 1 is finished ~30 plots actually seems to be working out the hardware pretty good. CPU is sitting between 85%-100% most of the time.

 

Memory usage seems to of peaked at around 165gb and settled down to between 120-160gb for the last several hours.

 

Drive usage is healthy, I am bandwidth limited as expected but not as much as I expected. Gonna juggle some drives around and might be able to make it work with the sas1 backplane by moving my slower drives to it (it is setup as a jbod) and moving the 10k drives to the native backplane.

 

It is also looking like using each 10k drive individually will be faster then raiding them even though I will have to run less plots in parallel and it will have a lot of idle time. It is possible I could run 3x plots on 600gb with the right stagger, if so then raiding 2x drives together might work out.

[sota]

Other than the "virgin" kick off, I'm not bothering with trying to control staggering.  I'm going to run 8/8/4 for now, let them auto cycle.  Then there's the "incident" from last night where the storage drive on the farm filled up, and I hadn't re-pointed the 'latest' share, so that the all 3 machines were "backed up" a little.  I'm going for Chaos Theory now.

[TexasUnraid]

Just now, sota said:

Other than the "virgin" kick off, I'm not bothering with trying to control staggering.  I'm going to run 8/8/4 for now, let them auto cycle.  Then there's the "incident" from last night where the storage drive on the farm filled up, and I hadn't re-pointed the 'latest' share, so that the all 3 machines were "backed up" a little.  I'm going for Chaos Theory now.

I will be using plotman once I get it all setup which handles staggering automatically, you just have to select the stagger you want.

 

 

[sota]

how does it handle the fact that not every plot is taking the same amount of time, and they seem to be random as to which process/processor it touches.

[TexasUnraid]

2 minutes ago, sota said:

how does it handle the fact that not every plot is taking the same amount of time, and they seem to be random as to which process/processor it touches.

All of the plot managers stagger based on phases instead of time like the stock gui.

 

They will start a new parallel plot when the first one reaches the 2nd phase and another plot when the first reaches phase 4 for example or however you set it up.

 

I just have not taken the time to settled on formats and names for all my drives so easier to test with the GUI for the moment.

Edited June 6, 2021 by TexasUnraid

[sota]

ah, I see.

I think now that I have my scripts in a pretty stable state, i'm just going to run with them and see how things go.

IF, I don't manage to mess anything else up any time soon, it looks like i'm doing about 20 plots per day, with the 3 plotter machines.

[TexasUnraid]

2TB per day is not bad at all. Not sure how much space you have to farm from. I have around 30-40TB of old drives I don't care about to toss at it at last count.

 

I have an order for some new 14tb drives I will be using for backups, I am backing up 12TB drives, so I could combine things to free up a few 14tb for extra space if it is worth it.

 

hpool freaks me out but if real pools are delayed awhile it might be the only choice.

[TexasUnraid]

In other news, let the plots keep running and learned some things. With proper staggering it looks like you can keep the max disk space used down to 400-420gb for 2 plots. I think with a proper plot manager 400gb should be doable meaning I just might be able to have 3 plots in parallel with 600gb. Worth a shot anyways.

 

A single plot does max out just under the 239gb they advertise but after formatting a 250gb drive only has like 234gb free, so not quite enough for a single plot. I will raid those drives together and try multiple plots to see what happens.

[sota]

don't forget though, if you're in the back half of the disk, its data transfer rates slow down.

[TexasUnraid]

In the process of setting up my system in a more long term arrangement.

 

I was looking at the docker I will be using and noticed that they added this

 

Quote

After testing showed between 15% to 30% speedup on plotting, adopting Chiapos multithreaded library.

 

Curious to see what effect that has on plot speed, think I am going to switch over to that docker for plotting now. Also thinking I will go ahead and start plotting and fill up some of my spare drives and then either setup a separate machine or a VM to farm them on hpool.

 

Figure I might as well get something for my efforts before the netspace gets so big you earn nothing at all.

Edited June 7, 2021 by TexasUnraid

[TexasUnraid]

6 hours ago, sota said:

don't forget though, if you're in the back half of the disk, its data transfer rates slow down.

 

This is true, at the slowest speeds of the drives I would only be gaining a little bit from a faster backplane.

 

I split the drives up to individuals today and going to run another test with plotman to see what the results from there are.