Chia farming, plotting; array and unassigned devices


Shunz

Recommended Posts

Well, after spending the last week chasing issues I am starting over and starting the blockchinas one by one to see what is causing the crashes.

 

I know it can run stable without crashing as I have randomly had it happen but could not find the common denominator that is causing the crashes.

Link to comment

Well, this sucks. The boot SSD for my farming machine just died.

 

On the plus side I think that it might of been failing for awhile and causing my crashes.

 

Now I get the fun of re-downloading all those blockchains. That will take weeks lol.

 

Wish there was a way to download a monthly updated blockchain directly or something to at least get a head start.

 

Thinking about spinning up a VM on my server to help with the load.

Edited by TexasUnraid
Link to comment

Here is what half the blockchains just trying to sync up the wallets and remaining blocks from that site you linked looks like on a 20C/40T server, it is getting HAMMERED. Pulling 500w from the wall lol.

 

It would be working even harder but it is IO limited since the VM is running on some 10k drives.

 

firefox_wDZtuk4v57.thumb.jpg.ac407d68e2379275d69a37d512165dea.jpg

Link to comment

Yeah, once synced it doesn't take much to stay synced.

 

Indeed, I will be taking a backup on a regular basis from now on as well.

 

Been sitting at 48TB for awhile but looks like I am going to have to pull my 12TB drive for proper service soon so will drop me down to 36TB. Just can't justify putting nice drives into chia at this point.

Edited by TexasUnraid
Link to comment

Got some pretty intensive video decode/encode processes I want to do here.  thinking about installing this AMD FirePro graphics card, with the hopes of passing it through to a VM to run these FFMPEG tasks quicker.

Link to comment

did you notice the quality issues when encoding to a video stream? or when decoding to still images?

I need to post-process a buttload of security footage, and yank 1 frame per second, to then later re-encode that back as a video file for quick scanning and archival processes.  I can live with the slower encode times via CPU, but the decode times for each file is pathetic, and that's a problem.  I can run up to 4 in parallel right now but that doesn't help the fact it's still days to extract frames from all the footage.

Edited by sota
Link to comment
1 hour ago, sota said:

did you notice the quality issues when encoding to a video stream? or when decoding to still images?

I need to post-process a buttload of security footage, and yank 1 frame per second, to then later re-encode that back as a video file for quick scanning and archival processes.  I can live with the slower encode times via CPU, but the decode times for each file is pathetic, and that's a problem.  I can run up to 4 in parallel right now but that doesn't help the fact it's still days to extract frames from all the footage.

 

Decoding should be better, I mostly noticed issues when encoding.

 

Although odd you have issues with CPU decoding, decoding is not generally very intensive.

 

That said, I would test it on a small section of videos first, depending on the footage you might not save as much space as you would think doing this. If the shot is mostly static H264 encoding is already quite good at optimizing the file size.

Link to comment

aye the issue is the decoding/extracting frames takes a brutally long time.

each storage drive is:

4 cameras (currently. more to come hopefully)

13 days (roughly)

24/7 recording

means I need to extract 60*60*24*13*4=4,492,800 frames total.  then recombine them back into an MP4 @ 30fps.  I'm fine with leaving the encoding part as CPU based, as that's usually running about 15fps, and once I have all the extracted frames I can easily batch process that to run.  That'd take 83.2 hours, or just about 3.5 days continuous, to process.  I'm fine with that part.  it's the extracting that takes over a week to do.

Yea I know shinobi does timelapse, but it sucks at 6 seconds per frame, so it's easy to miss things going by the cameras.  And I want the 1fps files so I can delete the "useless" footage, as I go through and tag only the time stamps that have interesting things going on in front of the cameras, but still have a useful continuity file if I ever needed this stuff as evidence.

Link to comment

Yeah, for decoding hardware should work for your use case. I never really cared about the decoding side of things.

 

Encoding on the other hand it was not uncommon to have to crank the bitrate 2-3x as high to match the CPU encoded files quality at naturally 2-3x the size. I have tried it several times over the years but always decide it is just not worth the time savings.

 

I still say that if the images are largely static, test it on a small file to start with and see what the end result is space wise.

 

I have had OBS videos of game loading screens with just minor movement only take up a few mb of space when the same time of in game footage can be 1-2GB.

Edited by TexasUnraid
Link to comment

That sucks, I had a scare last week with my main server suddenly getting errors on my most important drive. Thankfully I had a backup but I am out of spare drives so it would of still been bad if it had died.

 

Thankfully it seems to of been something besides the drive, I rebooted the system and scanned the drive again and it came back basically clean, 1 error in a mundane file but the rest of the files that had errors before were fine now.

 

So looking like it was a SAS card / backplane / cable issue. Gonna be keeping my backups updated a bit more regularly now though.

Link to comment

Still almost double what I am running.

 

On the plus side with the new SSD things are running much better so far, have most of them running now. Can't get taco and apple to sync but the rest seem to be working. Still syncing a few so not 100% sure.

 

Lucky, Silicoin and wheat are not working at all for me.

Link to comment
On 9/1/2021 at 11:29 AM, sota said:

*sigh*

looks like 2 more disks on farm1 and 1 disk on farm2 might be in trouble.

 

I guess look at the bright side, at least the failed before being put into real use.

 

This is why I decided to go with shucking drives vs used drives, I was just too scared of random drive failures and unraid doesn't handle failures efficiently at all.

Link to comment
  • 4 weeks later...

well, a month later and still farming. Had some real issues with getting corrupted blockchains and having to rebuild. Sometimes downloading from that site works and others it doesn't for some reason.

 

This system is also mining with my old GPU's and it has been crash prone while dialing in the settings.

 

I finally setup a VM on my main server with all the blockchains installed so I can update on there where there is lots of horsepower and then just copy over a fresh version if something goes wrong on the farmer.

 

Not sure how much longer I will farm though, had a few cold nights and could be hard to keep the drives warm enough in the garage. When I was looking up acceptable drive temps years ago I found that too cold of drives was bad, I like to keep them between 30-35c.

 

Getting close to 1xch mined at this point, thinking about getting an even 1 xch and calling it but will see what happens at that point.

 

Have you hit anymore XCH solo?

Edited by TexasUnraid
Link to comment

still at the 2XCH, and a monster amount of some of the sh!tcoins.  farms haven't dropped any more drives lately.  I'm not worried about them in the garage and the cold.  in fact, they might keep the entire garage warmer this winter. :D

 

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.