[Support] Streetmerchant


29 posts in this topic Last Reply

Recommended Posts

Overview: Support for Docker image streetmerchant in the mitchellriley repo.

Application: streetmerchant - https://jef.codes/streetmerchant/

Docker Hub:https://hub.docker.com/repository/docker/mitchellriley/streetmerchant

GitHub:https://github.com/kiowadriver/unraid-docker/tree/master/mitchellriley

Documentation: https://jef.codes/streetmerchant/

 

If you appreciate my work, then please consider buying me a beer via paypal.me

 

This docker is stock checker bot used to rapidly track the online availability of hard to find items such as GPUs, CPUs, Gaming Consoles, and more.  The maintainers are constantly adding new stores and products.  With this bot you can have a better chance at getting to those hard to find parts before they go out of stock again.

 

By running this docker you can alert your community via discord, twitter, SMS, email, pushbullet, twitch, MQTT, Phillips Hue (lights blinking), Pushover, Slack, and Telegram.  This docker sends a notification and a link to add the item to your cart.  Good luck hunting!

 

Usage

  1. Firstly you need to be running unRAID ver 6.0.0 or later, once installed follow the instructions below:-
  2. Navigate to "Docker" tab and then the "Docker Repositories" sub-tab in the unRAID webui
  3. Enter in a URL of https://github.com/kiowadriver/unraid-docker in the "Template repositories" field
  4. Click on the "Save" button
  5. Click back to "Docker" tab and then click on the "Add Container" button
  6. Click on the "Template" dropdown menu and select the desired Docker image
  7. Click the "Advanced View" toggle on the top right and fill in required fields e.g. volume data, environment variables etc
  8. Click on the "Create" button at the bottom of the window to begin pulling down the Docker image
  9. Once the image is downloaded you should see it appear in the "Docker Containers" sub-tab
Edited by kiowa2005
Link to post

I used the web interface to select the test:model, test:series, and test:brand.  The system sent out the screenshots over the notification mechanism but the web interface gave me the same error.  I looked at the png's via the console and it may be the group permission.  The important thing is the notifications work.  The web screen shots are just an archival reference.  I will take a look into more detail and see if I can get the issue resolved.

 

I don't think that mapping the /apps directory as a volume will result in the desired operation.  The /apps directory includes all of the web interface data as well as the models.  If this volume were mapped the system would not benefit from updates in the future.  Instead I think I need to work out the permission issue in the Docker file.  Also, I don't think mapping the volume will resolve the permission issue.  391957094_Screenshot2020-12-25at4_27_13PM.png.9e5822f0dbe8a0ccc95e4b97c3a1e09d.png2038262159_Screenshot2020-12-25at4_32_29PM.thumb.png.69414aab60ce341269d85108a1d71992.png

Link to post

Thanks for making this docker. Its really cool. I noticed an issue and unsure how to get past it. My log is showing this

 

[3:47:02 PM] [33mwarn[39m :: ✖ [amazon] [evga (3090)] ftw3 ultra :: CAPTCHA

All of the Amazon log entries show CAPTCHA. Is there a way to get around it?

 

Thanks!

Link to post
Posted (edited)

The default setting is for the browser to be trusted so make sure to leave that as default. If amazon is giving you captcha then you may be hitting it too frequently. Trying playing with the timing setting to back off on the frequency.

 

Also, I use mine in conjunction with bin-hex delugevpn and I use the delugevpn as my proxy with a vpn. This way I am not hitting the stores all the time from my home IP address so when I go to buy I am not coming from the same address all of the stock checking originated from. The only thing this does is ensure my home IP is not in any danger of a temp blocked by the stores for spamming them too hard.


I pretty much use the default timing.  However, going through a VPN perhaps that makes a difference (mine being slower).

 

image.thumb.png.02571190defbf5a293f89c250fee8433.png

 

 

Edited by kiowa2005
Spelling
Link to post
On 12/26/2020 at 11:48 AM, milfer322 said:

You can help me to send notifications to discord? i dont understund how works the notifications.

Thanks for you job!

If you go to this address you will see all of the variables supported by the discord notification:

https://jef.codes/streetmerchant/reference/notification/#discord

 

If the variable you want to define is not listed in template just add it as a variable with the name and value set as the "Environment Variable" defined on this page.  As far as the group values, I assume you have that information from your discord group you plan to send notifications to.

Link to post
On 12/28/2020 at 3:04 PM, cyberspectre said:

Very cool! Now, if only we could have auto checkout. 🤩

I agree. I am working with the team to get the add-to-cart working, but first I would like to see this moved over to a database driving configuration managed via web interface instead of the heavy environment variable use.  Once that is complete, the stores can each be supplied with login information from your configuration to enable add-to-card, the final step would be some logic for checkout if that is permitted. 

Link to post
4 hours ago, kiowa2005 said:

The default setting is for the browser to be trusted so make sure to leave that as default. If amazon is giving you captcha then you may be hitting it too frequently. Trying playing with the timing setting to back off on the frequency.

 

Also, I use mine in conjunction with bin-hex delugevpn and I use the delugevpn as my proxy with a vpn. This way I am not hitting the stores all the time from my home IP address so when I go to buy I am not coming from the same address all of the stock checking originated from. The only thing this does is ensure my home IP is not in any danger of a temp blocked by the stores for spamming them too hard.


I pretty much use the default timing.  However, going through a VPN perhaps that makes a difference (mine being slower).

 

image.thumb.png.02571190defbf5a293f89c250fee8433.png

 

 

Thanks for the info and Tip. I was going to use a proxy, but discovered the proxy settings do not include a username/password field. Are there any variables I can pass to the docker container? If so, could you let me know what those are? I tried looking around on the github page, but wasn't able to find anything. (maybe I'm blind) :)

Link to post

@kiowa2005 Thanks for this!

 

Got this working today with a Telegram channel and bot I set up.  Took about an hour and now I'm not getting stupid PS5 notifications from those "other" stock notifiers pushing to Discord/Twitter/Telegram 🤩.

 

I was monitoring my log file and even though I specified asus as a store endpoint it doesn't seem to be being checked for 30 series cards.  I know this has nothing to do with the container, but has anyone else seen that on their installs?

 

I'm also getting the CAPTCHA for amazon.  The store as a whole seems to 'greylist' you ... so PAGE_BACKOFF_MIN doesn't seem very helpful as you just nail it with requests for another product.  For now, I just turned off amazon ... I'll play with it when I'm not on there 💩 list.  It would be nice if there were different back off periods for different stores and store product pages ... but again, this is not the containers fault.

Edited by JesterEE
Link to post
On 1/1/2021 at 11:58 PM, zepheris said:

Thanks for the info and Tip. I was going to use a proxy, but discovered the proxy settings do not include a username/password field. Are there any variables I can pass to the docker container? If so, could you let me know what those are? I tried looking around on the github page, but wasn't able to find anything. (maybe I'm blind) :)

I haven't tried it, but the standard way to pass username and password fields to a proxy is:

protocol://user:password@proxy.server.com:port

 

so for example

 

https://username:p455w0rd@proxy.someproxyserver.com (you probably don't need the port if it's the standard port for the protocol)

Edited by JesterEE
Link to post

it is being pulled from the originator's main each week to keep it up to date with the latest products.  Discord seems to be the fastest notification as email and SMS are a bit delayed when routing through the email servers.

Link to post
2 hours ago, JesterEE said:

I haven't tried it, but the standard way to pass username and password fields to a proxy is:

protocol://user:password@proxy.server.com:port

 

so for example

 

https://username:p455w0rd@proxy.someproxyserver.com (you probably don't need the port if it's the standard port for the protocol)

This container assumes you are running another proxy service container.  I use delugeVPN proxy with PIA VPN.  You ensure this container is on the same network as your proxy container, include the IP and port to that proxy container.  Many other proxy container options out there.

Link to post
2 hours ago, JesterEE said:

@kiowa2005 Thanks for this!

 

Got this working today with a Telegram channel and bot I set up.  Took about an hour and now I'm not getting stupid PS5 notifications from those "other" stock notifiers pushing to Discord/Twitter/Telegram 🤩.

 

I was monitoring my log file and even though I specified asus as a store endpoint it doesn't seem to be being checked for 30 series cards.  I know this has nothing to do with the container, but has anyone else seen that on their installs?

 

I'm also getting the CAPTCHA for amazon.  The store as a whole seems to 'greylist; you ... so PAGE_BACKOFF_MIN doesn't seem very helpful as you just mail it with requests for another product.  For now, I just turned off amazon ... I'll play with it when I'm not in there 💩 list.  It would be nice if there were different back off periods for different stores and store product pages ... but again, this is not the containers fault.

I have not seen that behavior.  I usually only monitor newegg, amazon, bestbuy, sony, walmart.

 

Here are the cards the ASUS store supports.  If you have additional product URL's those can be added, just let me know.

 

3080

https://store.asus.com/us/item/202009AM160000001

https://store.asus.com/us/item/202009AM150000004

https://store.asus.com/us/item/202009AM290000002

 

3090

https://store.asus.com/us/item/202009AM150000003

https://store.asus.com/us/item/202009AM150000001

Link to post
2 minutes ago, kiowa2005 said:

I have not seen that behavior.  I usually only monitor newegg, amazon, bestbuy, sony, walmart.

 

Here are the cards the ASUS store supports.  If you have additional product URL's those can be added, just let me know.

 

3080

https://store.asus.com/us/item/202009AM160000001

https://store.asus.com/us/item/202009AM150000004

https://store.asus.com/us/item/202009AM290000002

 

3090

https://store.asus.com/us/item/202009AM150000003

https://store.asus.com/us/item/202009AM150000001

Oh there are definitely other cards...I'll get back to you in a while with a list of links.

 

Are you in the dev team? Do you want me to track this elsewhere (github)?

Link to post
Posted (edited)
3 minutes ago, JesterEE said:

Oh there are definitely other cards...I'll get back to you in a while with a list of links.

 

Are you in the dev team? Do you want me to track this elsewhere (github)?

If you could generate an issue here against the asus.ts model with your new links somebody can get those updated.  I keep the unraid version up to date frequently.  I am in the process of fixing some newegg and bestbuy definitions as well.

https://github.com/jef/streetmerchant/issues

 

https://github.com/jef/streetmerchant/blob/main/src/store/model/asus.ts

Edited by kiowa2005
Link to post
On 1/7/2021 at 3:45 PM, kiowa2005 said:

This container assumes you are running another proxy service container.  I use delugeVPN proxy with PIA VPN.  You ensure this container is on the same network as your proxy container, include the IP and port to that proxy container.  Many other proxy container options out there.

Thats what I was missing. Thanks. I installed a proxy container and got it working.

 

Thanks again!

Link to post
On 1/7/2021 at 5:45 PM, kiowa2005 said:

This container assumes you are running another proxy service container.  I use delugeVPN proxy with PIA VPN.  You ensure this container is on the same network as your proxy container, include the IP and port to that proxy container.  Many other proxy container options out there.

@zepheris @kiowa2005

 

Using the built in proxy support also works: https://jef.codes/streetmerchant/reference/proxy/

You just have to map the files corectly to the docker:

For example: -v '/mnt/user/appdata/streetmerchant/global.proxies':'/app/global.proxies':'ro'

 

I use this for amazon so I don't hit the servers too hard and start forcing CAPTCHA requests.

 

-JesterEE  

Link to post
  • 2 weeks later...

@kiowa2005 This project is updated very regularly.  Updating links, adding features, fixing bugs, etc. Is there a way to structure the dockerfile such that users can force an update so the image can pull from the github repo and run the latest code?  

Link to post

Normally dockerhub can automate builds based on github changes.  However, dockerhub does not support two stage automated builds at this time.  I am automating the process to update weekly but more steps are involved on my end.  Will try to get an update pushed out this weekend.

Link to post
22 minutes ago, D0N2k said:

Hey. Thanks for the great project.

Is it normal that the container is using between 7-9GB of Ram?

 

Something changed in the last update.  Not sure what, but this wasn't happening before.  I just checked my docker and after a few days or running, and streetmerchant was eating all my RAM.  I started limiting the RAM consumption of the docker now.  We'll see how this works out.

 

 

Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.