Download issues

[neilt0]

It's 2013. 30MB is not big! Not even for http. I think I've even downloaded a copy of an OS disc (1GB, 4GB?)  direct from Microsoft using a http connection. Years ago.

[nars]

I received a reply from midPhase support this morning:

 

As this is something that is working for some of your visitors, and not working for some others, and the traceroutes I was looking at seem to be able to make a successful connection to your account, the main issue appears to be with the timeouts on the server end or with the client's ISP while trying to download this file. Have you been able to try adding these files to your vps as opposed to the shared server account? There are many ways to have these files downloaded, such as using Anonymous FTP, breaking the files up into more than one file, and having two different files to download, or using things like Dropbox. As these files are pretty large, if there is a connection to the server using http, there are some timeouts in place that may end the downloading of this file when it is being requested by one of your website visitors. If you feel this is going to take more than 30 seconds for your client's to download, you may want to consider some of the alternative suggestions above.

 

Basically they are saying, "go find another download server".

 

The conclusion I take from their reply is: they don't bother or even don't really like such "pretty large files" downloaded from their shared hosting - not uncommon for cheap shared hosting... then most probably they will not do anything to sort it. And a hosting company suggesting you Dropbox is really...

 

Why not a vps for download server? or even... wouldn't it be worth getting an unmanaged dedicated server, probably not just for downloads, but you could also host mainsite... forum... etc... there are pretty decent ones for reasonable prices nowadays.

 

If you don't mind telling... how much traffic/month unraid downloads do?

[limetech]

If you don't mind telling... how much traffic/month unraid downloads do?

 

The download subdomain eats up around 180 GB/month.  The main site uses around 250 GB/month, which seems inconceivable to me, but appears most of it is consumed by the forum.  When both the downloads and rest of the site were on the same host we were constantly hitting up against the bandwidth limit, which is why they are hosted on two separate servers.  I'd set up my own server here, but our building only has crappy DSL, though "sometime in the spring" we're supposedly getting fibre in here.

[zoggy]

If you don't mind telling... how much traffic/month unraid downloads do?

 

The download subdomain eats up around 180 GB/month.  The main site uses around 250 GB/month, which seems inconceivable to me, but appears most of it is consumed by the forum.  When both the downloads and rest of the site were on the same host we were constantly hitting up against the bandwidth limit, which is why they are hosted on two separate servers.  I'd set up my own server here, but our building only has crappy DSL, though "sometime in the spring" we're supposedly getting fibre in here.

 

i might would suggest moving the downloads to sourceforge or some file hosting type site? (leverage other peoples infrastructure and mirrors).

also you prob dont want your forums (prone to hacks) and downloads being on the same box.... aka someone hacks forums and manages to get access to the downloads and plants malware or something like that. and if your hosting provider is telling you to go else where for the 30 mb download (taking longer to transfer before server is configured to wait for).. then you prob want to go else where anyways..

[doorunrun]

...The main site uses around 250 GB/month, which seems inconceivable to me, but appears most of it is consumed by the forum...

This probably says a lot about how good the community support is here! I'll admit I'm hooked; put me down for, oh let's say, 500MB/month at least 

[StevenD]

No issues here.

 

D:\>tracert download.lime-technology.com

Tracing route to download.lime-technology.com [173.255.128.88]
over a maximum of 30 hops:

  1    <1 ms    <1 ms    <1 ms  router1.xxxxxxxxx.org [10.1.1.2]
  2     <redacted>
  3    20 ms    11 ms    14 ms  g0-5-1-2.dllstx-lcr-21.verizon-gni.net [130.81.215.202]
  4     5 ms     7 ms     7 ms  so-5-0-0-0.dfw9-bb-rtr1.verizon-gni.net [130.81.199.34]
  5    16 ms     9 ms     7 ms  0.xe-3-0-2.br1.dfw13.alter.net [152.63.98.53]
  6    32 ms     8 ms     9 ms  xe-4-1-0.er2.dfw2.us.above.net [64.125.13.25]
  7     8 ms     9 ms     9 ms  ae2.cr2.dfw2.us.above.net [64.125.27.81]
  8    14 ms    15 ms    15 ms  ae3.cr2.iah1.us.above.net [64.125.21.138]
  9    42 ms    41 ms    59 ms  ae2.cr2.lax112.us.above.net [64.125.25.53]
10    56 ms    62 ms    56 ms  ae2.mpr1.slc1.us.above.net [64.125.30.137]
11    62 ms    67 ms    64 ms  208.184.34.186.ipyx-068416-004-zyo.above.net [208.184.34.186]
12    60 ms    72 ms    64 ms  206.130.126.10.west-datacenter.net [206.130.126.10]
13    64 ms    63 ms    62 ms  206.130.126.69.west-datacenter.net [206.130.126.69]
14    66 ms    68 ms    62 ms  download.lime-technology.com [173.255.128.88]

Trace complete.

[DaleWilliams]

And a hosting company suggesting you Dropbox is really...

What's the emoticon for "that's a knee slapper!"

[Hoopster]

Never had a download issue, but, as others have pointed out, my connection is fast enough that the download is complete before the hosting server can randomly time out.

 

Tracing route to download.lime-technology.com [173.255.128.88]
over a maximum of 30 hops:

  1    <1 ms    <1 ms    <1 ms  router.asus.com [192.168.0.1] 
  2     1 ms     1 ms     1 ms  pub-198-91-48-29.sumofiber.com [198.91.48.29] 
  3     2 ms     1 ms     1 ms  121-109-36-199-reverse.instavps.net [199.36.109.121] 
  4     2 ms     3 ms     1 ms  69.31.64.1 
  5     3 ms     3 ms     3 ms  as29854.xe-0-2-0.cr2.slc1.us.nlayer.net [69.22.153.70] 
  6     2 ms     3 ms     2 ms  206.130.126.10.west-datacenter.net [206.130.126.10] 
  7     4 ms     4 ms     4 ms  206.130.126.69.west-datacenter.net [206.130.126.69] 
  8     4 ms     4 ms     4 ms  download.lime-technology.com [173.255.128.88] 

Trace complete.

[garycase]

... using things like Dropbox ...

 

As others have already noted, this is ludicrous !!    You're PAYING them to host your downloads ... and they want you to use another service since they're having issues !!!

 

You hit the nail on the head r.e. what they're really saying:

Basically they are saying, "go find another download server".

 

You should take their (implicit) advice  :)

 

 

[EGOvoruhk]

the main issue appears to be with the timeouts on the server end or with the client's ISP while trying to download this file.

 

Considering that if you can't download it on your ISP, but then VPN in elsewhere, the download completes no problem, that would point to it being a server-side issue, as client-end ISP timeouts would affect the VPN tunnel and kill the download just the same

 

I'd switch providers ASAP, not even due to the download issue, just because of their responses alone. Suggesting Dropbox for major hosting?! Saying that a 35MB if "pretty large?!" Clearly they're amateurs. Did they at least offer to cover the price of a Dropbox Business account while they fix the issue, since it'll be alleviating an issue on THEIR end?

 

This "working for some" is misleading the investigation.

 

Exactly. The focus for them needs to be on the fact that it's not working for a large group of COMPLETELY UNRELATED SYSTEMS, both in ISP and locale. That's a HUGE red flag. The only common denominator in the people that can't download the file is the server they're downloading from? Hmmm, must be on the client end, all of them...

[nars]

If you don't mind telling... how much traffic/month unraid downloads do?

 

The download subdomain eats up around 180 GB/month.  The main site uses around 250 GB/month, which seems inconceivable to me, but appears most of it is consumed by the forum.  When both the downloads and rest of the site were on the same host we were constantly hitting up against the bandwidth limit, which is why they are hosted on two separate servers.  I'd set up my own server here, but our building only has crappy DSL, though "sometime in the spring" we're supposedly getting fibre in here.

 

180GB/month... average 72KB/s... sure there are burst hours, but still... hitting bandwidth limit with forum!? lol these guys are just fun... best thing you could do would be IMO get a decent vps... not sure if dedicated server would justify for such amounts... though it would be the best service quality... no shared resources with others... and full control on it... downloads isolation as zoggy pointed could be surely reached with virtualization on it, but... well if forum, blog, etc always updated, etc... and currently on shared hosting with other users doesn't sound any better...

 

Two other solutions would be... something like Sourceforge as zoggy also pointed, or... if you want something more professional... without redirecting to external site I mean... a cdn would also be an option... something like maxcdn.com, traffic is relatively cheap (though not as cheap as dedicated for very big amounts), it works fine from experience I had with it sometime ago, and it's very easy to setup, basically you setup a non-public known subdomain on your hosting with the downloads and then set downloads subdomain dns with a cname to point them... then they will work like a proxy cache to your non-public subdomain, there is also another way to setup it but I don't remember exactly now.

 

There is also Torrent solution, the cheapest, and very reliable re files integrity... think someone even talked on other thread about a custom tinny downloader for it...

[NAS]

I am a big advocate for torrent delivery however it should not be the primary means of delivery. That should stay in house with a  traditional download link. Anything other is a hurdle (however small) to new users to the detriment of sales.

 

However we should take steps to make torrenting a secondary option. I suspect a good portion of the active userbase would contribute to this but first we need a native client. I believe ctorrent is the best and simplest options.

 

Also we should look to being able to handle mirrors in a more formal fashion. 180GB a month is trivial amount of traffic for us to deal with if we can handle ad hoc mirrors.

[mr-hexen]

I believe ctorrent is the best and simplest options.

 

Why have you recommended this? Many torrent clients are already available as plugins for unRAID. Not to mention they are supported better as well. Ctorrent was last updated in 2008!

[NAS]

Because it is very very simple with hardly any deps. Remember we need the absolute bare minimum to download unRAID only.

 

General torrent support should and is done via addons with the excellent rT and even the abomination uT.

 

What we are talking about is a client that will be in every default unRAID install. But let us not dwell on the client that is of almost no importance compared with agreeing if this should be done at all.

[Frank1940]

My input on this issue is that I WILL NEVER download any software via a torrent!!!!  Any risk of getting a 'modified' package is totally unacceptable to me. 

 

The ONLY time that I ever downloaded anything via a torrent was an iso of a  BluRay Demo disk.  I figured that loading a piece of malware into a BluRAY iso that could actually do something was virtually impossible.

[garycase]

I'm also old-school on torrents => I do NOT download with them.

 

This download issue is relatively recent ... until a month or two ago you could reliably download the UnRAID release with no issue.  Not sure what happened ... but if it can be broken, it can be fixed !!

 

Tom => did you recently (last few months) change to a new download host?

 

[JonathanM]

My input on this issue is that I WILL NEVER download any software via a torrent!!!!  Any risk of getting a 'modified' package is totally unacceptable to me.

A private tracker combined with DHT being disabled on the torrent would virtually eliminate normal attack vectors. It could be set up in such a way that only a running unraid server could be authenticated as a peer, and the authentication could be linked to the license. That way, if you choose to enable the built in torrent updater, you would only be able to connect to the limetech tracker, and only download and share limetech approved files.

[limetech]

My input on this issue is that I WILL NEVER download any software via a torrent!!!!  Any risk of getting a 'modified' package is totally unacceptable to me. 

 

I agree with this sentiment.

 

Version 5.0.4 is about ready.  Added curl, updated samba, couple other things.  I'm going to add an FTP link alongside each of the downloads.  midPhase tells me this should be more reliable.

 

[garycase]

I'm going to add an FTP link alongside each of the downloads.  midPhase tells me this should be more reliable.

 

Hopefully that will prove correct.  If so, that resolves it as far as I'm concerned.

 

[dgaschk]

A torrent is fine as long as the file hash matches the original. Serving the hash value on https is preferable. Anyone using torrent should be aware of the security implications and correct procedure. The torrent link should have an obvious warning regarding checking the hash value.

[jbartlett]

It could be set up in such a way that only a running unraid server could be authenticated as a peer, and the authentication could be linked to the license. That way, if you choose to enable the built in torrent updater, you would only be able to connect to the limetech tracker, and only download and share limetech approved files.

 

If I'm reading this right, this would mean that the UNRAID server would be openly accessible by the Internet, something that his highly frowned upon by every mention I've seen of doing that.

[limetech]

It could be set up in such a way that only a running unraid server could be authenticated as a peer, and the authentication could be linked to the license. That way, if you choose to enable the built in torrent updater, you would only be able to connect to the limetech tracker, and only download and share limetech approved files.

 

If I'm reading this right, this would mean that the UNRAID server would be openly accessible by the Internet, something that his highly frowned upon by every mention I've seen of doing that.

 

No, the files would be hosted "in the cloud".  But I can't require our customers to use a torrent client, who else does that?  We will get this download issue solved.  I'm going to close the thread since I know what the problem is now.  Thank you to all who posted here!

[NAS]

My input on this issue is that I WILL NEVER download any software via a torrent!!!!  Any risk of getting a 'modified' package is totally unacceptable to me. 

 

I agree with this sentiment.

 

Version 5.0.4 is about ready.  Added curl, updated samba, couple other things.  I'm going to add an FTP link alongside each of the downloads.  midPhase tells me this should be more reliable.

 

This is a common misconception and technically incorrect. It is simply impossible for this to happen unless the torrent file is changed which is exactly the same risk as if someone changed the http download.

 

I agree the primary download should be HTTP but consider this, unRAID is the only distribution i know of that does not also offer torrent downloads as an alternative and this is at least the 3rd time we have had to fix this with no alternative to HTTP in place.

 

Edit: I just noticed this thread is locked. No more posts from me.