Accept headers are sent by the browser to the server to tell the server what kind of stuff the browser can handle. Servers don’t need an accept header. The accept header is described as optional in HTTP documentation.
If you don’t send my software an accept header you are blocked, banned, and reported.
How can this be? The reason is simple. All major browsers; IE, Chrome, FF, etc., send accept headers. If my software does not receive an accept header then the request is not coming from a browser. Humans use browsers to surf the web. If you are not sending me an accept header you are not human, but a robot.
It’s that simple. I am amazed at how many requests are blocked because there is no accept header. I get nervous and check them from time to time, and every time the request comes from a server farm or China or Russia and I can automatically assume that the lack of accept header is because a robot is hitting my site.
I also reject requests because of a missing http_host header, and when the method is “POST”, a missing http_referer header, or a http_referer header that does not match the website.
So if you are writing any software that needs to hit my site, including rss readers or automatic gets of my spam lists, then you must provide a set of real http headers.