User-Agent header

A user agent is the client application used with a particular network protocol; the phrase is most commonly used in reference to those which access the World Wide Web, but other systems such as SIP uses the term user agent to refer to the user's phone. Web user agents range from web browsers to search engine crawlers ("spiders"), as well as mobile phones, screen readers and braille browsers used by people with disabilities. When Internet users visit a web site, a text string is generally sent to identify the user agent to the server. This forms part of the HTTP request, prefixed with User-agent: or User-Agent: and typically includes information such as the application name, version, host operating system, and language. Bots, such as web crawlers, often also include a URL and/or e-mail address so that the webmaster can contact the operator of the bot.

The user-agent string is one of the criteria by which crawlers can be excluded from certain pages or parts of a website using the "Robots Exclusion Standard" (robots.txt). This allows webmasters who feel that certain parts of their website should not be included in the data gathered by a particular crawler, or that a particular crawler is using up too much bandwidth, to request that crawler not to visit those pages.

User agent spoofing

At various points in its history, use of the Web has been dominated by one browser to the extent that many websites are designed to work with that particular browser, rather than according to standards from bodies such as the W3C and IETF. Such sites often include "browser sniffing" code, which alters the information sent out depending on the User-Agent string received. This can mean that less popular browsers are not sent complex content, even though they might be able to deal with it correctly, or in extreme cases refused all content. Thus various browsers "cloak" or "spoof" this string, in order to identify themselves as something else to such detection code; often, the browser's real identity is then included later in the string.

The earliest example of this is Internet Explorer's use of a User-Agent string beginning "Mozilla/<version> (compatible; MSIE <version>...", in order to receive content intended for Netscape Navigator, its main rival at the time of its development. It should be stressed that this is not a reference to the open-source Mozilla browser, which was developed much later, but to the original codename for Navigator, which was also the name of the Netscape company mascot. This format of User-Agent string has since been copied by other user agents, partly because Explorer, in turn, came to dominate.

When Internet Explorer became the dominant web browser, rivals such as Firefox, Safari, and Opera implemented systems whereby the user could select a false User-Agent string to send, such as that of a recent version of Explorer. Some – e.g. Firefox and Safari – duplicate the User-Agent string they are trying to spoof exactly; others – e.g. Opera – duplicate the User-Agent string but add the genuine browser name to the end. This latter approach, of course, leads to a string containing three names and versions: first, the user agent claims to be "Mozilla" (i.e. Netscape Navigator); then, "MSIE" (Internet Explorer); and finally, the actual browser, such as "Opera". Opera also offers a full masking as Internet Explorer or Firefox, which hides "Opera" completely.

Beside browsers, other programs utilizing HTTP protocol, like most download managers and offline browsers, also had the ability to change the user agent string sent to servers to user's liking. This is presumably done in an effort to maintain compatibility with certain servers (some servers refused to serve those programs right away because they are mostly used carelessly, thus burdening the server).

This vicious circle is expected to continue in the area of web browsers. Some standards-based web developers have started the "Viewable With Any Browser" campaign which encourages developers to design webpages according to official standards, not for any particular browser(s).

As of 2005, many websites are more standards-compliant than at other times in the history of the web. However, out-dated JavaScript, which effectively locks out browsers other than Explorer or Navigator, is still in use on many websites. This is often blamed on use of cargo cult programming, in the form of copying and pasting older code without actually understanding what effect this will have on the website.

One result of user agent spoofing is that the usage share of Internet Explorer, the user agent browsers typically spoof, is probably overestimated, and the usage share of other browsers may be underestimated. User agent spoofing can also provide a security issue by spoofing search engine bots and bypassing key parts in a website.

User agent sniffing

The term user agent sniffing refers to websites that show different content when viewed with a certain user agent. On the Internet, this will result in a different site being shown when browsing the page with a specific browser (e.g. Microsoft Internet Explorer). An infamous example of this is Microsoft's Outlook 2003 Web Access. When viewed with IE, much more functionality is displayed compared to the same page in any other browser. User agent sniffing is mostly considered poor practice for Web 2.0 web sites, since it encourages browser specific design. Many webmasters are recommended to create an HTML markup that is as standardised as possible, to allow correct rendering in as many browsers as possible.

Websites specifically targeted towards mobile phones, like NTT DoCoMo's I-Mode or Vodafone's Vodafone Live! portals, often rely heavily on user agent sniffing, since browsers on mobile phones often differ a lot from each other. Many developments in mobile browsing have been made in the last few years, while many older phones that do not possess these new technologies are still heavily used. Therefore, mobile webportals will often generate completely different markup code depending on the mobile phone used to browse them. These differences can be small (e.g. resizing of certain images to fit smaller screens), or quite extensive (e.g. rendering of the page in WML instead of XHTML).

There are a number of ways to perform user agent sniffing within web applications, including using public domain scripts and even commercial products such as BrowserHawk. See the External Links section for details.

Encryption strength "U" / "I" / "N"

Netscape, Mozilla, Opera and some others use one of these three letters to specify the browser's encryption strength. Since the US government formerly would not allow encryption higher than 40-bit to be exported from the USA, different versions were released with different encryption strengths. "U" stands for "USA" (for the version with 128-bit encryption), "I" stands for "International" (the browser has 40-bit encryption and can be used anywhere in the world), "N" stands for "None" (no encryption). Originally the "U" version was allowed for download only from the USA, but since then the US government has loosened its policy and exporting high encryption is now permitted to most countries (see Export of cryptography for more information). Now Netscape and Mozilla distribute their browsers only in a "U" version, supporting up to 256-bit encryption, since an international version is no longer required.

See also

• Robots Exclusion Standard
• Web crawler
• Wurfl
• UAProf

External links

• Lists of user-agent strings in common use
  • User-agents.org, database of known user agent strings
  • Robots/crawlers DB From "ASAP Consulting".
  • Browsers capabilities registry — A listing of many user agents along with supported capabilities

• Tools for examining user-agent string
  • User agent string analysis
  • Display detailed analysis UA fragments with XML-RPC access

• A database of web browsers and the JavaScript properties which they support