This article discusses issues that affect absolutely all people who are on the Internet. It does not matter whether you visit the worldwide network from a computer or smartphone, it does not depend on the operating system: Windows or any option of Linux, mobile Android or iOS.
I have long been unpleasantly surprised by the intrusive Google AdSense ads based on my old search terms. It would seem that after the expressed interest, a lot of time has already passed, cookies and cache have been cleared in the browser more than once. But advertising still appears. How do they manage to get information about me anyway? As it turned out, there are many ways to do this.
Web tracking, or simply put, identifying and tracking users, is not a very difficult activity. For this, a special unique identifier for each browser is used, which is set on the client side when you’re visiting the site.
This tool was conceived for quite decent and useful purposes. For example, this method helps to distinguish live people from bots that have visited the site, or determine the preferences of a person, save and apply them on subsequent visits. But, everything has two sides. And web tracking has become a handy tool in the advertising industry, and it started back in the mid-90s.
A lot has changed since that time, the development of technology makes it possible to track users not only by cookies, but also using other tools. The easiest way to track a user is to use some kind of identifier like the same "cookies". You can also take advantage of the browser information they pass when sending requests. There, the HTTP header contains: the address, the operating system installed on the device, the current time of the user, etc. In addition, users can be distinguished from each other by their habits, for example, by frequently visited sections of the site or by the peculiarities of cursor movement.
So, in order to understand that a visitor has already been to a certain resource, you need to be on his or her side, it means, on a computer, phone or other device, set some kind of long-lived label, and also be able to read it in the event of a repeated visit to the site. In modern browsers, there are many options for doing this, and it is quite transparent to users.
First, these are the "cookies" already mentioned above. Secondly, there are certain programs that are similar to cookies in functionality. This can be LSO (Local Shared Objects) used by Adobe Flash, or its Silverlight counterpart. There are similar capabilities in HTML5, including those implemented through the localStorage, File and IndexedDB API tools.
There are other places for storing tokens - these can be cached data on the local device or various types of also cached metadata (Last-Modified, ETag). You can also use Origin Bound certification fingerprints that browsers generate on SSL connections, based on information from SDCH dictionaries, their content, or metadata, to identify users. In general, there are many options.
Cookies is the first tool that was created specifically in order to store small amounts of data on the client side. That is why it is the first thing you remember about when you think about identifiers.
When the user first contacts the server, he or she generates a unique label, which is more correctly called an identifier. At the first visit, the server generates it and transfers it to the client along with other information to be saved in cookies. On each repeated call, the client will immediately send this identifier to the server.
And while most browsers today are equipped with user-friendly options for cleaning and configuring cookies, and you can find a huge number of special utilities for cleaning or blocking cookies on the Internet, this tracking method has not lost its popularity. The reason is that most users rarely remember and clean cookies. Well, think for yourself, when was the last time you personally did this? One of the most common reasons is the fear of accidentally deleting the desired cookie, because the cookie often stores information for quick authorization.
Some modern browsers allow restricting the installation of third-party cookies, but this add-on does not solve the problem. Browsers often confuse third-party and "main" cookies, for example, when receiving information via HTTP redirects and other similar methods of accessing a page.
Using cookies, unlike many other methods, is one of the most transparent methods from the user's point of view. But in order to "recognize" a visitor, there is no need to store a unique tag in a separate file. The identifier can be collected from several cookies, stored in other data types, for example, in Expiration Time, etc. Therefore, it can be extremely difficult to figure out whether a particular cookie is used for tracking or not.
This mechanism is used by Adobe Flash to store information on the client machine. LSO is considered an analogue of "cookies" in HTTP, but allows you to save large amounts of data with fewer restrictions, so it is extremely difficult to analyze and check for foreign labels such objects.
Previously, before the advent of Adobe Flash 10.3, you had to configure LSO behavior separately from the browser; you had to use the Flash settings manager at macromedia.com. Later versions allow you to configure Flash cookies in your browser control panel. In addition, most modern browsers are quite deeply integrated with the flash player, so when you delete cookies and other temporary files, LSOs are also deleted. However, the integration is not so deep, therefore, setting the policy for working with third-party cookies in the browser does not always affect Local Shared Objects. Visit the Adobe developer site for information on how LSO is manually disabled.
Silverlight platform is in many ways similar in functionality and features of working with Adobe Flash. It also has its own analogue of flash cookies - Isolated Storage. But the privacy settings for Silverlight are very different. Firstly, this storage does not integrate in principle with browsers, therefore any clearing of the cache and other data in the browser will not affect Isolated Storage in any way. Second, the tool uses the same storage for all browser tabs and windows. The only exceptions are the tabs that you open in "Incognito" mode, as well as for all profiles that are installed on the user's device.
Technically, Isolated Storage is just as easy to store ID labels as LSOs. But it is not yet possible to get to it using the user's browser settings, so it is not often used as a user identification tool yet.
HTML5 is a whole set of tools that specialize in storing structured information of one type or another on the client's device. These include localStorage, File API, and IndexedDB. Each of these mechanisms has its own characteristics, but they all provide the ability to permanently store structured binary data without limiting their size, while binding them to a particular resource. It is important that unlike regular cookies or flash cookies, HTML5 has no limits on the size of this data. HTML5 repositories are located in the same place as other information from sites. But in the browser interface, managing them is difficult, since it can be difficult for an ordinary user to guess where the necessary sections are located.
For example, to delete information stored in localStorage, select "offline website data" or "site preferences" in Firefox browser and then set the time interval to "everything". For IE, there is a special feature that works in HTML5. Here data is stored only during the life of the tabs that were opened at the time of their saving.
Also, HTML5 doesn't really strive to comply with cookie-friendly rules and regulations. For example, in localStorage, the server will be able to write information and read it through cross-domain frames, and this method is available even in the mode of completely disabling third-party cookies.
It goes without saying that the browser will from time to time use requests in the If-Modified-Since header to check for any changes on the site. But if every time the server returns an answer that there are no changes through the special code 304 (Not modified), the file with the tag can be stored forever.
Of course, modern browsers allow you to completely clear the cache at any time. But in practice, we all do this quite rarely, if we perform this action at all.
Sites are regularly updated, so for caching to work properly, you need a tool that allows the server to inform the browser about changes on the page. In the HTTP / 1.1 standard, there are two options for exchanging information about updates. In the first case, the date of changes in the file is analyzed, in the second, a special identifier called ETag is used.
When working with ETag, the server, in response to a client request, generates a document with a version tag in the header. On repeat visits to the site, the client includes the value of this header in the If-None-Match variable in the request, and thus reports the local version. If it is up to date, the server responds with a 304 code and the existing cached data is used when loading the page. If not, a new version of the document is sent to the client along with a new ETag. This way of working is somewhat similar to the use of HTTP cookies, when the server stores some information on local machines for the sole purpose of reading it on repeat visits.
The variant of working with Last-Modified assumes the ability to save the date of the last changes, for which a volume of 32 bits or more is allocated. When a page on the site is accessed, this information is sent to the server in a header called If-Modified-Since. An interesting point here is that most browsers do not check the correctness of the date format in any way. It is important to understand that ETag and Last-Modified objects are not deleted along with cookies or site data, only clearing the cache helps here.
The Application Cache tool indicates which fragments of a site page are saved on the user's disk and will be available, including offline. Storage and access rules are specified using special manifests. AppCache, like regular caching, stores unique data both inside the manifest itself and as part of resources that do not have storage time limits, unlike the usual cache.
We can say that AppCache is an intermediate solution between the traditional cache and the capabilities of HTML 5. In some browsers, it is possible to delete this information along with cookies, for others, you need to clear your browsing history and cache.
SDCH is a special data compression algorithm that was developed by Google. It uses a specially compiled dictionary provided by the server, as a result the client receives a higher compression than using deflate or Gzip.
The method is based on the fact that sites massively transmit large amounts of repetitive information to the client. These can be, for example, page feet and "headers", JS and CSS values, other elements that remain unchanged when moving from one section to another. To save resources, the server sends a special SDCH dictionary once when accessing the site, which contains duplicate values. Further, when accessing the page, the server sends links to the dictionary elements to the browser, and the client itself collects the page from them.
It is clear that you can add any information to such a dictionary, including unique identifiers. They are placed in ID dictionaries. If the user visits the site again, the server requests the corresponding information from the dictionary and receives it in the Avail-Dictionary header. You can specify a label in the content itself, and then use it, as when working with a regular cache.
Origin Bound Certificates (aka ChannelID) is a feature that significantly increases the security of working with HTTPS. It stores special self-signed persistent documents called certificates, which allow the client to be identified during a secure connection. In this case, for each domain, its own certificate is generated and saved, designed to improve the security of work. But OVS websites are sometimes used for tracking, moreover, they do not take any additional actions that would be noticeable to the user. One example is the presence of a cryptographic cache. It is often used for an SSL handshake, but it may well act as a unique user label.
TLS also has similar mechanisms - session tickets and session identifiers. These tools are used in case the session was abruptly interrupted during the HTTPS connection. To prevent the user from going through the full handshake procedure again, cached data mechanisms are used. They allow you to determine the authenticity of the client's request if the interval between requests is small.
Another interesting tool is the browser's DNS cache. It stores the small information needed to improve the speed of validation and resolution of site names on repeated calls while protecting the system from DNS rebinding attacks. For example, 16 IP addresses are available. In this case, already 8-9 names, including encrypted ones, will allow identification of any device. But such a solution is significantly limited by the internal volume of the DNS cache, therefore there is a risk of conflicts between name resolution and the DNS provider.
All the previously described options were based on the installation of some unique identifier on the user's device, which the server can receive upon repeated requests. But there is another option for tracking users, less obvious. It is based on query or measurements of client device parameters. Each of the characteristics separately is nothing more than a few bits of data, but if you get several at the same time, you can track any computer or phone. In addition to the fact that the study of device parameters is much more difficult to track down and prohibit, this method allows you to identify a user who goes to the Internet from different browsers or uses the "incognito" mode.
One of the simplest tracking methods is to create a unique tag by combining parameters that are available in the browser. Each of them separately is not very interesting for analysis. But their combination helps to "recognize" and identify the user's browser:
Tracking unique users and parameters related to the architecture of the local network and features of network protocols.
This method will allow the user to be tracked regardless of the browsers they are using. They cannot be hidden using incognito modes, other private browsing options, or various security applications.
The list of this information includes:
Modern technologies make it possible to study not only identifier tags and equipment features, but also the behavioral factors of the person him\herself. These can be region settings and behavior features. Thus, it turns out to identify the user, even if he changes browsers or uses the "incognito" mode.
The following data are used to identify behavioral factors:
Any options for changing fonts and scaling the site, measuring the zoom level, using special features also help to determine the user. We study browser features that the user configures: blocking cookies, pop-ups and ads, DNS prefetching, flash security settings, etc. Paradoxically, if you diligently customize a convenient and secure work option for yourself, you, thereby, give a lot of information for your identification.
And this is only part of such information, the most popular and obvious. If you look deeper, there may be many more.
As you can see, in fact, there are many options for tracking users. Some are the consequences of release bugs or omissions, and theoretically can be fixed. Others are impossible to "close", unless completely changing the way computers, networks, browsers and software work. Some methods can be combated, for example, timely cleaning the browser and the system. Others you, with all your desire, will not notice, and it is almost impossible to fight them. Therefore, the most important rule of visiting the Internet: even if you use the best solutions for anonymous and secure browsing, remember that it is still possible to track you.