The Search Engine Eraser

Search engine technology relies on a number of algorithms for indexing, web crawling and matching user queries with pre-categorized web pages. One of the key components is the relevancy or search results ranking algorithm. Its purpose is to select 20 to 200 organic and 3 to 32 paid relevant results for each potential user query or keyword search, and to sort these search results according to some relevancy criteria.

The metrics used in the relevancy algorithm are part of the secret sauce, and (together with the size of the index), that's what differentiates Google from Yahoo, Bing and other search engines.

Nevertheless, two fundamental metrics are always used:

• CTR, or click through rate. Poor CTR always penalizes the score associated with any keyword / web page combination. Poor CTR means that you don't show up at the top of the search result page, no matter how much you are willing to bid for the keyword in question.
• For paid search, the max bid - how much the advertiser is willing to pay for a click - is of course a fundamental metric to determine the position on the search result page. But it is no longer the only metric: CTR is playing an increasingly important role.

CTR is deemed to be a good indicator of the quality of the link, be it a natural or paid search.

In this article, we explore the flaws found in relevancy algorithms currently used by companies such as Google, Yahoo and Microsoft. In particular, we explain how relevancy algorithms have successfully been manipulated by smart web crawlers to change web page rankings on search result pages, and in some cases, to ban some test websites, preventing them from showing up both in natural and paid search results. These tools have been used by advertisers to eliminate competitors from organic and paid search. Sophisticated SEO and SEM companies have been known to use them.

Eliminating your competitors from paid search results

Since ad relevancy heavily relies on CTR, it is possible to boost your search rankings by self-clicking on your own ads and generating bogus impressions against your competitor ads, for targeted keywords. This scheme is known as impression fraud. Sophisticated versions of impression fraud rely on standard or open Botnets (collusion) to generate an increasingly large number of impressions, targeting one advertiser at a time for a specific keyword, with a specific schedule that does not involve abrupt impression spikes, to eventually remove the advertiser in question from paid results, and repeat the process for each advertiser until most advertisers (for the targeted keywords) are gone.

Eliminating your competitors from organic search results

The same technique can be used.

Banning your competitors from organic search results

It is possible to permanently erase competitors. Just like you can be banned from search engines for doing anything that violates their policy (such as participating in click farms, weblog referral advertising or use artificial mechanisms to boost your page ranking), you can get your competitors banned by signing them up with click farms or get them enrolled in other nefarious schemes, without their consent, and even without them being aware of it. We successfully got a few test websites (owned by us) banned on Google, despite the fact that the test web sites in question were victims of relevancy manipulations, not culprits. Similarly, we know publishers that were eliminated from ad networks as a result of having their web sites manipulated by competing publishers.

Why this is happening? In a number of cases, the search engine is not able to discriminate between "victim" and "culprit". We need new algorithms that are good at this, and that can truly identify the real culprits. The problem is just like if you had your Internet account terminated by your ISP, because you have a virus on your computer, and your ISP believes that you purposely installed the virus yourself with a nefarious intent, while actually you are not even aware that there is a virus on your machine.

Banning your competitors from paid search results

Advertisers can be banned if they start advertising illegal stuff or if the landing page is infested with malware. We have not yet explored how can an advertiser be banned due to competitors exploiting flaws in relevancy algorithm. This is a subject for future research. In the cases that we know, the advertiser was banned because their servers have been hijacked and malware installed by the pirates on the landing page, resulting in a loss of the AdWord or AdSense account.