Content theft, data mining about competitors and price scanners
Bots put a strain on websites around the clock. They scrape content and post them unauthorized on other websites.
You don’t get any backlinks and no author credits for this. In fact you’ll never know that your content has been scraped unless you spend a lot of time searching for your content on Google and Bing.
The duplicate content that exists because of web scraping falls back on your website and lowers your Google page rank, resulting in less visitors and less revenue.
To put it in a nutshell: web scraping can limit your business’ ability to be successful and to grow.
Unscrupulous entrpreneurs use bots to gather valueble information about their competitors. E-Commerce websites are particularly susceptible to price scans, product matching, the scanning of product variants and availibilities. Online-Shops are an obvious target for bad bots not least because of flash sales and limited offers, e.g. sneakers.
When bots scrape price and product information they feed the aggregated data into an analysis system that enables your competitors to monitor your products and prices in real time. Often enough seconds decide whether you as vendor make the sale or the bot operator that steals your data.
Travel industry websites such as online booking portals, airlines or hotels are constantly being scraped by bots. Competitors and aggregators use bots to circumvent limits of existing APIs, to undercut prices or to find cross-selling and up-selling potentials and to steal content.
How to detect if you’re affected
You won’t notice anything unless you look closely. Clear signs for your being a victim of bad bots are for instance
- You see your content in many different places on the internet, often on websites of your competitors
- You notice that competitors adapt their prices rapidly after you have changed your prices
Example: Amazon and diapers.com
Diapers.com was a flourishing web shop with a large assorment of baby products. Amazon.com was a successful book dealer that wanted to enter other markets, among them baby products.
Amazon started to scan diapers.com’s products and prices and used that data to learn their business strategy. Based on the data Amazon mimicked diapers.com’s product portfolio and undercut their prices in real-time.
Within one year Amazon had launched Amazon Mom and bought a drastically devalued diapers.com. That strategy lead much faster to the goal than if Amazon had started their baby shop from scratch.
(Source: https://slate.com/technology/2013/10/amazon-book-how-jeff-bezos-went-thermonuclear-on-diapers-com.html)