How much does web scraping cost - the ultimate guide

HOW MUCH DOES WEB SCRAPING COST - THE ULTIMATE GUIDE 

In recent years, web scraping has grown to become an essential part of many large companies due to the edge that comes with making data-based decisions. While a lot of businesses employ third-party applications / agencies for their web scraping, a growing number opt to also have theirs built. A common question we get asked are how much would it cost me to web scrape? The answer is always it depends on your project.

In this article we will go through the detail of how much web scraping really cost looking for all the different options you have and the different use cases

The Options available to web scraping are:

1. Build a web scraper internally

2. Hire third party web scraping contractors

3. Outsource to a web scraping service provider

4. Use third party web scraping self-service tools

 

1. Build a web scraper internally

Without any technical skills this will almost be an impossible challenge as the learning curve is too high so we can eliminate this option if you have no dev skills within your company. If you are a developer or have a dev team it will seem a relatively straightforward process at first, and it can be - if everything goes right. Unfortunately, it rarely never does. As you spend time making incremental improvements and fixing bugs, you end up realizing you've spent a lot more time and money than you expected. We documented some of the challenges with this option here

How to prevent getting blocked when web scraping

Step by Step Guide to Web Scraping JavaScript Content using Puppeteer and Node.js
 

Types of web scraping project complexities

As obviously there is no “one fit” web scraping project, we have broken down web scraping projects into 3 broad categories with the below characteristics

	Simple	Standard	Complex
Web page structure	Static web   pages	Dynamic website  Content is spread   across multiple   pages	Dynamic website  Content is spread across multiple pages  Web page URLs can not be known  ahead of time  Asynchronous loading of content using  JavaScript or other techniques  ASP sites, files on sit es, images, PDFs
Update frequency of   the structure of   website	Annual	2-3 times a year	Weekly / monthly
Update frequency of   the content of   website	1-3 months	Weekly	Daily to Real time
Number of   webpages	1-100	1-50,000	50,000+
Number of websites	1-10	10-100	100 +
Number of pages  per  month	100- 10,000	10,000-1M	1M +
Anti scraping defenses	none	IP address blocking, user agent blocking	CAPTCHA, password, IP address blocking, user agent blocking, region blocking, others
Delivery	CSV file	Structured tables, APIs	Structured tables, APIs, custom integration, SQL updates, XML, insert into business intelligence, statistical models, and machine learning applications
Quality control	What you see is what you get	Content verified Monitoring for website layout changes	Timely completion of run Content verified Extraction is corrected based on website changes Real-time monitoring of extraction success metrics Quality reporting and tracking
Data Storage	> 1gb Storage	1-20gb	20gb +

 

Ref: Opimas Research

Factors to consider

The current cost of a building a web scraper internally is determined by a host of factors, including:

• Server cost, data storage & Proxies cost 
• Developer time for building, maintenance, and normalization of data - The developer cost for building the scraper depends on whether you are building from scratch or not. 

 

Developers time Implementation

	SIMPLE	STANDARD	COMPLEX
EXPERT	<2hrs / website	2-8hrs / website	15-40hrs / per website
MID LEVEL	1-4hrs / website	4-20hrs / website	20-60hrs / per website
BEGINNER	4-8hrs / website	10-30hrs / website	-

 

Developers time to maintain

SIMPLE	STANDARD	COMPLEX
5-10hrs / month	10-20hrs / month	20-40hrs / month

 

Server, Storage & Proxies

	SIMPLE	STANDARD	COMPLEX
Server, Storage & Proxies	<$ 200	$ 200- $ 700	$ 1000 +

 

Now depending on your region and internal cost of you developer you can work out costings eg If you have a developer on $ 60,000 / Year and you were scraping 5+ websites, it could actually take up 25% of his time which will work out to ~ $ 1.25k / month  

So let's summarize a typical scenario:

Developer time for web scraper implementation: $ 400 / web scraper

Server cost to run scrapers and store data: $ 300 / month

Proxy costs : $200/month

Developer time to maintain web scrapers: $ 1250 / month

 

2. Hire third party web scraping contractors

This is another popular option in recent years with the rise of the gig-economy. With sites like Fiverr, Upwork, Guru , Stack exchange & Toptal providing you with access to expert talent globally.

Source: Upwork.com . Filters for Web Scraping jobs hourly rate

Source: Codementor.io

Low Cost location e.g. India, Pakistan rates for experts is about $40/hours. mid level $20/hour

Higher Cost location e.g. U.S., UK experts is about $80/hours. mid level $50/hour

The benefit of using freelance developers for web scraping will be that they will usually be experts who do this frequently so will spend less time than generic developers who work internally for your company

To summarize the costs of using a freelancer, we use simple, standard & Complex project case scenarios with the average developer times. Lets average out the freelancers rate at $40/hour

 

	Simple	Standard	Complex
One time Setup	$80	$80-$320	$600-$1600
Maintenance	$200-$400	$400-$800	$800- $1600

 

3. Outsource to a web scraping service provider

Outsourcing to an expert agency is a good option when you are looking to transfer liability. With Service agencies you can agree to a service level agreement (SLA) and ensure you get your data when you want it, how you want it and not have to worry about all the details and difficulties. All you do is pay a service fee monthly but this is usually a more expensive option

When getting quotes from service providers you will typically get a breakdown in this format

• Initial Setup fee per website
• Managed service fee paid monthly
• Fee based on number of records e.g. <10k records, 10-100k records, 1M+ records
• Fees for any extras like integration, reports etc

For a standard project e.g. scraping an e-Commerce website for about 5-20k products

Top end

One off Setup fee	$400
Monthly maintenance	$450

 

Lower end

 

One off Setup fee	$99
Monthly maintenance	$99

 

4. Use third party web scraping self-service tools

Finding the right tool is a dream for most users who have gone through the trouble of building web scrapers writing code and maintaining infrastructure. The right tool could save you time building systems or maintaining infrastructure like servers and proxies

There are many software companies out there that provide software that allow you to scrape data for a subscription fee. This is the most cost effective and best for most of the use cases/websites where complex requirements are not needed

Some of these tools will have a Free trial or Free plan with limited features or cap on number of pages that can be scraped. Get scraping now with our webautomation.io free Web Scraping tool - up to 25,000 pages scraped.

Most tools would typically have 3 option pricing tier with a number of requests limits per package. Package 1 will roughly range with requests limits of 1,000-25,000 pages/requests , Package 2 - 25,000-250,000 pages/requests and Package 3 - 250,000- 1,0000 pages/requests

See breakdown below

 

Package 1	Package 2	Package 3
$49-$149	$99- $499	$249- $999

 

One-time Web Scraping jobs

A lot of these quotes are based on ongoing web scraping jobs. But what about one-time jobs?

For simple one-time jobs you can get quotes from $20 from freelancers for small amount of data. Webautoamtion.io offer a free plan with up to 25,000 pages scraped, this is usually enough for one-time simple jobs

 

GIVE WEBAUTOMATION A TRY

Thanks to webautomation.io building a web scraper to help your business is not so expensive anymore. For Free you can submit a Concierge request to get an expert build a web scraper for you. You are also able to use any of the Pre-built Extractors for sites like Amazon and Ebay to start getting product data in minutes .

Webautomation.io offers a free plan for your one-time scraping needs or paid plans from $49 for ongoing projects, additionally, we can setup and run the entire scraping job for you - just contact us to request a quote.

*Please note that costs mentioned in this article are estimates based on current market rates and our own internal experience