How much does web scraping cost - the ultimate guide

In this blog, we will discuss how much web scraping really costs, the different options, their detailed costs and what are the best options for you..

By Victor @November, 10 2020

how much does web scraping cost - the ultimate guide

HOW MUCH DOES WEB SCRAPING COST - THE ULTIMATE GUIDE 

In recent years, web scraping has grown to become an essential part of many large companies due to the edge that comes with making data-based decisions. While a lot of businesses employ third-party applications / agencies for their web scraping, a growing number opt to also have theirs built. A common question we get asked are how much would it cost me to web scrape? The answer is always it depends on your project.

In this article we will go through the detail of how much web scraping really cost looking for all the different options you have and the different use cases

The Options available to web scraping are:

  1. Build a web scraper internally

  2. Hire third party web scraping contractors

  3. Outsource to a web scraping service provider

  4. Use third party web scraping self-service tools

 

  1. Build a web scraper internally

Without any technical skills this will almost be an impossible challenge as the learning curve is too high so we can eliminate this option if you have no dev skills within your company. According if you are a developer or have a dev team it will seem a relatively straightforward process at first, and it can be - if everything goes right. Unfortunately, it rarely never does. As you spend time making incremental improvements and fixing bugs, you end up realizing you've spent a lot more time and money than you expected. We documented some of the challenges with this option here

How to prevent getting blocked when web scraping

Step by Step Guide to Web Scraping JavaScript Content using Puppeteer and Node.js
 

Types of web scraping project complexities

As obviously there is no “one fit” web scraping project, we have broken down web scraping projects into 3 broad categories with the below characteristics

 
 

 Simple 

 Standard

 Complex 

 Web page structure

 Static web   pages

 Dynamic website

 Content is spread   across multiple   pages

 Dynamic website

 Content is spread across multiple pages

 Web page URLs can not be known  ahead of time

 Asynchronous loading of content using  JavaScript or other techniques

 ASP sites, files on sit es, images, PDFs

 Update frequency of   the structure of   website

 Annual

2-3 times a year

Weekly / monthly

 Update frequency of   the content of   website

 1-3 months

 Weekly

 Daily to Real time

 Number of   webpages

 1-100

 1-50,000

 50,000+

 Number of websites

 1-10

10-100

 100 +

 Number of pages  per  month

100- 10,000

10,000-1M

 1M +

Anti scraping defenses

none

IP address blocking, user agent blocking 

CAPTCHA, password, IP address blocking, user agent blocking, region blocking,

others

Delivery

CSV file

Structured tables, APIs 

Structured tables, APIs, custom integration, SQL updates, XML, insert into business

intelligence, statistical models, and machine learning applications

Quality control

What you see is what you get

Content verified

Monitoring for website layout changes

Timely completion of run

Content verified

Extraction is corrected based on website changes

Real-time monitoring of extraction success metrics

Quality reporting and tracking

Data Storage

> 1gb Storage

1-20gb

20gb +

 

Ref: Opimas Research

Factors to consider

The current cost of a building a web scraper internally is determined by a host of factors, including:

  • Server cost, data storage & Proxies cost 
  • Developer time for building, maintenance, and normalization of data - The developer cost for building the scraper depends on whether you are building from scratch or not. 

 

Developers time Implementation

 

SIMPLE

STANDARD

COMPLEX

EXPERT

<2hrs / website

2-8hrs / website

15-40hrs / per website

MID LEVEL

1-4hrs / website

4-20hrs / website

20-60hrs / per website

BEGINNER

4-8hrs / website

10-30hrs / website

-

 

Developers time to maintain

SIMPLE

STANDARD

COMPLEX

5-10hrs / month

10-20hrs / month

20-40hrs / month

 

Server, Storage & Proxies

 

SIMPLE

STANDARD

COMPLEX

Server, Storage & Proxies

<$ 200

$ 200- $ 700

$ 1000 +

 

Now depending on your region and internal cost of you developer you can work out costings eg If you have a developer on $ 50,000 / Year and you were scraping 5+ websites, it could actually take up 25% of his time which will work out to ~ $ 1k / month  

So let's summarize a typical scenario:

Developer time for web scraper implementation: $ 200 / web scraper

Server cost to run scrapers and store data: $ 500 / month

Developer time to maintain web scrapers: $ 500 / month

 

  1. Hire third party web scraping contractors

This is another popular option in recent years with the rise of the gig-economy. With sites like Fiverr, Upwork, Guru , Stack exchange & Toptal providing you with access to expert talent globally.

Source: Upwork.com . Filters for Web Scraping jobs hourly rate

Source: Codementor.io

Low Cost location e.g. India, Pakistan rates for experts is about $40/hours. mid level $20/hour

Higher Cost location e.g. U.S., UK experts is about $80/hours. mid level $50/hour

The benefit of using freelance developers for web scraping will be that they will usually be experts who do this frequently so will spend less time than generic developers who work internally for your company

To summarize the costs of using a freelancer, we use simple, standard & Complex project case scenarios with the average developer times. Lets average out the freelancers rate at $40/hour

 

 

Simple

Standard

Complex

One time Setup

$80

$80-$320

$600-$1600

Maintenance

$200-$400

$400-$800

$800- $1600

 

  1. Outsource to a web scraping service provider

Outsourcing to an expert agency is a good option when you are looking to transfer liability. With Service agencies you can agree to a service level agreement (SLA) and ensure you get your data when you want it, how you want it and not have to worry about all the details and difficulties. All you do is pay a service fee monthly but this is usually a more expensive option

When getting quotes from service providers you will typically get a breakdown in this format

  • Initial Setup fee per website
  • Managed service fee paid monthly
  • Fee based on number of records e.g. <10k records, 10-100k records, 1M+ records
  • Fees for any extras like integration, reports etc

For a standard project e.g. scraping an e-Commerce website for about 5-20k products

Top end

One off Setup fee

$400

Monthly maintenance

$450

 

Lower end

 

One off Setup fee

$99

Monthly maintenance

$99

 

  1. Use third party web scraping self-service tools

Finding the right tool is a dream for most users who have gone through the trouble of building web scrapers writing code and maintaining infrastructure. The right tool could save you time building systems or maintaining infrastructure like servers and proxies

There are many software companies out there that provide software that allow you to scrape data for a subscription fee. This is the most cost effective and best for most of the use cases/websites where complex requirements are not needed

Some of these tools will have a Free trial or Free plan with limited features or cap on number of pages that can be scraped. Get scraping now with our webautomation.io free Web Scraping tool - up to 25,000 pages scraped.

Most tools would typically have 3 option pricing tier with a number of requests limits per package. Package 1 will roughly range with requests limits of 1,000-25,000 pages/requests , Package 2 - 25,000-250,000 pages/requests and Package 3 - 250,000- 1,0000 pages/requests

See breakdown below

 

Package 1

Package 2

Package 3

$19-$149

$99- $499

$249- $999

 

One-time Web Scraping jobs

A lot of these quotes are based on ongoing web scraping jobs. But what about one-time jobs?

For simple one-time jobs you can get quotes from $20 from freelancers for small amount of data. Webautoamtion.io offer a free plan with up to 25,000 pages scraped, this is usually enough for one-time simple jobs

 

GIVE WEBAUTOMATION A TRY

Thanks to webautomation.io building a web scraper to help your business is not so expensive anymore. For Free you can submit a Concierge request to get an expert build a web scraper for you. You are also able to use any of the Pre-built Extractors for sites like Amazon and Ebay to start getting product data in minutes .

Webautomation.io offers a free plan for your one-time scraping needs or paid plans from $19 for ongoing projects, additionally, we can setup and run the entire scraping job for you - just contact us to request a quote.

*Please note that costs mentioned in this article are estimates based on current market rates and our own internal experience 

 

                                                                               

Save Costs, Time and Get to market faster

Build your first online custom web data extractor.

Leave a comment:

You should login to leave comments.