Home page / Cases / Ecommerce scraping

Ecommerce scraping

The dropshipping company needed a way to automatically monitor prices and stock availability for over 100,000 products from over 1,500 stores. We created a system using custom scripts and a web interface that could check 60 million pages daily. This led to a reduction in manual work and errors, and improvements in customer experience and a $50-70k increase in monthly profits.

Retail

1000h+

manual work reduced

60 mln

pages processed daily

About the client

The dropshipping company specializing in home furniture deals with more than 130k orders monthly. The retailer (supplier) owns more than 1.5k local stores with varying prices on the same products.

Tech stack

Python

Celery

Pandas

PostgreSQL

Elasticsearch

GCP

Challenges & solutions

Challenge

Keep infrastructure charges below $1000 per month.

Automatize daily price relevance and stock availability monitoring for more than 100k products through the entire network(approximately 18 mln requests per day).

Reduce the amount of manual work and operational errors during updates and new products listings.

Add cross-checks of the same products listed on Amazon, Walmart, Lowe’s, etc.

Create a system for monitoring all products available looking for ones with the biggest price variations.

Create a notification system that automatically notifies the admin about order problems, changes, delays and controls orders statuses and shipments.

Automatize cashback website tracking, adding an option to identify orders with no/incorrect cashback to avoid missed orders.

Create daily reports with detailed statistics.

Automatize updates and new listings at the selling platform.

Solution

Developed a full monitoring process covered by custom scripts through distributed server architecture.

Created a solution for controlling the system via web interface.

Created an algorithm to monitor more than 60 mln pages daily.

6 employees were discharged saved 1000 hours of manual work monthly.

Improved customer experience. Reduced the number of canceled orders by 69%.

Tailor-made algorithm for the product's price arbitrage gave an additional positive impact by $50-70k per month.

They always find cutting-edge solutions, and they help bring our ideas to life.

Jonathan Lien

CEO Advanced Clear Path, Inc., Ecommerce Company

Steps of providing
data scraping services

Step 1 of 5

Free consultation

It's a good time to get info about each other, share values and discuss your project in detail. We will advise you on a solution and try to help to understand if we are a perfect match for you.

Step 2 of 5

Discovering and feasibility analysis

One of our core values is flexibility, hence we work with either one page high level requirements or with a full pack of tech docs. At this stage, we need to ensure that we understand the full scope of the project. Receive from you or perform a set of interviews and prepare the following documents: list of features with detailed description and acceptance criteria; list of fields that need to be scraped, solution architecture. Ultimately we make a project plan which we strictly follow. We are a result-oriented company, and that is one of our core values as well.

Step 3 of 5

Solution development

At this stage, we develop the scraping engine core logic. We run multiple tests to ensure that the solution is working properly. We map the fields and run the scraping. While scraping, we keep the full raw data so the final model can be enlarged easily. Ultimately we store data in any database and run quality assurance tests.

Step 4 of 5

Data delivery

After quality assurance tests are completed, we deliver data and solutions to the client. Though we have over 15 years of expertise in data engineering, we expect client’s participation in the project. While developing and crawling data, we provide midterm results so you can always see where we are and provide us with feedback. By the way, a high-level of communication is also our core value.

Step 5 of 5

Support and continuous improvement

We understand how crucial the solutions that we code for our clients are! Our goal is to build long-term relations, so we provide guarantees and support agreements. What is more, we are always happy to assist with further developments and statistics show that for us, 97% of our clients return to us with new projects.

Success stories

Check out a few case studies that show why DATAFOREST will meet your business needs.

All Success Stories

Data Insights & Forecasting

Data Science

Retail

Business process automation

Demand forecasting

We built a sales forecasting system and optimized the volume of goods in the warehouse and the range of goods in different locations, considering each outlet's specifics. We set up a system that has processed more than 8 TB of sales data. These have helped the retail business increase revenue, improve logistics planning, and achieve other business goals.

88%

forecasting accuracy

0.9%

out-of-stock reduced

Andrew M.

CEO Luxury Goods Retail

View case study

I think what is really special about the DATAFOREST service is its flexibility, openness, and level of quality and expertise.

Data-driven marketing

We created a solution that helped optimize the customer base to get the most out of the customer data. This solution notifies the client about the services/goods, which they would likely buy, according to the gathered information.

20%

sales growth

200%

traffic boost

Jeremy Groves

CEO ThinkDigital, Digital and Advertising Agency

View case study

They developed solutions that brought value to our business.

Web app for dropshippers

The Client wanted to create a web app for people who sell products online (dropshippers) to show them the most popular products and calculate the potential profits. DATAFOREST designed and built the web app from scratch, creating high-load scraping algorithms to extract data from different eCommerce marketplaces, developing AI algorithms to calculate profits, and integrating the payment system with various functionalities.

100k+

hourly users

1,5 mln+

Shopify stores

Josef G.

CEO, Founder Software Development Agency

View case study

If we experience any problems, they come back to us with good recommendations on how the project can be improved.

DevOps Experience

The ML startup faced high costs during its growth for a data-driven platform infrastructure that processes around 30 TB per month and stores raw data for 12 months on AWS. We reduced the monthly cost from $75,000 to $22,000 and achieved 30% performance over SLA.

2k+

QPS performance

70%

cost reduction