Home page / Cases / Data parsing
Data parsing
We helped a law consulting company create a unique instrument to collect and store data from millions of pages from 5 different court sites. The scraped information included PDF, Word, JPG, and other files. The scripts were automated, so the collected files were updated when information changed.
14.8 mln
pages processed daily
43 sec
updates checking
About the client
Law Consulting company based in South America. Client automate, manage, classify and store files of court cases, documents and contracts of all kinds via AI algorithms.
Tech stack
Python
Celery
Pandas
PostgreSQL
Elasticsearch
GCP
Challenges & solutions
Challenge
Scrape data out of millions of pages and documents from five separate judicial websites; don’t overload them.
Daily collect updated data from the judicial cases files. Not only the structured data but also built-in PDF, Word, JPG files, etc.
Keep scripts continuously running, add new cases files and update old ones if something had changed.
Solution
Created a distributed system architecture with Linux nodes and dynamic pipeline which makes managing high peaks and set priorities possible.
Created an algorithm, which scrapes new files immediately during the daytime based on traffic and makes massive updates during the nighttime.
Created a cloud SQL database with daily dumps to Elasticsearch to keep data we use; files directly uploaded to Elasticsearch.
Implemented proxies and AI technologies used to overcome bot protection and process 14.8 million pages daily. Daily we download about 14 Gb of important data.
These guys are fully dedicated to their client's success and go the extra mile to ensure things are done right.
Steps of providing
data scraping services
Store heatmap
The electronics retailer, wanted to improve their sales and customer service by analyzing the flow of people into their stores. We created a system using Machine Learning, image detection, and face recognition. The system tracks visitors' movements and the most viewed shelves and products. This information helps the store to focus on selling popular products and to avoid unpopular ones, ultimately improving the sales process.
DATAFOREST provides meaningful shopper-behavior Insights. They are very responsive and effective, trying to engineer and offer the best fit solution.
Supply chain dashboard
The client needed to optimize the work of employees by building a data source integration and reporting system to use at different management levels. Ultimately, we developed a system that unifies relevant data from all sources and stores them in a structured form, which saves more than 900 hours of manual work monthly.
Their technical knowledge and skills offer great advantages. The entire team has been extremely professional.
Infrastructure Audit & Intelligent Notifications
An eCommerce company had issues with managing its complex IT infrastructure across multiple cloud providers. We helped to analyze the current architecture and develop a strategy for unification, scaling, monitoring, and notifications. As a result, we implemented a single cloud provider, CI/CD process, server unification, security and vulnerability mitigation actions, and improved reaction speed and reliability by 200%.
Not only are they experts in their domains, but they are also provide perfect outcomes.
Latest publications
All publications
18 min
Web Price Scraping and Price Monitoring Guide by DATAFOREST
18 min
Efficient Cloud Infrastructure Monitoring: Secure and Scalable Solutions
16 min
15 Best Data Warehousing Tools: Cloud-based Platform & Services - DATAFOREST
We’d love to hear from you
Share the project details – like scope, mockups, or business challenges.
We will carefully check and get back to you with the next steps.
