Data Forest logo
DevOps Experience case image
Home page  / Cases / DevOps Experience

DevOps Experience

The ML startup faced high costs during its growth for a data-driven platform infrastructure that processes around 30 TB per month and stores raw data for 12 months on AWS. We reduced the monthly cost from $75,000 to $22,000 and achieved 30% performance over SLA.

2k+

QPS performance

70%

cost reduction
DevOps Experience case image

About the client

Deduce is an IT Services and IT Consulting company that protects businesses and their customers from unauthorized account access, and identity fraud.

Tech stack

AWS EC2 logo
AWS EC2
AWS RDS logo
AWS RDS
PostqreSQL logo
PostgreSQL
Python logo
Python
Kafka logo
Kafka
Redis logo
Redis

Challenges & solutions

Challenge

Create AWS infrastructure that performs 2k queries per second and has 99.9 % service availability.

Optimize redundancy and 2 times decrease in cost of infrastructure.

Create failover strategy and possibilities for larger scale.

ML startup has encountered infrastructure cost issues during the extensive growth.

The main goal was to decrease the monthly operational cost ($75 K) for a large data-driven platform that handles ~ 240 bln entries monthly ( ~ 30TB), storing raw data for 12 months on AWS.

Solution

Removed managed services and set up a self-hosted DB over EC2.

Used a cluster of servers for DB sharding, adding Elasticsearch, Kafka and Redis for different streams of data-based on industry standards.

Create master/master-slave mirroring in different regions to have a failover strategy.

DB architecture was tuned to execute most-often queries faster.

Updated ETL pipelines to reduce load on DB.

Cost reduction from $75k to $22k per month with performance 30% over SLA.

DevOps Experience slider image
gradient quote marks

They have very intelligent people on their team — people that I would gladly hire and pay for myself.

Robert P. photo

Robert P.

CTO Cybersecurity

The way we deal with your issue and achieve result

Consultation icon

Step 1 of 5

Free consultation

It's a good time to get info about each other, share values and discuss your project in detail. We will advise you on a solution and try to help to understand if we are a perfect match for you.
Analysis icon

Step 2 of 5

Discovering and feasibility analysis

One of our core values is flexibility, hence we work with either one page high level requirements or with a full pack of tech docs.  

Depending on project objectives, DevOps activity requires auditing the current approach, running metrics measurement, performing monitoring and checking logs. By having a set of interviews, we ensure that we understand the full scope of the project. Ultimately we make a project plan which we strictly follow. We are a result-oriented DevOps service provider company, and that is one of our core values as well.
Solutions icon

Step 3 of 5

Solution development

At this stage, our certified DevOps engineers refine the product backlog. We deliver great results within digital transformation, cost optimization, CI/CD setup, containerization, and, last but not least, monitoring and logging. We are a result focused company – it’s one of our core values.
Data delivery icon

Step 4 of 5

Solution delivery

After quality assurance tests are completed, we deliver solutions to the client. Though we have over 15 years of expertise in data engineering, we expect client’s participation in the project. By the way, a high-level of communication is also our core value.
Support improvement icon

Step 5 of 5

Support and continuous improvement

We understand how crucial the solutions that we code for our clients are! Our goal is to build long-term relations, so we provide guarantees and support agreements. What is more, we are always happy to assist with further developments and statistics show that for us, 97% of our clients return to us with new projects.

Success stories

Check out a few case studies that show why DATAFOREST will meet your business needs.

Data parsing

We helped a law consulting company create a unique instrument to collect and store data from millions of pages from 5 different court sites. The scraped information included PDF, Word, JPG, and other files. The scripts were automated, so the collected files were updated when information changed.
14.8 mln

pages processed daily

43 sec

updates checking

Sebastian Torrealba photo

Sebastian Torrealba

CEO, Co-Founder DeepIA, Software for the Digital Transformation
View case study
Data parsing case image
gradient quote marks

These guys are fully dedicated to their client's success and go the extra mile to ensure things are done right.

Entity Recognition

The online marketplace for cars wanted to improve search for users by adding full-text and voice search, as well as advanced search with specific options. We built a system application using Machine Learning and NLP methods to process text queries, and the Google Cloud Speech API to process audio queries. This helped greatly improve the user experience by providing a more intuitive and efficient search option for them.
2x

faster service

15%

CX boost

Brian Bowman photo

Brian Bowman

President Carsoup, automotive online marketplace
View case study
Entity Recognition preview
gradient quote marks

Technically proficient and solution-oriented.

Operating Supplement

We developed an ETL solution for a manufacturing company that combined all required data sources and made it possible to analyze information and identify bottlenecks of the process.
30+

supplier integrations

43%

cost reduction

David Schwarz photo

David Schwarz

Product Owner Biomat, Manufacturing Company
View case study
Operating Supplement case image
gradient quote marks

DATAFOREST has the best data engineering expertise we have seen on the market in recent years.

Supply chain dashboard

The client needed to optimize the work of employees by building a data source integration and reporting system to use at different management levels. Ultimately, we developed a system that unifies relevant data from all sources and stores them in a structured form, which saves more than 900 hours of manual work monthly.
900h+

manual work reduced

100+

system integrations

Michelle Nguyen photo

Michelle Nguyen

Senior Supply Chain Transformation Manager Unilever, World’s Largest Consumer Goods Company
View case study
Supply chain dashboard case image
gradient quote marks

Their technical knowledge and skills offer great advantages. The entire team has been extremely professional.

We’d love to hear from you

Share the project details – like scope, mockups, or business challenges.
We will carefully check and get back to you with the next steps.

DATAFOREST worker
DataForest, Head of Sales Department
DataForest worker
DataForest company founder
top arrow icon