The ML startup faced high costs during its growth for a data-driven platform infrastructure that processes around 30 TB per month and stores raw data for 12 months on AWS. We reduced the monthly cost from $75,000 to $22,000 and achieved 30% performance over SLA.
Deduce is an IT Services and IT Consulting company that protects businesses and their customers from unauthorized account access, and identity fraud.
Create AWS infrastructure that performs 2k queries per second and has 99.9 % service availability.
Optimize redundancy and 2 times decrease in cost of infrastructure.
Create failover strategy and possibilities for larger scale.
ML startup has encountered infrastructure cost issues during the extensive growth.
The main goal was to decrease the monthly operational cost ($75 K) for a large data-driven platform that handles ~ 240 bln entries monthly ( ~ 30TB), storing raw data for 12 months on AWS.
Removed managed services and set up a self-hosted DB over EC2.
Used a cluster of servers for DB sharding, adding Elasticsearch, Kafka and Redis for different streams of data-based on industry standards.
Create master/master-slave mirroring in different regions to have a failover strategy.
DB architecture was tuned to execute most-often queries faster.
Updated ETL pipelines to reduce load on DB.
Cost reduction from $75k to $22k per month with performance 30% over SLA.
Step 1 of 5
Step 2 of 5
Step 3 of 5
Step 4 of 5
Step 5 of 5
pages processed daily
These guys are fully dedicated to their client's success and go the extra mile to ensure things are done right.
Technically proficient and solution-oriented.
DATAFOREST has the best data engineering expertise we have seen on the market in recent years.
manual work reduced
Their technical knowledge and skills offer great advantages. The entire team has been extremely professional.
Share the project details – like scope, mockups, or business challenges.
We will carefully check and get back to you with the next steps.