DataLake and ETL Process

QUIXOTIC 360

QUIXOTIC 360 is a SaaS platform for energy marketers (and communities). It automates repetitive processes and optimizes communication with the distributor, generating, exchanging and interpreting XML files in a bidirectional and end-to-end manner. They also bill energy, which places them between a back office and an ERP.

https://www.quixotic360.com/

Challenge

The technical challenge was urgent: extract and manage information from a daily amount of 8 million data generated in Salesforce and collected by a bot.

The objective was to implement a telemetry project that required interpolation, loss calculations, obtaining the best measurements and the ability to visualize data through queries, visual representations and an API. This large amount of data made manual data processing impossible, generating operational and human costs.

Solution

An AWS cloud solution was implemented, storing the data in Amazon S3 bucket, and managing it with AWS Step Functions workflows and AWS Glue ETL jobs, including vital interpolation and loss calculations via Lambda functions. Post-processing, data was stored in Amazon DynamoDB tables for optimization. The data was then loaded into Amazon DynamoDB tables. Data visualization and querying was enabled through Amazon Athena and a query API was developed with the API Gateway service.

This cloud-based solutionempowered QUIXOTIC 360 to conquerdata processing challenges, improving their analysis capabilities and getting a breakdown of the best measures of their data.

Results and Benefits

Increased data processing capacity. Including interpolation and loss calculations, improving data quality and reliability.

Facilitating data visualization and analytics for end users through AmazonAthena, driving collaboration and data-driven decision making.

Implementation of an API with API Gateway for programmatic access to data, promoting integration with other systems and applications.

Improved data quality and accessibility. Resulting in competitive advantage, operational efficiency and better utilization of data assets.

Ready to get started?

To learn more about how AWS and Magic Beans can help your business, contact us at team@magicbeans.pt

Portugal:

Lisboa
Rua Dr. António Cândido, 10 - 1º Andar
1050-076, Lisboa

Porto
Avenida de França, n.º 20, sala 507
4050-275, Porto

Óbidos
Parque Tecnológico de Óbidos, Edifícios Centrais, Rua da Criatividade
2510-216, Óbidos

Country Phone:
(+351) 308 814 821
Country Email:
team@magicbeans.pt

 
Netherlands:

Amsterdam
Weteringschans 165 C
1017 XD, Amsterdam

Country Email:
team@magicbeans.nl

Spain:

Barcelona
Ronda de Sant Pere, 16, Bajos
08010, Barcelona

Madrid
Paseo de la Habana, 9-11, Bajos
28036, Madrid

Country Phone:
(+34) 609 129 074
Country Email:
info@magicbeans.es

 
Belgium:

Brussels
Avenue Louise 54
Ixelles, 1050, Belgium

Country Email:
team@magicbeans.be

 

Italy:

Trieste
Riva Tommaso Gulli 12
34123 (TS)

Country Email:
info@magicbeans.it

© Copyright - | magic beans | All Rights Reserved | Powered by: valkirias