DataOps

Continuous monitoring of the health and quality of your data.

Compared to other roles, DataOps is relatively new in the data industry.

 

Crimson Macaw’s take on the role is to apply their data expertise and DevOps best practices to data pipelines, monitoring the processes so that the health of the data can be surfaced to Business Intelligence Dashboards – your stakeholders should be informed of the quality of the data before making any decisions on the data presented!

 Illustration

What makes DataOps hard?

DataOps involves knowing what normal look like. Does your data source normally supply a thousand records an hour, but it suddenly supplies only ten records an hour?

Everything may be running operationally, and data is flowing as it should, but a change in volume, frequency or even distribution of data records themselves can all have an effect on the system.

Training Data Science models on unhealthy data can have a negative effect, stopping this data from being applied to your machine learning algorithms could be critical.

How can we help

Our DataOps Engineers will identify the key measures required to understand if a data source, data integration or data transformation process is operating normally. They will perform the analysis to understand what normal looks like and set up the required alerting rules to highlight any anomalies.

Any identified issues will be classed as a new data source, following a normal Data Engineering process so that they can be referenced by the downstream process that depends on that data.

Where we can help

Our DataOps Engineers can work closely with:

DevOps to define metrics and alerts within an existing system

Data Engineering to ensure that the relevant and important metrics are being produced

Business Intelligence Engineers to surface data system health into your Data Warehouse and Reports

Robert Bruce Chief Technical Officer

Data Engineering is headed by our CTO, Robert Bruce, who is one of the four founding partners of Crimson Macaw. Robert was one of the first 10 people in the UK to be certified as a Databricks Apache Spark Engineer.

Robert has worked in IT for over 20 years, in multiple industries from telecommunications, retail and data aggregators, applying his passion for software quality to every aspect of his work.

He has always been hands-on and creative. In his spare time, Robert enjoys woodworking and carpentry.

Testimonial Logo

We have lots of data and information on the transport network and the challenge is capturing all this information, the dependencies and getting insight from it. When we don’t have to worry about technology, infrastructure, storage or performance, it means we can concentrate and focus on getting insight to improve the transport network and travel of people in Greater Manchester.

Malcolm Lowe, Head of IT at Transport for Greater Manchester

Our experts can help you uncover the hidden stories in your data.

Contact us today to learn more.

    Subscribe to our newsletter