Table of Contents
- nannyML 5-step plan to stop the end of the human race as we know it
- We need to build data drift detection for the modern world, built from the ground up, with dynamic systems in mind.
- We need to build concept drift detection, which detects when the model output and target variables change.
- We need to build robust and accurate AI performance estimation techniques for use cases where the ground truth never comes.
- We need to keep track of business KPIs in relation to model KPIs.
Do not index
Do not index
Canonical URL
AI cannot be allowed to make paperclips. That’s right, paperclips. If allowed to do so, it will lead to the end of the human race as we know it. How, you ask? It’s simple.
AI will quickly optimize every part of the process for increased paperclip production. It will then require more resources to do so, in turn, building robots to start gathering those resources. Eventually, the resources will start to run out, leading to the creation of an army. It’ll then use that Army to take over every Aluminum mine in the world.
But it won’t stop there.
Eventually, AI will seize all means of production, separating families (a bit of a bourgeois construct anyway), and create a centralized system to produce and train human subjects…
That is a bleak future for us. Being the subjects of an AI powerhouse restructuring the world and the entire human race just to enhance paperclip production.
Fortunately for humanity, NannyML has a 5-step plan to prevent this from becoming our new reality.
nannyML 5-step plan to stop the end of the human race as we know it
The first thing to know about how to stop AI from subjugating humanity is to consider how AI works, and how it is built.
The lifeblood of AI is data. An AI engine is first trained on historical data and then deployed into a more dynamic setting. To understand how AI makes decisions, we’ll take a closer look at when the AI data is trained on changes, leading to data drift. Data drift is the idea that the underlying distribution [of data] can change over time. Detecting this drift is essential. The problem is, detecting this drift can become very complicated very fast.
Traditionally, statistical tests like the two-sample Kolmogorov–Smirnov test have been used to detect data drift. This test was created to test whether two samples of data are generated from the same population. (Note: they were not created to measure changes in dynamic systems.) However, many AI systems are greatly impacted by the decisions they make, creating feedback loops. For example, in churn rate measurements, a model can predict whether or not a customer is likely to churn. The company can then take actions to prevent a customer from churning, and then the next day the model makes a prediction on that same data point again.
Â
We need to build data drift detection for the modern world, built from the ground up, with dynamic systems in mind.
AI can be looked at as a mapping between model inputs, and model outputs. (g: X -> Y where X is the model inputs, and Y is the model outputs. g is the mapping that our AI seeks to find.) The problem is, in the real world, g is constantly changing while AI is static. This means unless the engine is re-trained or rebuilt (usually manually) the mapping that lies at the core of AI does not change. This change in mapping can be found by detecting concept drift.
We need to build concept drift detection, which detects when the model output and target variables change.
We also need to know how our AI is performing in the real world. For prediction use cases that predict the near future, it is rather simple. Simply predict something is likely to happen, wait until it happens, then compare the real world to the original prediction.
However, AI is being used to automate previously human-based tasks (like insurance premium pricing), and predict events that will happen far into the future (like loan default). In the case of automation, the future never comes. The price output generated by AI will become the real insurance price. It becomes impossible to measure real performance in production. Instead, it’s necessary then to estimate the performance of the machine learning model.
We need to build robust and accurate AI performance estimation techniques for use cases where the ground truth never comes.
When AI is developed, normally there are 2 types of KPIs, business KPIs, and technical KPIs, which are the most crucial for determining success.  Looking at churn as an example, the business metric can be something like churn rate. The technical metric can be an F_beta score. The business metric is normally set by the business stakeholders or company strategy. For example, it could dictate keeping the churn rate below 5% a month. The technical metric is then the level of performance a model needs to hit this business metric. In the case that could be an F_beta score of 0.7. Normally, this is where the tracking stops. This point here is also where AI could start taking over the world via paperclip production. But again, we swore we wouldn’t let that happen. Not on our watch.
The problem between the data drifting, the mapping between inputs and output changing, and not knowing the model performance in the absence of ground truth creates a conundrum. It becomes impossible to know whether an F_beta score of 0.7 still produced a churn rate below 5%, three months after the model is put into production. Heck, it’s not even measurable a single day after it is put into production. These two metrics are always in flux.
We need to keep track of business KPIs in relation to model KPIs.
Â
To summarise, our secret 5 step plan is:
- Build data drift detection for the modern world
- Build concept drift detection
- Build robust and accurate AI performance estimation
- Keep track of business KPIs in relation to model KPIs
- Keep AI away from paperclips‍
Â
Â
These things only lay the groundwork of a much more ambitious plan, it is only DAY 1. The big plan will transform how companies operate. It will facilitate the era of AI. This plan will allow for AI to be used in all aspects of life. It will allow executives to know what is going on across their organisation with the blink of the eye. It may even usher in an era of world peace, who knows…but for that plan you will need to have a chat with me.
Â
Ping me on WhatsApp to learn more +32468180483.
Written by
