Hey, it’s Wiljan from NannyML. Indeed, a lot has happened since our last launch here. I have been mostly talking to data scientists for the past few months. So we can understand their pains and needs better, allowing us to build a better product for them. We have also been talking to business users, model owners, and other stakeholders. AI models are almost never isolated services and most of the time part of a bigger business process. One of the questions I often get is why we need to monitor AI models after they have been deployed. I thought I share my answer here: AI models are trained using historical data. Yet, they are tasked to make predictions on data of the present. Today's data might be very different from yesterday's data. The world is constantly changing and so does the data it generates. Assumptions and patterns that were true during training won’t hold forever after deployment. AI models are bound to deteriorate. It’s a natural process and a matter of when not if!

Hey Product Hunt! Hakim here from NannyML 👋 Today we are launching NannyML v0.8.0. In the past 6 Months we have added: - Regression Support (with Direct Loss Estimation) - Command Line Interface - Database exports and cloud integration - Official docker container - A ton of new univariate data drift tests (Jensen-Shannon Distance and L-infinity distance) - A bunch more improvements All of these things bring you closer to having a complete view of your ML performance in production, catching failures, and resolving them as fast as possible. Most notable is regression support. NannyML now allows you to detect silent model failure on all ML models with tabular data that are classification or regression, integrated anywhere, with any stack. After the launch of NannyML Open Source 6 months ago here on product hunt, we have put our heads down and kept iterating on the library with your input. 8 major releases have happened since, alongside 15 minor releases. Your input has been vital to this velocity and has allowed us to continuing to build a library loved by 1000’s of data scientists worldwide. I hope you like these new features and I am looking forward to hearing all of your feedback! - H

Hey, Niels the only engineer here. It’s been amazing to see how much our team has achieved since our last launch. We launched mission-critical features such as support for regression and expanded the range of drift detection methods. We turned a library into a CLI and a container, exporting our metrics into databases and dashboards, pushing forward to give NannyML a well-deserved place within the fast-moving MLOps landscape. We’ve completely overhauled and reinvented ourselves to make NannyML as easy to use as we can. I’m already looking forward to tackling the next set of challenges together with this group of exceptional people. Give NannyML a try, I hope you love it as much as I do.

Congrats on the launch! 🎉🚀 I've been following Hakim and the team for a while now and I'm really impressed with their progress and also the overall approach. Although I'm not currently an active AI/ML practitioner, I understand the problem they are solving and it is definitely a big one and will become only more and more evident in the future. Btw the CLI looks amazing! The dog with 🕶 is awesome :D

Hey, Jakub here, research data scientist at Nanny! Over a year ago, when I found out that NannyML is trying to estimate performance of ML models without having targets I was pretty sure it’s impossible so I quit my job and joined them. I was proven wrong when we came up with CBPE that estimates performance of classification models without ground truth. Now we have a completely different (yet similarly simple) algorithm to estimate performance of regression models. I’m twice as wrong as I was after releasing CBPE. When I joined Nanny I also said that detecting concept shift without targets is impossible, I guess you now know what we are doing now :) Reach me out to discuss anything!

@pymaster210 great product

Hello, I am Nik, a Research Data Scientist at NannyML. I am very happy to share that my journey with NannyML continues. 6 months ago we launched our open source library supporting model monitoring for binary classification models. We have added a lot more features since then and are now launching support for monitoring of regression models! We will not stop here. Please join us in enhancing your Model Risk practices and helping to improve your AI decision making.

Go NannyML :D Congrats guys!

yay =) as the life sciences have a lot of regression models when trying to ascertain the evolution of a process along stages this will enable the pool of users for NannyML to increase 😊 Best of luck! You are all fantastic !

That's so cool! Great job guys. Looking forward to test this out. :)

Hey, I’m Wojtek, co-founder of NannyML and responsible for Product and Research. Support for regression is a big deal, especially from a Performance Estimation perspective. As far as I know, NannyML is the only product (OSS or not) that can estimate the performance of your models without ground truth. The core algorithm we developed for the Performance Estimation of Regression models (Direct Loss Estimation) is surprisingly simple. Here’s a simple recipe: 1. Take model features, model outputs, and targets for the period where target data is known. It can be your test set. 2. Construct the performance target: compute the loss metric you’re interested in, like Squared Error, for every data point. 3. Train the monitoring model that takes model features, model outputs, and targets as inputs and predicts performance targets. 4. Use this monitoring model to predict this loss for every point where you don’t have targets. This is your analysis period. 5. Take the mean of the loss per point to get the Mean Squared Error. 6. Profit! And it works well in practice! NannyML uses a familiar .fit() and .estimate() interface, so you can quickly implement it with two lines of code. Check it out on our GitHub: https://github.com/NannyML/nannyml

Wow, another good one from the Nanny. Good luck guys!

What a team and what a project!!!

Congratulation on the release! I've been following NannyML for a while now, and feel we need to make something together!

This is awesome guys! Congrats on your launch! 🍾🎉🥂🎊

Congrats on the new launch!!

NannyML's Product Designer here! With this update we’ve made such big strides forward in terms of usability, flexibility, and catering to even more use cases. Huge thanks to all the user testers and research participants who have helped us to learn more about all the different ways people use ML models, how they monitor them, and how they want to monitor them. In the year since I joined NannyML we’ve come leaps and bounds forward with our understanding, and hope this update will be the first of many that make a real difference to folks who want to better understand how their models are working once they’re running.

Congrats NannyML team, on the launch.

Congrats @hakim_elakhrass @pymaster210 @wojtekkuberski and team on the progress. It would be super interesting from now. Keep going! :D

Congrats to the whole NannyML team for building such an awesome tool!