Braintrust
p/brainstrust
Rapidly ship AI without guesswork
Ryan Hoover
Braintrust — Rapidly ship AI without guesswork
Featured
54
Evaluate your AI applications with Braintrust: the enterprise-grade stack for building high quality AI products. From experiment tracking, to prompt playground, to data management, we take uncertainty and tedium out of shipping AI.
Replies
Ankur Goyal
Thanks so much for hunting us @rrhoover We're excited to introduce Braintrust, a platform for running and tracking AI evaluations (“evals”) [1]. At my previous startup Impira and leading AI at Figma, we had this recurring problem where we never knew if changes we made to our products would improve or regress key user scenarios. We built some tooling to solve this problem and after talking to other developers learned that it was a widespread issue. Specifically, it’s challenging to establish a great dev loop that lets you systematically improve and ship high quality AI products. We worked with the teams at Zapier, Coda, and Replit to refine Braintrust. We consistently heard that they were facing challenges with evaluation, so we built Braintrust to help them. Today we’re releasing the product for anyone to use — including a free plan [3] There are a lot of LLM tooling products on the market. Here are a few ways Braintrust is different: - Rather than showing eval metrics in your observability tools, Braintrust offers an “experiment tracking” workflow, meaning you can try out changes while developing them, and drill down into diffs between other experiments and git branches before you ship. Check out our docs [3] for more details. - We believe strongly that you should own your data and support on-premises and private VPC deployments. - We natively and equally support Typescript and Python. - We have a flexible free plan for builders and an unlimited free plan for academic and non-commercial open source projects [3]. We will introduce a self-service paid (”Pro”) tier, hopefully with feedback from this community. Our mission is to enable developers to build high quality, reliable AI products. We couldn’t be doing this without Elad Gil, who helped me incubate the initial idea and team, which today includes founding designer, Coleen Baik, and founding engineer, Manu Goyal. Also big thanks to David Song, from Elad’s team, who is also helping us. We’re excited to launch today [4], but we know there’s a lot left to build and are excited to hear your feedback. [1] https://www.braintrustdata.com [2] http://www.braintrustdata.com/do... [3] https://www.braintrustdata.com/p... [4] https://www.braintrustdata.com/b...
Ryan Hoover
We're seeing a massive shift in how some software products are built, which of course introduces new challenges not covered by incumbent infra. Braintrust is worth watching, founded by @ankur_goyal, former Head of ML Platform at Figma (by way of acquisition).
Adam Lui
@ankur_goyal @rrhoover why does it say "Featured on September 27th, 2023" already? If you hunt it it's automatic? (And if so, why is it set that way)
Ankur Goyal
@rrhoover Thanks for hunting us!!
Kenny
Braintrust has significantly optimized Coda's AI operations! Initially, we started with a rather manual process for GenAI evals, and with Braintrust, we've implemented a robust workflow that facilitates scalable evaluations. Moreover, their datasets feature has streamlined the management of benchmark datasets, thereby minimizing the workload on our AI team. This efficiency enables us to concentrate more on feature and model development tasks.
David Kossnick
@siuheihk +1, using Braintrust at Coda has sped our team way up. We now have a single link we can send to understand how a feature's AI is performing, not to mention what's working and failing.
Ankur Goyal
@siuheihk Thanks for the kind words! We have learned so much from collaborating with you. It's great to work with AI innovators :)
kiwicopple
this is 🔥
Ankur Goyal
@copple Thank you!! Hopefully we can help avoid 🔥's in prod :)
Naoto Shibata
I like this challenge. We use AI for our product, morphdb.io , but I have huge issue of evaluation of AI we’re building. I want to try it and we provide AI infrastructure with our client using Morph as well. Your product seems help us to deliver!
Anton Cherkasov
This sounds like a great tool! How does Braintrust handle the privacy and security of user data, especially for on-premises and private VPC deployments?
Ankur Goyal
@a007mr Thanks Anton! When you run the product in your own VPC, we guarantee that your data never touches our servers -- your browser talks directly to the VPC to request and analyze data. You can read more about our architecture (https://www.braintrustdata.com/d...) and authentication model (https://www.braintrustdata.com/d...) in our docs.
Cai GoGwilt
Congrats on the launch!! This is a massive problem for productionizing AI agents!
Ankur Goyal
@cai_gogwilt1 Thank you!! Indeed, no small task.
Reid Robinson
It's been amazing getting to see Ankur Goyal work on Braintrust to help those building with AI evaluate these fun non-deterministic models. 😅 At Zapier, we've used it to successfully measure and improve our AI-first products. 📈 Congrats on the launch Braintrust team! 🚀
Vitor Balocco
We've been early users of @braintrustdata at @zapier and is has been instrumental in our efforts to measure and improve our AI features while avoiding regressions! Definitely recommend checking them out!
Esus 🛡️
Congrats on the launch! Looks lovely. Love that it's unlimited free for open source projects ⚡️
Ankur Goyal
@esus Thanks so much!! Please try out the product and send us your feedback!
Marie Schneegans
congrats on the launch Ankur and team! 🚀
Ankur Goyal
@marie_schneegans2 Thank you so much!!! Hopefully we can help Markprompt at some point :)
Dan Tegzes
As an ML engineer, I'm curious to try it Kudos on the launch!
Ankur Goyal
@dan_tegzes Thank you!! You can sign up directly on https://braintrustdata.com/. Let us know your feedback! Questions, comments, complaints :)
Maksim Mamchur
Congrats on the launch! It's refreshing to see a product that prioritizes data ownership and provides such comprehensive language support. Their commitment to fostering the community with plans for academia and open source projects speaks volumes about their vision🔥
Ankur Goyal
@maksim_mamchur Thank you!! Indeed, we think data ownership is existential to businesses especially with all the forthcoming innovation fueled by AI. I'm glad our vision resonates with you.
Olivia Martinez
You're doing a great job. Keep going. Such a wonderful launch. 💯
Ankur Goyal
@mirah_shah Thank you so much!! Appreciate the kind words
Debashree Ghosh
Congratulations for the great launch.
Vivienne Stricklands
You have done an amazing job here. Congratulations on the launch.
Ankur Goyal
@vivienne_stricklands Thanks so much!! Please let us know if you take the product for a spin.
Ajinkya Bhat | Notion X Startups
I've built user facing interfaces with AI backends before and I see exactly what Braintrust is doing! Clear pain point here, I've faced it way too many times! Awesome idea :)
Steve Lewis
Congrats Anur & team on the launch!
Ankur Goyal
@steve_lw Thanks so much!!
Ghost Kitty
Comment Deleted
Ankur Goyal
@servelogy_technologies Thanks so much!! Please let us know if you take the product for a spin :)
Bill
congrats @ankur_goyal!
Ankur Goyal
@bllchmbrs Thanks so much!! Excited to catch up sometime soon :)