Llms.txt Generator — Get your site AI ready with a llms.txt

Hi everyone! I've been adding llms.txt to most of my projects over the past few weeks and decided to build a free generator that anyone can use to create a basic llms.txt for their sites. If you're not familiar with what llms.txt is, it's a proposal to add a markdown file to your site that provides brief background information and guidance, along with links to markdown files (which can also link to external sites) providing more detailed information. This can be used, for instance, in order to provide information necessary for coders to use a library, or as part of research to learn about a person or organization and so forth. You can read more about this proposal here: https://llmstxt.org/ The generator will take your URL, crawl your site to get your pages, and then use AI to summarize the site and each page and provide a formatted llms.txt that you can add to your site. Would love to hear your feedback and suggestions! Cheers, Herman

Very cool initiative! I guess this is needed so that AIs can use less context memory while scraping the web? And as such use less energy etc. Good for the environment if all of the web starts to use it etc.

Fantastic concept @hermanschutte.