Remembered that I wanted to update my site’s robots.txt and just in general prevent unnecessary scraping of my website without my consent, so did a quick search and found these two articles that have some blocks of text I could insert into my site:
- How to block AI Crawler Bots using robots.txt file — got the list of AI scrapers from here
- How to Stop ChatGPT and AI Platforms from Scraping Your Website (2025 Guide) — specifically #2, for the meta tags
Mostly this is just me testing this out to see if the data I get from my website analytics would be different, as I did notice the other day that there were a bunch of pings to random old posts of mine (technically this helped me see that my URL routing is still scuffed, so something I need to fix sometime soon 😅) so now after setting this up, I’ll check back in a week or so to see if the same behavior can be observed.
I generally would only like to use AI when I consent to it, and I’d like for it to not randomly pull data from me without me doing so before. I do feel it’s kinda moot considering I am online a lot, so I do have a lot of data online that can be read and possibly used for whatever purpose… but I’d like to exercise a bit of agency on this aspect, since this is my personal website. :))
Let’s see how this goes in the coming days.