Why would you write a search engine?

I dunno man, I saw a post on fedi where somebody was talking about doing it and it seemed cool! The first iteration was all backed by a gigantic JSON file which quickly grew too large to handle. The current version has been ported to PostgreSQL which is obviously a little more suitable!

After the SQL refactor, I started tinkering with boolean search terms. My implementation is inelegant but works okay! It simply iterates over each word in the query and checks the next word. On an and or an or, it puts the next and previous word in an array and does some SQL sorcery to pull the data. On top of that, it pre-calculates n-grams[1] during the indexing process to enable "phrase search" with quote marks.

The feature I'm most excited about at the moment is the ability to index youtube videos by leveraging youtube-dl and Google's auto generated captions.

Currently I've managed to get it down to about 30-50ms request times which is pretty solid!

I'm not sure where I want to take this thing next. I'll probably just crawl around on the internet for a while and see when it breaks down in it's current iteration. I don't think this will ever become a public thing such as marginalia.nu [2] or anything, this is just a fun for me type of deal!