👽 clseibold

My Search Engine has now indexed more pages than Kennedy and TLGS, and has crawled more domains than Geminispace.info! :)
gemini://auragem.letz.dev/search/

2 months ago · 👍 willowf, maxheadroom, docedelgasforgot2renewcert

Links

[1] gemini://auragem.letz.dev/search/ (auragem.letz.dev)

Actions

👋 Join Station

2 Replies

👽 clseibold

@remy Thanks for informing me of how many pages that site has. I try not to get involved in what the crawler indexes, because I want it to be representative of as much of geminispace as possible, but I do sometimes make exceptions. I will think about it.

Also, the search engine doesn't actually store all of the content of a site, it only stores headings and link text, iirc. · 2 months ago

👽 remy

Your crawler is indexing gemini:/gmi.noulin.net/stackoverflow/
It has 3.2 million pages (18GB). · 2 months ago