engine room hey everyone shep here again.

its been a month since my last update and i wanted to check in even though we havent dropped any shiny new features yet. this month has been all about infrastructure - boring but critical stuff that keeps janitorai running.

the invisible work that matters

our server have been hitting capacity limits with our growth so we have been migrating our infrastructure while keeping everything running. which has been super challenging but will be done soon (its a really fun problem to have).

migration stress

tech improvements under the hood

we completely revamped our gpu infrastructure for better cache sharing between nodes. in simple terms, our servers can now communicate more efficiently with each other.

what we are building is a kvcache centered system inspired by projects like mooncake and lmcache. this means we can efficiently share the heavy computation across different nodes in our gpu cluster.

so basically our system now handles key value caches in a distributed way, allowing us to decouple prefill (llm inputs) and decode (llm outputs) phases. this all reduces bandwidth bottlenecks and lets us efficiently cache computation for individual users. basically saving the top parts of each users prompts for quicker processing.

the results are already showing:

  • we can handle more concurrent users
  • significantly improved stability
  • a better substrate to build new features off of
  • reduced costs while scaling up

huge shoutout to leon for this one!

speed improvement

why does the site go down? a few reasons:

  • most recently its been due to our infrastructure migration for the new llm system
  • database growing pains - weve hit scale faster than expected and our db setup needs upgrades
  • sometimes i break something on accident during updates (our bad, part of being a dev lol)
  • occasional traffic spikes that temporarily overwhelm the servers

the good news is most of these are temporary and the migration work will make the site much more stable in the long run.

community growth

even with zero marketing we keep growing. your word of mouth is everything.

we also hit a new record last week - 1.93M people using janitorai in a single day. sweating watching those numbers.

whats coming next

the lore system is still in development and progressing well. im excited about how its shaping up but dont want to overpromise on timeline or specific features yet.

we are taking our time to make sure it works properly at scale before releasing it to everyone. quality matters more than rushing features out.

community highlights

these creators have been building some of the most impressive profiles on the platform:

speaking of custom css profiles, check out tigerdroppeds css compendium for learning css and this template for getting started, and check out this more detailed css guide and this channel in our discord profile stay cool, keep creating.