Queuey queues for chewy chewing

Quick announcement for The Sample: I've noticed that in some cases, it's taking a long time to export your subscribers CSV, and sometimes it never finishes. I'm planning to fix that today. If you've ran into this, thanks for your patience!

Last week I got sucked into re-architecting some behind-the-scenes stuff for Yakread. It's mostly uninteresting unless you're a programmer (if you are, here are the deets—see the "in-memory queues" section). The gist of it is Yakread and The Sample involve a lot of background data syncing and computational stuff, and if you aren't organized about it, it can overwhelm the server and cause it to crash. For example, a few weeks ago I added OPML import to Yakread, which is a feature that lets you import your blog/newsletter/news website subscriptions from other reader apps. I tested it by importing a relatively small OPML file that only had a few subscriptions in it. But the first time that someone else imported their OPML file, Yakread attempted to import all of their many subscriptions at the same time and it promptly brought the server down 🙂.

I made a quick fix so that the subscriptions would be imported one-at-a-time, but it only works on a per-user basis. If a bunch of people tried to import their OPML files at the same time, Yakread could still get overwhelmed.

The correct way to handle this is to use queues (a term-of-art in software engineering): instead of importing a users' subscriptions right when they upload their OPML file, instead they should be placed on a special data structure called a queue. Then there is a different part of the code which has sole responsibility for taking the subscription import jobs off of the queue and executing them. This way we can add lots of things to the queue at the same time, but the actual work only happens one-at-a-time. Queues are useful for many things that Yakread and The Sample (and just about any other web app of sufficient size) do.

So anyway, as I was working on the new discovery stuff for Yakread, I decided I really needed to start actually using queues, so I spent a bunch of time thinking about the best way to do it, then on Thursday I added queues as a feature to my web framework (a bit like adding a hammer to a toolbox, for example). This week I'll use the new queues feature to implement the discovery stuff that I was originally planning to work on last week.