Innovating Cron: Announcing Norc
Last week at Python NYC we open sourced Norc, a task management system that replaces Unix cron. At Perpetually we let anyone archive any web site on any schedule. One of the big challenges we faced early on was to create a flexible, traceable and scalable scheduling system to handle this problem. While cron is great, it’s not geared toward solving this problem: Tasks are tied to a single computer, and they’re managed independently for each host and user from the command line. In addition, with cron we’d have to build all sorts of infrastructure to handle error reporting, logging, etc. So there was a clear need for something better.
After looking at systems like AutoSys, Amazon’s SQS and RabbitMQ, it didn’t seem like anyone had solved this problem in a simple, open way. AutoSys is a closed, proprietary system, and SQS / RabbitMQ are queueing systems, without support for scheduling.
So I started developing Norc back in April, and for the last few months it’s been the backbone for all our scheduled archiving tasks. As we built out our infrastructure we started to face the same system administrative tasks as everyone else: various system and database backups, daily reports, that kind of thing. While cron is typically used for managing this kind of thing, the self-documenting, fault-tolerant and error reporting features of Norc proved a more compelling solution.
It’s critical technology for us, and we think a lot of people could get a lot of value out of it. There’s also a lot more that could be done to extend Norc, such as building an awesome web-based or mobile front-end. That’s why we’re open sourcing it: We hope it proves useful for others, and we hope it develops into a more mature product.
So, see & check out the source code on Git Hub:
And let us know what you think!
Also, we’re hiring developers, so please contact us if you want to work on something awesome in New York!
Update: There’s some great discussion of Norc happening at Hacker News.
