I wanted to give you all a bit of an update for what I'm working on and what my priorities for the next few months are. I'll warn you right now, my eye is towards the tech and the nuts and bolts underpinnings of the site, so this may lean a bit technical.
Publicity and growth
The first thing, which is complete, is to stabilize the publicity and self-sustainability to the site. I've recently let googlebot back in, and turned on the webmaster tools for the site. The first thing that you'd all be pretty interested to know is that it's kind of hard to come across the site. If you look at the kinds of search queries that bring you here, you find very specific references to very specific articles. What would you say this site is about? "Writing help
"? "A database of ideas
"?. Part of our lack of identity hurts traffic and future growth. Without the slashdot plug that we had been getting, it makes it so that individual pieces of content is the biggest driver. There are a lot of solutions for this, and I'll get to them a bit later in this article.
We are again looking at a hosting change, and are in talks with University of Michigan to see what they can provide. We're hyper focused on availability and site stability as they are serious impediments to growth. Those talks will be continuing hopefully next week.
On the coding front, I've been spending lots of time on the infrastructure basis for the site. In the short term, this means not a lot of very visible updates as I make our change management and release structure saner. The code is over at github
so feel free to message me here if you'd like to contribute. I make a push every couple of days, so you'll see a steady stream up updates. The priorities there, in order are:
- Get all of the nodetypes in the database to cleanly export to XML so that we can work out of, bootstrap off from, and diff each of the site-important nodes in the database. This allows us to work in local disposable sandboxes and check in XML to git, and then we can diff it against production.
- Develop a new nodetype for mysql stored procedures, since the site uses a few. They aren't captured in the current bootstrapping environment, so that's one of the challenges there.
- Develop a new nodetype for database migrations. Essentially, we need to be able to modify the database schema, but capture the upgrade in a way that can be tested non-live.
- Export the live database and reintegrate it with the development sanitized snapshot that Swap put together.
- Test an upgrade and a migration on the live server
Once every piece of the above system works, then we can move on to making larger changes and picking up the visible pace some more. Compilation is a medium-term performance measuring step, and that is a big technical change to undertake. Instead of running code out of the database directly, we'll be running it out of a specially written perl module which will emulate the evaluation environment via some symbol table trickery. This will reduce the CPU of the site greatly, the database load, and the memory load on the servers. The only downside is that it is technically very complicated, and will take a lot of testing to get right. A selenium test harness entry for every one of the above bugs is going to be how we get through this and other items without regressing.
Looking ahead to the next few months, there are lots of very basic changes that I'd like to make. I'm open to feedback on these items certainly.
Easier node-specific collaboration. I think that discussion on nodes would be an incredibly huge win (displaytype=discuss ?), given a few constraints. Discussion should have three modes: Off, private to the author (and editors and admins), and On. Public commentary should be votable, and comments should be able to be flagged for abuse if need be. Collaborative discussion on items is something that we overload the chatterbox for, and we already have code in place to make it work.
We should start to consider what changes we need to make to add a layer of structure on top of E2 to make things easier to find. I'm going to give some examples, but these are by no means inclusive or exclusive of what the final piece might look like. For instance, we should support structured sections that people can add very specific types of content to, and with my next point (arbitrary metadata) add easier ways to find that content. Here's an example:
Let's say that you or a user goes to fiction.everything2.com. The default page might show you the ability to browse by genre, or by top authors this week. It might encourage people the right way to start considering a fiction writeup, and what some resources might be for it. Everything in fiction.everything2 might also be in writing.everything2, a larger more inclusive slice of the database that shows prose, poetry, non-fiction, etc. Search would only show things from inside the section (unless a box is checked), and softlinks from outside the section might be hidden. If a user doesn't want to stay inside of their section, they can go back to www.everything2 and that will be everything.
A user goes to recipes.everything2. Here are tons of new and different recipes, and a user can search by tagged metadata, such as what the cooking time is, or perhaps for types of ingredients. The writeups here are more structured, and there's a different entry and editing procedure, where ingredients have specific data associated with them, so that it'd be simple to have a button which displays things in metric instead of cups, tablespoons, etc.
There could also be tech.everything2, where you'd have bits associated with different programming languages or platforms.
The goal is to make it so that you can find something in the thick of things, but then if you want to browse to something else random, go to a completely different section if you so choose.
My next short-term big idea is that the site needs the ability to have arbitrary metadata to each type of node. I'm still tossing around the proper implementation for this, so it might be some time, but we need to be able to "tag" nodes in a structured way for things like "adult content: sex" or "insured" or "non-fiction" or "english" etc. By having better information about writeups, the system can help people find content, avoid content, or allow for smarter editing.
I have a pretty long series of thoughts about how to reform the reward system on the site, and allow for more gradual editing powers, but that's another pretty long node in and of itself. One quick peek into it is that users above a certain level should be able to propose edits to a particular writeup, which would show up below the writeup, and might be able to be voted on. There might also be some process to make them each their own writeups in a way.
Lastly for right now ( though this is not inclusive by any means ), I'd like an explicit setting on writeups for whether or not you want to allow derivative works. I think that we can shape the culture to be more in favor of collaboration, but I think it's fair to allow for authors to say in a blanket fashion, "I don't want derivative works". This means, no small edits, no translations, no major reworks. Right now, this is enforced socially which is problematic. A writeup that doesn't allow for derivative works would be unpublished by the editorial staff if it is wrong or if it absolutely required an edit.