Bless me father, for I have sinned. It has been one month since my last confession. Here is what I have done and ask for the forgiveness of all:
Feb 2nd
- Changed the ways softlink works slightly until I can get the right technical underpinnings for arbitrary metadata to exclude non Google AdSense-able nodes. For instance a softlink to Assassin's Creed would cause the node to not have any ads on it when guest user sees it. I put a change in to simply hide the softlink. I normally would not have touched it, but it was eating up about 40% of our advertising revenue. Again, this will get undone once we can tag nodes on a case-by-case basis for not being advertisable.
Feb 4th
- Cleaned up all of the infrastructure stuff, and every type now exports the minimal amount necessary to capture relevant data. I still need to pare down the usergroups, but I think that's a smaller mission later on
- I took a snapshot of the production database with ecoretool and it showed me some areas where I can improve the paring down of various nodes, so I will be doing that first,and grabbing a snapshot again.
Feb 8th
- After export is mostly nailed, I now need to work on merging the development snapshot I'm working off of, and production, and that means narrowing that gap some, changing some node_ids in development, and working through the differences. A lot of development nodes are getting deleted out of git because I want it to represent production. If it is lost work for anyone, it's pretty simple to undelete them.
Feb 12th
- You can see that my github checkins are a lot about churn of individual testing nodes and creating tools to handle more change. I've been aligning my data dump of the dev server with production, including changing node_ids, etc, etc. After this has been made a lot more robust, I have my first ecoretool.pl import planned against the fields taken from an export of the live database
Feb 13th
- Removed the 'mvtest' sqltable from the mvtest type, because it doesn't exist
- Created the 'mvtest' dbtable, so that we can test phasing it out, if need be. This is purely a transitional move
- Updated the writeup at Getting the ecore development environment to work at the request of Oolong to make it a bit easier to get up and running for git newbies
Feb 14th
- Changed displayWriteupInfo (See Patch Manager for the exact code done), so that we don't go to the database again to get the hits information for a particular node; we just get it out of WRITEUP. This has the small downside of not always having the most accurate information if you get balanced to the exact same thread which has the node in cache, but whenever we need to re-get the node after it falls out of cache, it's fine. It's a pretty slim chance, and since the data is okay in the DB anyways, whichever. This one small change has slashed our database load pretty seriously.
- Commented out assign_test_condition() for now. in Everything::HTML
- Added root back into gods, since I'm going to make that the default scripted "gods" user.
Feb 15th
- Commented out the hits code live on production so that we can get better performance out of the database until we figure out a better way to do it.
Feb 19th
- Been tackling a lot more invisible crap in the site, to really clean up some old practices that I don't think are particularly scalable. These are some pre-cursors to fixing the utf-8 to latin1 problems that I'm seeing on export.
- Got homenode pictures working again. I'm so sorry folks.
Feb 20th
- Working on a new way to handle backups. I've converted all of the tables in the database over to InnoDB tables, and I'm going to be working on a way to do InnoDB checkpoint backups instead of the old mysqldump method. This is so that we have some disaster recovery available. The problem here is though that we don't have enough hardware or disk for the time being to really do it on-site, so I am using off-site resources to make it happen. For those that are curious, we are using Percona xtrabackup to handle this.
Feb 23rd
- Percona xtrabackup turned out to be kind of a dud. It was taking literally hours and the backup swelled over 6x before I cancelled it to 200Gb. So forget that. We'll now have a backup strategy that involves a read lock with --single-transaction with mysqldump.
- Created a script to dump a "staging" snapshot of the database. This will help in integration testing and in finalizing the last of the data dump and migration from our "testing" snapshot that used to live on dev1.everything2
Feb 25th
- Staging dump went in after 9 hours of processing, so that's good. It'll allow me to kind of frost that away for later use when I need to test something against a more integrated setup.
- Today for chicken-and-egg import reasons, I need to look into removing the nodetype_id from nodes and replacing it with either the type or let where it exists in the repo be implied.
- Patched Login to have a unique id for scripting purposes
Feb 27th
- Edited static javascript to add the title of the node that we are on, plus use standard JSON encoding