About today's little drama in the chatterbox...
14:59 <NotFabio> DONT USE THE APOSTROPHE OF DOOM
It was fun! Weee! Can we do it again? It was perhaps a little dangerous, a little silly, and I hope nobody got hurt, but it was really fun! Gauging other's people response as witnessed in the catbox archive, I wasn't the only one who enjoyed it.
So what happened? I think in part the story starts somewhere in the LiveJournal everything2 community with a little rant of mine complaining about not enough attention from the upper level admins development-wise (I tried to avoid talking about editorial policy, whole other can of worms). One of the things about which I was complaining is that the chatterbox and /msg system for breaking lines was messing up with being able to put several HTML character entities together, such as foreign character sets like Greek.
The problem with the linebreaker basically meant that if you tried to type a word in, say, Greek or Cyrillic characters that was longer than two characters, then the linebreaker would break up the "long" word. "Long" because the linebreaker sees the actual underlying HTML code, not the symbols HTML is supposed to render instead. Hence, according to the old linebreaker, something like αβ is 12 characters, αβ, not two. So the linebreaker splits up the code itself resulting in utter gibberish if you tried to type long sequences of HTML character entities.
I had some interest in this getting fixed, because I would like to talk in /msgs and the chatterbox in Russian once in a while without using transliteration. Just because I'm anal-retentive about spelling things the way they were meant to, and demanding about the features I want for e2.
So fellow edevite call answered the call (sorry), and created a linebreaker patch to be implemented. This must have been about two weeks ago, maybe longer. Now, the way edev works is this: you can join pretty much by just asking, and once you're in edev, you have the power to see quite a bit (but not all) of the code that makes up e2. If you want to modify this code, you suggest a patch, a modification. Presumably then, the modification will get reviewed by admins, who will then approve it and incorporate it into the code.
In my LiveJournal rant I was complaining of how nobody seemed to be implementing call's very useful patch. I don't know if this is what spurred coder-type admin mkb to finally implement call's patch, despite a discussion that call had had with kthejoker about the code being somehow incorrect or obsolete.
15:01 <QXZ> IT IS TRULY THE END TIMES
And boy did that patch cause problems. It took the site down. We had 500's, 503's, and finally we were treated to nate's word galaxy generator as a confused nate came online with "wtf?" all over his face. nate left a note on the word galaxy generator saying that pages were looping endlessly and requested help on AIM. Having read myself of some activity on my LiveJournal (of which I am aware because I have email notification on for LiveJournal comments), I contacted nate asking him if mkb had already contacted him and told him what the problem was. nate answered that he was indeed talking with mkb, and a few minutes later the site was back up. Thanks nate! Thanks mkb!
This is where the real fun begins...
15:03 <NotFabio> LOL ROFL HAHAHA 1EXCLAMATIONPOINTEXCLAMATIONPOINT
15:03 <princess loulou> fabulous. bit of excitement for my dreary friday afternoon.
15:04 <glindsey> This is COMMA by far COMMA the most entertaining thing that will happen in my workday today PERIOD OR FULLSTOP FOR YOU EUROPEAN TYPES
The chatterbox was totally fubar, fuxx0r3d, if you will, and so were the Message Inboxes. Lots of Server Error Messages in bright red in the chatterbox and the same error messages in the Message Inbox except in raw, unparsed HTML. Punctuation was a problem. If anyone tried to type a message with a slash, an apostrophe, and perhaps other kinds of punctuation (I think it happened with parentheses too), then instead of the message the chatterbox would display a Server Error Message. Typing a comma would cut off everything else that followed from the message.
Chaos. Total chaos. Our agents are still trying to decipher what happened, but you can read the transcripts here if your stomach is strong enough to take it, soldier. Counterintelligence reports suggest that the universal message XML ticker was completely unaffected by the confusion, so that the archive was able to record all messages. When reading the report, remember that every message with the punctuation mentioned above would be a Server Error Message instead.
Nevertheless, noders being the little addicted puppies they are, tried to talk anyways. Tested to see what was wrong. Tested to see which characters weren't working. Tested to see if other chatterbox clients besides the nodelet were working. Joked. Laughed. Cried. Came up with ad-hoc solutions consisting of placing all punctuation in caps FULL STOP Their inboxes were very unreadable, but they tried to talk anyways. I'd like to call this debugging. At least the linebreaking across HTML character entities was fixed, but everything else seemed broken.
Enter Teh Bonci Man...
15:06 <Laura Elizabeth> Who gave edev crayons?
I bet Jay was still a little sleepy and confused this morning when he walked into the pandemonium we had created in the chatterbox. He even tried to talk in the chatterbox, and instead uttered a server error message. Oh, dear God, no. If not even Jay can talk, what is to be done? Oh, woe betide us, what is to be done?
15:09 <Chiisuta> ANARCHY NOW!
15:13 <Cool Beans> color="#CC0000"><b>Server Error (Error Id 9621944)!</b>
15:14 <Chiisuta> SERVER ERRORS ROCK
jaybonci quickly took control of the situation. In order to curb further silliness (aawww...) he shut off the public view of messages and the chatterbox and got to work. All we saw instead of chatterbox messages or our private /msgs was a note from Jay saying that he was working on the problem. During this time, a few noders were still trying to talk with other chatterbox clients. I myself was reading the catbox archive.
15:20 <Halspal> We finally got Uri to quit talking in colons
In a few more minutes, barely noticeable really, everything was ok. We emerged at the other end of the tunnel with everything working and a /msg from Jay to the edev usergroup with a brief explanation of what had happened.
2005.6.10@15:37 (edev) @% jaybonci says The broken part happened because of the way the call to htmlcode worked. It's different in 0.8 than in 1.0ish. Arguments don't perk to @_ the same way
And everything is ok.
Overall, I'm pleased. I'm very pleased. I'd like to thank nate, kthejoker, call, mkb, jaybonci and anyone else behind the scenes who participated in all of this. Thank you. You brought the server back up. You implemented a linebreaker and now I can type in funny characters to my heart's content. You fixed the patch when it wasn't working. You gave us unintentional entertainment in the catbox when things were broken. And everything happened very quickly and efficiently.
Thank you all. I take it back. I take it all back. You do care about us. There is someone looking after the database. We aren't neglected. I feel the love. Thank you.
15:30 <Swap> There were several heroes this morning. nate got the site back up because mkb could tell him what was fucked up. It's crises like this that bring the whole family together.
P. S. Upon further testing, it seems that all linebreaking has been turned off. Oh well. Maybe that will get fixed next time. I'm still glad that the site got back up and that for now I can type in Russian.