The Writer’s Block Dev Diary
(archived)
This is the development diary of my now-discontinued CMS, ‘Writer’s Block’. These posts were collapsed into this single page for permanent archiving, and are ordered from oldest to newest.
The RSS and Atom feeds now no longer display an ellipsis (…) if the text of a post or comment is shorter than 500 characters.
The RSS feed has also been extended to grab the details of a Mini-Blog.
I’ve taken down the demonstration installation and moved it here to the live site so it can be tested properly. So far I haven’t picked up on anything that hasn’t been fixed.
I’m now happy with what I see in v3.3. Any bugs left will most likely come from the install or upgrade scripts, and I cannot test those myself (my server seems to be lenient towards bad mySQL syntax) so I’ll have to ask some other people to try it out.
This release would have been made yesterday as I had promised a few people, but I thought of Mini-Blogs just recently and coding the massive Admin page for handling Mini-Blogs (it’s a full 8KBs larger than all the other Admin pages) and bugtesting the whole thing left me a bit wanting for time. But it’s better to have a late release than a buggy one, and one day late isn’t all that bad anyway.
I just sent out some notification emails to let people know that <a href=writersblock”>Writer’s Block v3.3</a> is now released. It’s heaps sexy.
I can’t think of any new features I can put into the next release. Is this the end of development? I hope so. Ending development can only mean that the software is bug-free and at its best.
No, if I can’t think of anything then subsequent releases will be bug fixes and code improvements. This might be the end of development, but there’s still room for refinement.
Of course, you can always contact me if you have an idea for a new feature. I’ve already got blogging with changeable dates and Draft status, commenting with great spam protection, non-blog pages, mini-blogs, RSS feeds for the main blog, latest comments and for each mini-blog and an Atom feed, also for the main blog. There’s multi-user capability for the Admin section with three different permissions, permalinks for pretty much everything, archive for the main blog and all mini-blogs, categories for main blog posts and non-blog pages, and excellent security checks all the way through the public site and authentication through the Admin section.
Have I missed anything the should really be in Writer’s Block?
A terrible, terrible bug that I thought I squashed before release has reared its ugly head. When you tried to change your user details using personal.php the script would never operate. It now works correctly.
The current download of writersblock-current.zip contains the fixed personal.php file. Hail!
There was a bug where Writer’s Block would display a PHP error on the generic articles page if no Anypage articles existed. It now displays a simple “No categories exist” message instead.
That fix is also in the writersblock-current.zip.
The blogging software I used to use, CheesyBlog, had a separate language file where every piece of display text output by the software was kept. This allowed people to translate the software very easily.
Maybe I should do that here too? I have two options: using a massive array CheesyBlog-style or using a lot of constants. I will have to find out which one is faster, but I have a feeling I’ll end up using arrays.
The next build of Writer’s Block will have Languages Other Than English (LOTE) support. All the English text and the comment forms now live in their own language file so that anyone who wants to make a Writer’s Block site in a language other than English just has to edit the things in the one file instead of having to trawl through the entire script.
Lots of small matters are also fixed.
Expect Writer’s Block v3.5 sometime soon.
Wow! After a total build time of just five hours, the new Writer’s Block with LOTE support is now available!
v3.5 doesn’t require an upgrade script. Simply download the upgrade and put all the included files in their right places.
I’m working on the first ever Code Rewrite release of WB. I don’t expect to add many new exciting features, just rewrite the code to make it cleaner and quicker. Right now I’m looking at eliminating as many temporary variables as possible with each temp var removed my smoking-fast software gets closer to blazing-fast.
Well actually I am looking at adding a major new feature: Trackback with Spamguard Extra’s spam protection. It will be a big boost because so few programs support Trackback, and to have a little program like mine support it would really mean a lot to me. I just hope I don’t have to change to an XML-RPC API to do it.
Actually, XML-RPC wouldn’t be too bad if I could figure out how to do it. There’s an idea hot off the brain! XML-RPC would allow WB users to connect through lots of different software and hardware, would allow integration with the browser and provide extendability! Cool! This would change the release from Code Rewrite to Dragnet/Functionality (read the ChangeLog of any of my scripts to know what I’m talking about).
Okay, so on my to-do list for Writer’s Block v4.0 (new version because of the BIG new changes planned):
That’s what I’m sayin’. The first step is to finish cleaning the existing code so that I have a clean canvas to work on. Clean code equal fewer things that can go wrong, right?
Huh. Well I shouldn’t have gotten so far ahead of myself. The specs for Trackback and XML-RPC, while informative and straight-forward, aren’t very self-explanatory. It looks like a lot for a single developer to do. I’ve decided to leave out Trackback and XML-RPC for now and instead consolidate Writer’s Block as it exists now. As such I’ll continue with my original idea of a plain code rewrite with maybe a few new features if I can think of them. Here is the abstract for the rewrite.
Goals of code rewrite
v3.6 has experienced a spurt of activity in the past few days. The documentation is being completely rewritten to be more definitive and helpful, the code has been completely looked over and made semantically correct, and that Admin section has finished being redesigned. It now looks quite nice and is much easier to use.
Right now I’m looking at all the queries run by the Public pages to see if I can add indexes to the tables to make queries move faster. Adding even one index in the right column will dramatically speed up queries on large tables.
I’m also finished redoing the authentication system. Some users had some login issues, and I think I’ve discovered why. This is straight from the mySQL documentation:
The PASSWORD() function is used by the authentication system in MySQL Server; you should not use it in your own applications.
Do’h! I took the documentation’s advice and am now using MD5 to one-way encrypt passwords for storage. It should work evenly across all servers.
The upgrade scripts for v3.5 to v3.6 are finished. I’m ready to put v3.6 Dev on my website to test it for bugs, but I’m always nervous that a new version might break my website forever (though I know that won’t happen).
More likely my apprehension is caused by the expectation of massive befuddling bugs in the code, but I’m quite happy with the UI I’ve designed. My satisfaction with a job well done always makes up for the inconvenience.
I’m betatesting the Admin section of v3.6 before I do the public files. I actually had an attack of the login bug some people were telling me about, and I believe I have demolished the bug with a two-pronged attack:
Either way the bug is done for. Yesss!
I’m now betatesting the public pages. If you see any errors this is why (though I don’t expect many errors).
The only thing I’m not satisfied with in the new User Interface is that the forms still look dumb. I could’ve applied lots of styles and stuff to them to make them look all sexy, but styles would have made it harder to know that these are actually form fields.
Still, since Writer’s Block doesn’t have a WYSIWYG interface its audience is net-savvy people; I would expect net-savvy people to know a form field in any of its guises, especially in a form-powered application like this. I’m not releasing v3.6 yet — there’s still time to style the forms, I suppose.
Maybe tonight.
Yes, the v3.6 test is going really badly. Lots of bugs in the new public pages, especially one in latestcomments.php. For some reason it outputs letters instead of numbers.
I’m now working through all the scripts, and I’ve cleared all but five of them. A long night ahead.
I uploaded the new RSS and Atom feeds because they’ve been cleared and they have lots of cool bug fixes in them, like working categories and no syntax errors. Very cool.
Everything seems to be fixed for the release of v3.6. I’m just checking everything today to make sure it works as expected.
v3.6 won’t come with a complete manual (or even the very-abridged manual) like previous releases because the manual will now live online on this site. This allows me to update the manual for everyone while keeping the size of the download down. I’m writing a script to handle the manual’s display, so I won’t release v3.6 until everything works well and the manual script is online.
Writer’s Block v3.6 is now released. I’m quite sure everything works great.
As I said in the previous post, v3.6 doesn’t come with any documentation aside from an installation guide. The upside is that the online manual has been greatly improved.
Go get it.
P.S. the latestcomments.php bug was a programmer error. I created an array called $comment and then on the very next line assigned a new value to it. Pish.
P.S. The manual is displayed with another small script I wrote, txt.page.
Spamguard’s regex is greedy. Here’s what it is now:
<a(.*)($BlockedUrls)(.*)>
What that’s supposed to mean is “find any blocked URLs inside all <a> tags. What it actually does is search the entire text of the comment after the first <a> tag. Not good.
I’m trying to write up a new regex that’s smarter, that will only scan the href element of an anchor, but it’s tough work. Maybe a regex book for Christmas would be nice.
Edit 10:56: I may have to make Spamguard more complex. What it does now is simply search the entire text until it comes across a blacklisted string, which is not what I want. What will probably happen is I will use preg_match_all() to retrieve all the links and put them into an array (or a long string), then use either a simple regex (like eregi(""(porn|sex|poker)"", $_POST['comment']); to just search for all literal strings at once) or strpos($string, $_POST['comment']) to see if the string occurs. I’m leaning more towards the simple regex because it may be faster than applying a new strpos() for each blacklist entry.
I’ve fixed the Spamguard regex. A quick run-through:
It used to scan the all the text that came after an <a> tag in the comment text, which is not what I wanted. It was only supposed to scan the contents of the href attributes in all <a> tags to find suss URLs.
The new behaviour is more complex (and probably a bit slower), but it does exactly what I want. It checks out only the href part of a link while completely ignoring the rest of the comment.
The fix will appear in v3.7 because there are some other things I’d like to change in the code. v3.7 will be a code-only upgrade: no upgrade scripts to run, just two or three files to replace.
I’ve decided that what was released as v3.6 is as far as Writer’s Block goes with a one-man development team. Without any help I don’t think I can implement XML-RPC or Trackback (which I’d love to be in there).
Subsequent versions will probably be refinements of existing features or various minor additions. Here are some additions I have slated:
I’m really interested in the flooding prevention. I can’t use sessions because PHP only supports them when built as an Apache module (not everyone uses Apache, see?) and I can’t use cookies because the system would break if a user disallowed cookies. I’m considering writing the IP and unix timestamp to an array in a log file, but it’s complicated and might slow down the script by forcing it to perform garbage collection too (to get rid of old array entries).
So, well, I guess expect flooding prevention in v3.7. It’s a really good idea.
It sounded so difficult: make a comment flooding prevention system that didn’t rely on cookies or sessions. It took me a few hours of code-bashing, but I’ve written it.
It couldn’t rely on cookies or sessions because
so what I did was use a .txt file to store the IP and timestamp from when a user first commented. When a user tries to comment again their IP is found and the current timestamp compared with the stored one + 60 (to see if a minute has elapsed). It allows commenting if 60 seconds have passed, and bans it otherwise.
I call it Levee. It’s a bit more complicated than how I explained it. I’ll release it under the Lesser General Public License so that other developers can integrate it into their software.
I was just reading some MovableType blogs that talked about comment flooding (or crapflooding) and notice how lacking my own flooding prevention library would be.
Many bloggers confirm that crapflooding software uses multiple open proxies to fake new IP addresses: Levee’s current IP tracking system assumes that a commenter is using their ‘real’ IP for all comment attempts.
Obviously this means that Levee as it stands can only block ‘part-time trolls’ and repeat comments sent from the same IP. Maybe there’s another identifier I can use.
Don’t worry, Writer’s Block v3.7 is on-track. I’m just making the commenting system more resistant to comment flooding.
What I have completed is a system that records all IPs and the time at which they commented, which stops normal single-IP from commenting before a certain amount of time has elapsed.
The second system I’m programming is a throttle, a popular means of lessening the impact of crapflooding. What it does is track the number of comments made that day, and disables commenting when a certain number of comments is reached. While this won’t prevent crapflooding, it greatly reduces its impact: you’ll only have to delete tens of comments instead of thousands.
To make clearing crapfloods easier, I’m also recoding the Moderate Comments script to let you delete multiple comments at once (which you can’t do now).
It should be said that no Writer’s Block installation to my knowledge has been crapflooded or even link spammed. This is just a security measure.
v3.7 is going very, very well. I’ve finished Levee’s IP policing and throttling, and all I have left to do is give users the ability to delete multiple comments on the one page. Currently you can only delete comments one-at-a-time, even though many comments are displayed in one page.
I also fixed the install script so it doesn’t throw up syntax errors any more. For some reason a single quote made an error even though I had escaped it properly.
I’ve finished testing the multiple delete function of the new comments.php. It works great and kicks ass.
I’m now up to testing the new commenting system with its intricate crapflooding safeguards. It should work spectacularly.
It’s time I finished beta testing v3.7. I’ve been lagging this whole time. The new commenting system with flooding prevention is now being beta tested. It should work.
Expect a release by Christmas day.
While the beta testing of the new flooding prevention is going really really badly (the code isn’t doing what it’s supposed to: I suspect it’s over-complicated), my friend Elena emailed me a translated language file.
Hurrah! Writer’s Block now really is multi-lingual!
I still intend to release v3.7 by Christmas day.
I told you I would release v3.7 by Christmas and I, apparently, have delivered.
Levee was overcomplicated and must be rewritten, so it is not included in v3.7. Still, v3.7 comes with some very important bug fixes, especially:
And a very important addition:
That surely makes things easier! Grab v3.7 now!
I’ve finally fixed the next/previous pagination links. The Next link used to point to the newest item, which was really annoying. Download the v3.6 to v3.7 patch and use the index, permalink and miniblog scripts included.
Writer’s Block is a joy to operate and look at, its bugs are either dead or very obscure and it does exactly the job it’s designed for without complication. It looks as if Writer’s Block is in its final shape.
Of course there are other features that lots of people want, like an image gallery or a funny little chat room that will probably be empty all the time.
If I do make an image gallery or chat script it will not be integrated with Writer’s Block. WB’s role is clear: to present dynamic pages of information very quickly and easily. I would prefer to make any other scripts separate, as I have already done with the shoutbox/tag board script.
As I have said many times prior, future releases of Writer’s Block (yes, there will be more) will concentrate on fixing bugs, stepping up security and making things more user-friendly for both admin and visitor.
Chalked future features include:
As well as your usual fix of smaller file sizes and faster load times.
A pleased user emailed me telling me that RSS feeds should allow links, and I totally agree. For some reason (I’m sure it was an anti-spam reason) I told the feeds to remove all HTML from feeds. The feeds in v3.8, which is soon to be released, now allow links.
In my stumbling about I also found a massive bug: typos in the Mini-Blog code limited users to the first page of a Mini-Blog’s archive. That fix will also appear in v3.8.
I should get back to working on comment flooding prevention, even though I have gotten no reports of it occuring yet. All this Super Nintendo has made me lazy.
So many spamments! I’ve been remiss in regularly checking my blog for spam comments and adding new spam URLs to the blacklist that I now have in excess of 1200 spamments spread across eight posts. I could probably block a thousand of them just by adding five or six new URL strings to the blacklist.
The other kind of spamment — the kind that doesn’t use any links — is what I’m worried about. I hope I won’t have to resort to a full-URL blacklist like Movable Type used to have (a ‘dumb’ list that requires tens of lines to reliably block one domain). Maybe I can block by given username instead.
All 1200+ spamments I recently received could actually have been blocked by Spamguard. The problem was that I had not been vigilant and kept an eye on my Recent Comments feed, and therefore hadn’t added new spam URLs to the blacklist as they appeared. Be aware, loyal Writer’s Block user, that the human is always the weakest link.
Owing to this outbreak, I’m now providing my blacklist file for download. I should be able to keep it updated from now on.
Spamguard has hit a wall: spammers are now using public hosting services. I’ve received two spamments recently, both linking to Bravenet public services like polls and guestbooks. Spamguard blocks things by scrutinising the URL a link is pointing to, seeing if any part of the URL appears in the blacklist, and blocking the entire comment if it does.
This approach fails when spammers start using public services because millions of legitimate users may very well link to the same services when commenting. If bravenet.com were to be blacklisted, then anyone who linked to Bravenet would be blocked. If I just blocked the serial number of a particular spammer, then the blacklist would grow unwieldly with all the serial numbers I’d have to track, plus there’s the risk that blocked serial numbers could be found in legitimate URLs.
I could change Spamguard’s code to also scrutinise the link text, but this would compromise the entire system; people will invariably use blacklisted strings inside link text, often for harmless reasons, and be blocked for it. As the blcklist stands, even using the word adult in such a system would result in a block.
Since I never get any Bravenet users commenting on my site, I’m blocking the entire Bravenet domain. This change has been added to the downloadable blacklist.
With spammers now starting to link to public access sites under Bravenet and CGISpy, it’s become harder for Spamguard’s blacklist to cope. Spamguard is now limited to the equally sorry options of either blacklisting all possible spam URLs, or blacklisting the entire domain. If spammers start using really popular public access services like Geocities we will be in a real predicament: what if a legitimate user wants to link to their Geocities ‘site’?
In delving into this problem I’ve noticed that only a few posts of mine were attracting spamments. Only eight posts out of currently seventy-five were being spammed.
This reminded me of other popular blogging software, where comment threads are closed after a certain period of time has elapsed, or can be made indefinitely open or closed by the administrator. By closing old comment threads automatically, they cannot be spammed, which means that your site will a) stop receiving spam, or b) have spam in only the newest posts, making it easier to police and delete.
Rockin’. I’ll get to work.
I’m now adding code to the commenting system to allow for automatic or optional closing of comment threads.
I’m also adding code to allow for email and body text filters. These filters are very, very general. They’re likely to match most anything, so users should continue relying on Spamguard’s URL string blacklist.
I’m very slowly getting work done on WB 3.8. It’s hard to code sweeping new features when you wrote the program so long ago that you barely recognise any of the code. I find myself having to learn PHP and mySQL all over again.
But we’ll get there.
3.8 is a security update to counter spamments, and will include new features like mass deletion of comments using URL fragments (built on Spamguard’s blacklist system), broader blacklist options and a News box on the Admin section’s index page so that you’ll know exactly when I’ve updated the blacklist or fixed a bug.
Much of my time developing WB v3.8 has been employed in finding and squashing bugs in the existing code. Take, for instance, the serious bug I found not ten minutes ago, where submitting an empty form to a tool that deletes comments by searching them for keywords resulted in the deletion of all my previous comments.
Oh yes.
I cannot stop laughing.
Great inroads were made into WB v3.8 last night. Things are being done, man.
I got comment thread closing to work, which you can see at work now (and which completely stopped the flow of spamments for the night). Spamguard now has separate blacklists for email addresses and body text, and some new entries were added to the language file to reflect the new comment closing feature.
I’m currently working on wildcards for the blacklists so they can flexibly match patterns, and also developing a mass-delete utility that uses Spamguard’s keyword/pattern matching system to delete many spamments at once.
| As you can see from the footer section of Paper Tiger, the site is now running on the Dev build of Writer’s Block v3.8. I got automatic comment closing working, and also employed blacklists on the Email and Body fields of a comment. Wildcards from REGEX like a period (.) standing for any character are also confirmed to be safe for use on all blacklists, so I can now use a pattern with wildcards to condense several blacklist entries into one(e.g. *(Well.done | (Nice | Good).site | (Good | Great).work | Good.design | Thank.you)! .<a would take seven or more entries without the new extensions). |
Couple that with a mass-delete utility that operates on Spamguard-like keywords and can delete comments based on a URL search, body text search, or both.
What you have is a robust anti-spam package. I used to get 500 new spamments a day. Since I implemented these new features, I have not had a single new spamment for an entire week. I’ve had to create spam to test the features.
It’s that awesome.
This weekend I’ll add a News box to the front page of the Admin section, sort out the install script, check for major bugs and send out emails to all known Writer’s Block users telling them that Writer’s Block v3.8 is out.
Spammers, where is your messiah now?
I’ve decided not to include a news box. A news box that called up a simple text file from my domain would require certain PHP settings to be enabled (which may not be consistent across servers) and might introduce security vulnerabilities. I may set up an occassional mailing list for new blacklist releases or something, but a regular news service is unnecessary seeing as active development on new Writer’s Block features is winding down.
A Writer’s Block v3.8 release this weekend. Promise.
Before the v3.8 release I’m going to spruce up WB’s pages here on Paper Tiger. More screenshots will be added to a condensed main page, and new entries will be made to the WB Online Manual while old ones are rewritten.
Otherwise, v3.8 looks set to rock and roll. Rock and rooooll!
v3.8 is now released. The manual has been updated to explain the new features, being the proper use of the blacklists and comment closing.
WB’s About page has also been spruced up. It looks much more interesting now.
I’ve made a late addition to v3.8: entry summaries. In the archive pages of the main blog or Mini Blogs, a new template tag called {SUMMARY} in the language file lets you insert the first sentence (or, failing that, the first hundred characters) of an entry to use as a teaser.
I’ve also allowed the use of {POST_ID} in the permalink template.
Both these features will be available in a v3.8a patch, soon to be released. Maybe I can have another brainwave before then.
I figured out that the display error that people got when looking at desiquintans.com/writersblock (the CSS would not be linked properly, and therefore the page would be entirely bland) was caused by the server being confused about whether it should serve articles.php?page=writersblock (which is what we want) or the /writersblock/ folder which holds the WB download files. It ended up doing a bit of both (/writersblock/?page=writersblock) and so there was an error.
Don’t name Anypages the same thing as you’ve named a folder.
I fixed an RSS bug across both Writer’s Block and Ligature that made feed links faulty if the installation was in a subdirectory.
Just download the installation .zip again and replace you existing rss.php and atom.php with the new copies.
My Crafts
My Software
My Reading
My 3D Printer
About
Contact
Sitemap