I’ll be developing crabgrass for (at least) the next 6 month and want to keep everyone updated with the progress. You can find the development roadmap on the group wiki.
Week 1 & 2
For the first month I am mostly focussing on upgrading rails and the gems (libraries) we are using. I want to get crabgrass to at least work on top of rails 3.2. As that is still receiving security updates as of today. Updating the underlying libraries requires quite a lot of code changes all over the place so it can introduce new bugs. That’s why i want to get it out of the way as early as possible so I can make crabgrass more stable in the coming months. Upgrading is also a good start to become more familiar with the code base again and it will allow me to fix bugs using the more recent syntax.
Besides the upgrade I started out focussing on a few database performance issues i had observed. I also looked into optimizing caching of things that are reused on every page.
I will hopefully finish the rails 3.2 upgrade tomorrow. I will use the rest of this week to test it on the development server, write tests for bugs i run into and fix them. At the end of the week i will decide if i dare push things further and migrate to rails4. Rails4 comes with strong_parameters, nested_caching for views and turbo links. Each of them are very interesting but also might have their issues with the current codebase. So i will probably play with the gems available for rails 3.2 this week to find out if a migration seems feasible.
All in all I still run into lot of things in the codebase i would like to improve. I try to focus on the dependencies for now and leave other improvement for later.
TLDR: Things are going as planned. Crabgrass now runs on rails 3.2 and we’ll stick to that for now.
The upgrade went rather smooth. I also started cleaning up the use of deprecated code. However we are still using rails plugins which are deprecated as of rails 3.2 and will be removed in rails 4. I replaced all external plugins with more recent gems. I also moved two really small ones that are not maintained anymore into our own lib folder and removed one entirely.
Plugins are completely removed in rails 4.0. I cleaned up the external ones but we also make use of plugins for a lot of things inside crabgrass – namely the page types and install specific mods. We also use them for a number of internal libraries. So moving to rails 4.0 will require quite a bit of moving things around.
In addition rails 4.0 is phasing out of security support. Once rails 4.2 is released only rails 3.2, 4.1 and 4.2 will be supported. 3.2 will be supported until rails 5 is release middle of next year. So I will not migrate to rails 4 for now.
However rails 4 has a number of interesting additions that have been backported to rails 3.2:
I already included 1).
plans for week 4 – tests and security
I ran into a few smaller security issues this week. We were too permissive in a few places. So I will use this week to make our security policies more strict:
At the same time we should have tests that fail if we are being too restrictive or too permissive. So I will also make sure there are tests for all controllers and they verify the permissions in use.
In short… I only got to the tests part of “tests and security” last week. However that worked out really well and i will focus on security this week.
First look at route security
We had a few routes for accessing controllers from the different page types that were not specific enough. For example they allowed to send GET requests were POST should have been required or they allowed specifying the controller and action directly. This is a security issue because one can construct urls from it that would trigger actions in ways they were not meant to be used. Allowing GET instead of POST also means these urls can be triggered from links and sources loaded automatically on pages such as images.
From this it was quite clear that the page routes would need a rewrite.
test one two – test test
So now we have a set of tests that test the main functionality of all page types by “clicking through” them for us.
back to security
Looking at the routes again I decided it would make sense to turn the overly flexible routes into restful routes. Restful routes make use of the standard http verbs to indicate the type of action that should happen: GET, POST, PUT, DELETE. GET will lead to actions that show things (show and index) only and thus can not be used to trick people into changing data.
At the same time we need a bit of flexibility so that page types can use their own controllers without having to specify routes for each and every one of them. Turns out there is a way to combine restful routes with picking your own controller:
scope path: 'pages/:page_id/:controller' do resources :page_items, path: '' end
This will determine the action using the HTTP verbs and thus cleanly separate GET from POST actions and still allow to pick any controller in the url.
plans for week 5
The new routing limits the actions that can be used in page controllers to the default ones. I added sort and print actions for good measure because they are used by a lot of page types. But still some page controllers have to be split up. We have already tried to follow this pattern during the last development sprint but had not applied it to all page types. Now I am going through the remaining page types and turning actions like ‘add_task’ in the TaskListsController into default actions like ‘create’ on a new TasksController. I will definitly finish the route rework this week. I am also looking into adding strong_parameters as a way of securing against manipulated parameters. But I am not sure i can finish that this week.
I notice that this report got a bit technical I’m happy to explain the technical parts in plain english. Feel free to ask.
The big picture
The first month is almost over. I’ve focussed on the bigger reworks that touched large parts of the code. I will finishing the security changes this week and parts of the next. There’s still a bunch of larger changes I would like to make such as upgrading to newer versions of rails and ruby, using jquery instead of prototype and so on.
However I will leave those things for later and move on to the bug fixing phase. I will go through the bug reports, add tests for them and fix them one after the other.
Thanks a lot WhilelM. I have not deployed the current state to the test server yet.
I think i will do so next week. It would be great if you could play with it some at that point. In particular i might have fixed some bugs during the rewrite. I’ll go through the list of bugs and assign priorities. Maybe I can send some your way at that time so you can check if they still need to be fixed. That would be sweet.
Last weeks report is a bit late this week because I wanted to finish the last bits and pieces before reporting back. I moved on locking down the routes to the ones we really need and just merged that. There’s two routes i still need to take care of but then we will have all routes use fixed http verbs and point to a hardcoded controller. No more setting of the controller or the action in the url itself. I also removed routes to actions that did not exist and made controller functions that are not meant to be used as actions protected.
All in all this means we now have a fixed set of ways of accessing the application and can focus on securing those without thinking about funky ways of accessing controller methods that were not meant to be accessed.
Which brings us to the next step… securing the actions against malicious parameters. I’ll start including strong_parameters and look into the authorization code on a per controller basis this week. That will conclude the security overhaul and bugfixes are next.
I had planned to use a the first month for upgrading and the second for bug fixes. The upgrade was basically done by the end of October as planned. I moved on to reducing the attac surface so I can catch new bugs this introduced. So I’m moving on to bugfixes about two weeks later than I had planned. I will see how the bugfix phase goes update the timeline on we.riseup.net/cgdev at the end of the month.
Week 7 – bugfixing time
After a little bit of final cleanup for the strong parameters I now finally moved on to fixing bugs and writing tests that would catch regressions. I’m done with all issues related to the task list page and the page search so far. The page search was not working at all and that was really bothering me but it turned out to be a quick fix.
Week 8 and 9 – stuck (a bit)
Last week I set out to continue the bugfix run and fixed some bugs initially. When writing an integration test for one of them I noticed that test was failing for weird reasons so i started digging deeper and discovered an issue with our permission system. We’ve replaced large parts of the permission implementation in the core rework and it’s still not 100% smooth – especially because there are so many cornercases to keep in mind.
So this week I looked into options to fix that particular issue and just send an email out to elijah to hear his opinion. I’m also recovering from a cold so not as productive as I would like to be.
I hope to be back in full swing next week and resolve the permission issue and resume the bugfixes. Right now I am at 9 / 54 bugs fixed (one actually was a feature request).
Week 10 – moving on with bugs
The permission issue and the randomly failing tests that were holding me back are finally fixed. So I was able to work through the list of bugs one by one. I mostly focussed on page search and the actions one can perform on pages. I also found some new bugs so now the bug count is 14 / 56 fixed.
I will apply the new approach to autocomplete dialogs all over the place and then move on with bugfixes.
three months in
last three weeks
woohaa – time sure runs by quick lately. I was a bit stuck in weeks 11 and 12 and came back to speed in week 13 knowing that last weekend would be the first bigger QA session. You can see the difference on the contributions chart on www.github.com/azul
I stayed focussed on bug fixes most of the time. But i also reworked the people and group directories some. Now you can search your own groups and your friends and peers.
Some friends in Paris met to continue writing a manual for the new version of crabgrass. So the test server had to be updated and I spend the last week and some of the weekend fixing bugs so their documentation went fine.
Naturally the documentation session also revealed a number of new bugs so now the bug count is 25 / 70 fixed. However bugs are also getting smaller and less severe.
I started talking to micah about setting up the new server. We’ll meet again next week.
So while I will continue fixing bugs I will also work on getting things ready for the new server and start looking into customization for we.riseup.net.
I’m about to tag the core rework with 0.6.0-beta. This marks the first crabgrass release in a long time. I spend the day going through the issues list, QAing and closing a lot of issues and organizing the bug tracker. The 0.6.0-beta version has 62 closed issues now – most of them fixed during the last months.
There’s 49 open issues in 0.6.0 (36 bugs + support and tiny feature requests). While micah is preparing the new server i will continue fixing bugs. Then we will start test deploys and customize crabgrass for the new we.riseup.net.
Next week i will send out a call for help with translations to the translation group. Once the new server is up we can probably use a hand with testing.
Hi Betta, thanks for asking.
I just released crabgrass 0.6.0. It has seen quite a bit of QA on the dev server by a number of people. Currently I am working on setting up the new server together with micah.
The process was delayed a bit by a number of factors:
Never the less we should have the new server up and open for broad testing and customization soon. That will bring along a number of fixes to be included in 0.6.1.
I also updated the translation keys. They’ll remain frozen and I uploaded them to transifex. I’ll ask people for translations in the translators group on we.riseup.net. We have some great support from a french group that’s been using crabgrass a lot.
I’ll post some instructions for helping with testing and translations.
I’m a sophomore in college and computer science student interested in doing crabgrass development as much as I can (hopefully 10-15 hours a week). I just finished learning Ruby, and am now beginning to peruse the Rails documentation.
I’m particularly interested in developing new tools for consensus decision making within and between groups, but am happy to do back-end nuts and bolts stuff as well.
Is there anyway I can help?
tartar_socks, great to hear you’re interested. I’ll be very busy this week and the next. However the best way to get started probably is to setup crabgrass on your own machine for development. That’s also super helpful for us in order to verify our documentation. If you run into anything send me a message or open an issue on labs.riseup.net/code/projects/crabgrass
After that i would suggest picking a tiny little thing you would like to work on. That way you can become familiar with the dev process. For getting started i recommend forking crabgrass on github: github.com/riseuplabs/crabgrass-core. Once you have sth. to include – send us a pull request.
One example that comes to my mind would be implementing this feature request: labs.riseup.net/code/issues/8440 – a navigation thing that takes you back from the print view of a page.
server and translation progress
The new server is moving along nicely. micah and me are working hand in hand there. We still need to finish the last symlinks and probably update the database one more time and then it should be ready for testing. I’ve added a bit of customization and it looks like guido will also be working on that.
There’s also progress with translations. The english original is on transifex and there’s a callout for translation. So far portuguese has been translated fully. French is at 97% or so. Please help translate so we can release crabgrass with a fully translated ui for as many languages as possible.
I’ve also taken a look at the current data on we.riseup.net and checked each record for validity. Turns out there were > 70.000 invalid records. Most of them page history entries for pages that have long been deleted. But also duplicate logins and group names and so on. I created scripts to clean them up. I’m not yet fully done but I want to migrate to the new server without invalid records. That way we can keep track of the new ones and fix the issues that lead to invalid records in the first place.
Great news about the work about the data validation and tools to clean things! I had suspected with we.r.n’s long history that this might be the case and I fully agree with your decision to clean things before moving to the new version and server. Another thing we’ve never really done with we.r.n is any sort of mysql tuning, slow query analysis, etc. As you/micah are migrating things if there are opportunities to look at this that would be cool. The new server is faster hardware, which always helps, but it’s always better if the code is more efficient. (on a someone related note, we still see a slowdown on current we.r.n at 1am PST due to backups, strategies for creating backups that don’t slow down the site and also capture data in a consistent way would be good to have)
Review and bug fixes under way
The new server is up and running. It’s been tested by the riseup collective for about a week and i fixed a bunch of issues that were found in that time.
Migration ahead but still unscheduled
The timeline for the final move is yet to be scheduled so that we have the capacities to get it done smoothly and fix things that might come up. I would say it’s a few weeks ahead. I will let you know here once a date has been picked.