Blogs

RSS feed for Code Sprinters blogs available from the Planet Code Sprinters

Andy's Mind: A blunder to learn from

A high profile case of personal data leaking onto Internet because of poor design of a web application just occurred in Poland. It is worth looking at, I think, as it shows clearly one problem with how many web applications are created today.

First a brief recap of the situation: one of major Polish banks - Bank Pekao, part of UniCredit group - started a hiring campaign. As part of that they hired a PR agency that also built them a special website - zainwestujwprzyszlosc.pl (”Invest in your future dot pl”) where candidates could submit resumes and cover letters. And submit they did - hundreds of them. Then one blogger noticed resumes submitted showing in Google results when he searched for his last name and found someone’s resume available for download. He checked the link and to his amazement found that under http://zainwestujwprzyszlosc.pl/files/0 you could find a listing of more than a thousand resumes and download them. Hundreds of files full of personal details and - as it is usually the case - people making fools of themselves in cover letters.

Following his blog post getting immensely popular the story was picked up by mainstream media, someone notified the authorities and the bank. By Monday the whole nicely designed site was replaced with a brief message posted in a hurry saying that the site is closed.

Now this classic story shows many things one could write about - for example total ignorance of media exploited by bank’s PR who said that this situation “was due to a hacker attack” when it was clearly a major design flaw. Or the fact that after the bank officially stated this and added that they closed the page and people’s data is now secure everyone could easily pull those files from Google’s cache. Or that - worse even - coupling of ignorance with arrogance when they accused the blogger and others of “stealing data”.

But I want to concentrate on the root cause of all this mess which I think is not treating web applications as serious software.

I reckon that all this happened because web is still treated as “pages” - not software. So everyone worries about how their sites are designed in terms of graphic beauty, not engineering. Consequently, building web sites is entrusted to “web designers” or “interactive agencies” or - even worse - PR firms (as in this case). Those well meaning companies or people are usually good at whatever their primary business is (graphic design, communication) but lack knowledge of software engineering.

In fact there are many people now who even claim to be “programmers” because they can build a “script” in PHP or even put things together with Ruby on Rails. This obviously doesn’t make them programmers - just like being able to replace oil in your car doesn’t make you an engine designer - but they still claim that to totally ignorant customers who, let me restate that, don’t treat their sites as software. Bank in question has - I’m sure - a very capable IT department, but I bet this website was ordered by someone from marketing or HR who never bothered to consult his own IT colleagues - because it is just “some pages”.

This approach might have been ok 8 years ago when few web sites were rarely anything more than pages + images. But by now it is hard to find such “pages” anymore - the web abounds with true applications. And applications means software - with all the things to consider “interactive agencies” never heard of like data models, database design, scalability and - yes - security!

But failing to recognize web applications for what they really are (software, getting more complex every year) companies hire wrong people or outsource to wrong teams. And then they have problems.

At Code Sprinters we take a totally different view. We don’t claim to do “pages”, we don’t call ourselves “interactive agency”, openly admit we suck at PR and are not good at graphic design. We are a software company specializing in delivering web applications and systems.

And that we take seriously. We hire only true software engineers, that is people who know what data model is, how database design works, what design patters are - and are trained (and mentally capable) to consider consequences given design will have for scalability and security. Work done by such people does cost more - not only because they are more expensive than “web designers” (which is not always true), but also because they put more work into building each web application - and that because they don’t skip building it to be secure, don’t skip building a suite of automated tests around it and do take care to write the code someone else will be able to read & extend in the future.

To be honest I won’t say we never ever have bugs in our applications - but major design flaws like in this case are simply not possible (leaving important files on a publicly accessible directory and feeding it to Google to index them is outrageously bad design). We would die of shame if anything this bad would ever be delivered by us.

The Peako incident shows clearly that is not wise to entrust building software to people who don’t have a clue about it. Pekao has just learned the hard way what the real cost of doing that is. Consider how much will this bank have to pay now all the people who will be smart enough to sue them for damages? Consider damage to their image and reputation. And take into account that under EU privacy protection laws they even face penalties from the government and are under investigation now.

Others should learn from this example. After all this is a huge bank and it will survive it - hiring is not their primary business and most of their clientele is not geeky enough to understand the problem. But think what would a blunder like this do to an HR company?

So, to sum it all up:

• quality - and security - do cost,
• web applications are software,
• software should be built by people who are qualified to do it,
• “interactive agencies” or “web designers” on average don’t have skills and experience to build complex software.

If you save on your web applications or hire wrong people to build it well - you’re in for trouble.

hauru.eu: Failed with Google App Engine

Andy's Mind: The power of Dilbert

It is always with amazement that I find looking at my statistics, that the set of key words that brings most of random visitors to this humble blog is “Dilbert Scrum”. This is so ever since I’ve commented on an episode of Dilbert in which agile is mentioned. Based on it I’ve moved on to discuss Scrum - and probably no one else did exactly that, because if you type “Dilbert Scrum” into Google that post of mine is now number 1. I suspect this post will strengthen that effect.

Interestingly, I’m not sure Dilbert ever referred to Scrum directly but even so people think he must have - so they look for it. Also, this shows that people want to find an image, not a text. Texts are boring, you have to concentrate (which is hard) and think sometimes (which is even harder). Images are much much easier. Which is, probably, why Dilbert brings so many visitors to my page who come for only one thing: the link to the comic strip (BTW: It was wrong, I just fixed it).

Andy's Mind: Thank you, Ireland!

So, the Irish have overwhelmingly rejected the “EU constitution” disguised as “Lisbon Treaty”. Thank you, Ireland! You did what all those who were denied their say by their supposedly democratic governments hoped for.

But this is not over yet for Ireland - the pressure will now mount, because the euro-socialists didn’t expect anyone to say no this time. Eurocracts will probably try to bully Ireland to get back in line with all kinds of threats - from being “left outside” to economic loses. For now they are expressing their anger by saying that “the NO vote is Ireland’s problem“.

Irish prime minister was already summoned to Brussels to explain himself - which shows clearly what the role of national PMs will be in the future EU super-state. And there is already talk of re-doing the referendum - a nice example of how much respect the supposedly democratic leaders have for the people’s vote.

So, dear Irish friends: hold on tight now!

Andy's Mind: Search monopoly one step closer

Not even two weeks have passed since I wrote here about why the ubiquity of Google worries me and now Yahoo is falling into Google’s hands as well. TechCrunch first broke the news yesterday and now is reporting about the conditions of the deal (which are much better for Google) and criticizing Yahoo executives for, basically, giving up on their company and its fight for a place in the Net.

In any case this is clearly a step in the direction of Google becoming the only search engine known worldwide. That would mean a single entity having monopoly over who gets traffic and who doesn’t. Or, in other words, deciding which content is visible and which is not. And this is for sure bending the worldview of its users - if not intentionally then as a result of the SEO experts’ efforts.

This is why I did choose to use Microsoft’s Life Search instead. Joe Ziz commented asking why switch to a search that is not better and is run by a corporate behemoth too.

You see - the point here is not using something technically better but different. If my worldview - as affected by search results - has to be skewed I prefer it to be skewed in a different way. And the problem with search is that with current technology a good search engine requires resources no startup can build. That leaves Microsoft as the only viable competitor - they can match Google’s resources because they can afford it. Probably no one else in the industry can.

And I’m less afraid of Microsoft’s domination of the past than Google’s (near)monopoly of the present. Microsoft just reaped huge profits by selling low quality software, Google is dealing with a much more delicate matter: information.

Now, the big picture behind all this is whether freedom of speech on the Internet will be preserved or not. It is much more likely to survive if there is not too much concentration - that is if the Internet is indeed a neutral pipe connecting small and big alike and putting them on equal footing. If Google (and a few sites like it) dominates and if Net neutrality goes away (which is something all telcos would love to see - selling access to major sites like TV channels is a great idea for Mammon worshipers) then Internet will become as much a censored propaganda channel as TV and radio have became already.

And this is not impossible - the naive thinking of the early 90ies that because the Internet was designed to function after a series of nuke blasts it will be impossible to censor it was proven wrong by China and its Internet Police. In the end it turns out that even if it is technically doable to go around Internet censorship it doesn’t matter if it is too difficult for the majority of the population.

This is a grim vision. It might or it might not become reality. But it is worth knowing how much the shape of the Internet will affect the shape of the society in world’s industrial nations. Google’s influence is not to be underestimated.

Square Wheel: new_record?

Just in case anybody is interested - when a transaction fails (is rolled back), all the objects created in that transaction still respond false to new_record?, even though they are not saved to the db. One more reason to go for identity-mapping ORMs.

Square Wheel: squarewheel

There was a recent post on pylons-discuss about creating a python web framework not based on WSGI, but on higher-level abstraction - WebOb.

Well, I’m doing some research on web applications for my MSc thesis, where I’ll need a framework that is simple and stays off my way - in other words does as little to possible. I’d have used CherryPy, but I found that even it is doing too much. So, with such a suggestion on discussion list I’m trying to build a framework around WebOb.

It’s simple with a layered architecture - each layer accepts an dictionary with a few objects in it (especially webob.Request object) and returns webob.Response. The topmost layer performs application-specific tasks, while lower layers add functionality - much like WSGI middlewares. And because WebOb is a helper for WSGI, I can use any WSGI middleware.

Because of that I’m not wasting time writing things like session support - I just put beaker into the stack and wrote a simple wrapper (7 lines of code).

Perhaps I’ll release the code when it’s usable. Maybe it’ll just serve as a proof of concept - but at least I’ll have some environment for my thesis-related research.

Andy's Mind: Switching to Microsoft Live Search

I’m switching to Microsoft Live Search. I’ve changed the default search engine in my browser and I vow to use it as my primary search engine from now on. And I think my reasons for doing it are worth sharing here.

First, make no mistake: I’m not a Microsoft fan and I never was. I was a Linux evangelist about 12-10 years ago and I’m an avid Apple and Mozilla user now. I think MS’s operating systems suck and always did - they are in fact responsible for entrenching bad software as a standard and degrading people’s expectations about software quality in general. And as a company they are as bad as you can get.

So I’m not doing it because I love Microsoft or because I think Live Search is a better search engine. I’m just fed up with Google - and also a bit concerned: my concern is that relying exclusively on Google’s search results affects my worldview too much.

Google has a de-facto monopoly on web search. Therefore everyone fights (with all kinds of SEO techniques) to be on top of Google’s ranking (because almost no one looks further). The effect is that on the first page of Google’s results you are more likely to find someone actively pushing his mediocre content than people who just have good content and no active “SEO strategy”. Also, Google has been known to censor its search results in the past. No matter what reasons they have and whether I agree with them by doing so they try to shape the worldview of millions - including mine - by removing some sites and through that some points of view. Now, that is not ok.

So, I want to see the world through a different lens - not necessarily a fundamentally better one, but at least a different one. And I’ll be actively encouraging others to do the same thing - not necessarily switch to Live Search, but at least throw away Google-Matrix glasses and look at the web from different angles.

hauru.eu: Blog Finished

Andy's Mind: We have landed on Mars

Media have announced with much ado the landing of the Phoenix Mars probe, one newspaper going so far as to say “we have landed on Mars”. Well, in fact “we” have landed nowhere - this is just an automatic drone that will be digging some ground and doing some experiments on it, after which it will be effectively a webcam on Mars (for some time).

Now, this would have been a great achievement in the early seventies or late sixties. It was a great achievement when the Viking crafts landed on Mars. It is nothing to be proud of that three decades later all we are capable of as a civilization is sending just another robot. This lander may be more sophisticated than the Vikings but it weights roughly half their weight (350 kg vs. 572 kg) - which means we can now haul less mass to Mars surface than 30 years ago! And it will have more sensors etc. but what it will in fact do will be a large repeat of Vikings - dig some soil, analyze it, snap some pictures around, measure the winds.

The sad fact is that no man left low Earth orbit since December 1972 when the last Apollo mission was launched. And even the probes sent are less numerous and smaller than those sent thirty years ago. The space programs of major Earth powers have went stale or were abandoned. NASA facilities in Cape Canaveral smell like an old museum and they in fact are one. Shuttles were a failure, even though no one admits that and no replacement is in sight. The most powerful launch vehicle developed - the Russian Energia rocket - was abandoned too.

I don’t know why it is so, why our progress into space is held back. There may be many reasons for that - ranging from social to all kinds of conspiracy theories. However, in any case I can’t stand media applauding menial landings of small probes as great achievements. This is not fair and real journalism as it lacks historical background that would put those “achievements” in prospective.

Andy's Mind: The right balance

In any team work a certain level of discipline is necessary to achieve organized progress. In creative work, however, too much discipline can hinder both the progress and the quality of the results delivered by the team. It has long been known that software development attracts a certain kind of individuals. Usually above-average intelligence comes with an above-average ego plus (usually) some weird interests and kinks. And a certain obsession with tools combined with the love for the art and craft of building software.

For those reasons managing developers has been jokingly compared to herding cats. The manager has to strike a balance, constantly, between the discipline and freedom. This requires discerning what is crucial and has to be monitored closely from what is incidental and can be left for the team to decide or do.

Agile methods make this much easier because they focus on what is really important and really needed for orderly progress. For example Scrum calls for a Daily Scrum each day and besides that it is left for the team to decide how they will work to achieve the goals they committed to during the Sprint Planning. Also, technical level methods like XP or TDD focus more on how the code being built should look like or how should it be built rather than with what, when etc. The rest is and should be left for the people to decide. Some of this deciding will take place on the team level, some will be up to individual developers.

I believe that this is indeed the right way forward and this is how we work at Code Sprinters. We maintain a very disciplined process coupled with lots of freedom in the choice of tools and a very relaxed, informal atmosphere.

One of the things we don’t force on our developers is the choice of tools. They can use Windows or Linux if they want (I wish we could afford to give everyone a Mac as an option too, but this is not possible yet) - and whatever distribution of Linux they want. They can use a company laptop or their own. They can use an IDE like Eclipse or they can use traditional but powerful tools like VI or Emacs. It is so because for the good quality of the end product - software - the developer has to feel comfortable with the tools he uses. Forcing them to use the same “standard” tools would be as wise as forcing them to wear the same size of shoes. I’ve seen comments that “lack of unity in tools” is a bad thing - I strongly disagree with that. I think diversity in the tools used is a sign of a good, creative team.

Of course, there is a minimal set of tools everyone has to have and use, but all are independent of the OS/platform used. Everyone has to use our Banana Scrum tool to manage their tasks and update on progress, do planning it, register impediments etc. Everyone has to have Skype running on their laptop to stay in touch with others and the clients. Everyone has to use Google’s Calendar, our SVN repositories, project wikis etc.

We also don’t strictly enforce working hours. What is enforced is presence on the Daily Scrums - everyone on the given team has to be there - and there are penalties for being late. Besides that it is just said that being in the office is expected and encouraged, but there are no set hours. So we have people running in just before the Daily Scrum and people who sit in the office from early morning. Surprisingly, even without a card-clock etc. most of the time whole teams sit together in our “war room”. It turns out making it both enjoyable and palpably productive to be here works much better than enforcing presence.

Finally, there are certain standards re. the coding style, the test coverage, the way repository is to be used (what is acceptable as commit and what is not), a procedure for starting a new project etc. They are applied universally and teams working on a given projects may (and frequently do) add on top of that additional standards that for their particular project. Good example is the release procedure which looks different in each project and ranges from just tagging the release in the repository to working with the client’s server(s) to actually put the new release in production.

This approach - enforce strictly few key things, be relaxed on others - has worked extremely well for us. I’m not going to say everyone should do it exactly like we (it might be for example a tad difficult with larger teams from the logistics point of view) but I think the general principle is sound and should be part of good practice in any agile team.

Square Wheel: squarewheel

What kind of trouble-minded teenager would IM a randomly choosen strange person at 00:39 AM to talk about how stressed she is about high school final exams? What happened to good old face-to-face discussions with friends or parents?

I guess there are a few people I could phone late at night if I were in a dire need to talk to somebody - but these are people whom I know for a long time in real life and spent hours talking and listening to.

To all my friends, if you happen to stumble upon this - thank you for all the time, all the words and all the understanding.

PS. 19-yr old girls should really watch out whom they meet on the internet. Lying over the wire is so easy…

PPS. Sorry for a non-technical post.

Sceptical Point of View: Gone sailing

It’s time to unplug for some time and calm down, disconnect not only from the Web, but also from the daily life. In case you need me during the coming week, I’ll be sailing along the Spanish coast — no mobile nor WiFi coverage, sorry. I have several drafts of blog posts waiting patiently to be polished, so be prepared for something new when I return with my mental batteries reloaded.

ShareThis

Sceptical Point of View: Why it’s good to be lazy

Recently I’ve presented my talk on Functional Programming with Python at the RuPy conference in Poznań, the slides (this time in English) are available below. Organizers promise that video recordings from all talks will be published shortly, I will keep you informed when it happens.

ShareThis

Andy's Mind: How easy it was

I’ve just spent (I wouldn’t say wasted) half an hour browsing through a collection of old photographs made available by the Library of Congress on Flickr. Images of a world long gone, so old that even children depicted on those frames are most probably long dead. And a thought came back to me that I had years ago when first really reading up on the history of late 19th century: how easy it was, in a sense, to live one’s life then. The society’s values and roles were very clear then. No doubt as to what was wrong and what was right - everyone was believing in the general set of values based on the ten commandments and moral teachings of Christianity. Not everyone followed them - liars, murderers, thieves, deviants and the like were with us always - but no one questioned them. Most of insanity we see every day on the news now, including all possible perversions, was not thinkable or - at the vert least - was limited to single cases on the fringes of the society. No question then why general decency prevailed - no one posited immorality as a norm.

It very well might be that while we have much developed since then technologically as a culture we - Europeans - have rather declined. The turn of the 19th and 20th century was, I think, the golden age of our culture - even though the first seeds of the catastrophic 20th century were there already. Good that at least we have those images to remind us of times when right meant right and wrong meant wrong.

Rambling about Software: Łukasz

Last weekend I visited Poznań for RuPy 2008 conference along with a good Code Sprinters representation & Piotrek Czajkowski.

The conference was held in UAM’s Collegium Mathematicum, which was pretty hard to get to. I really enjoyed the conference though, thanks to some interesting talks. I really enjoyed Zed Shaw’s talk on getting statistical results while measuring web app performance. It was interesting & funny (in spite all of Zed’s Ruby & Rails negativity :-)). I also really liked Micheal Foord’s talk on IronPython & Silverlight 2 (not to mention his lightning talk about Resolver One) since I missed it on SFI conference held in Cracow. Adam’s talk on functional programming in Python also went really good, and gathered many people in the auditorium. Unfortunately I didn’t see the PyPy talks which were said to be excellent. Overall the talks were a load of fun. There were some glitches though. First of all, in my opinion speaking in English shouldn’t be mandatory for speakers, as majority of the talks presented by Polish speakers was terrible to listen too. Because of the language barrier speakers couldn’t speak up their minds. So I think it would be best to let the speakers choose the language they speak in. There were some exceptions though for example the “A need for REST” talk by Łukasz Piestrzeniewicz. The second thing I didn’t like, was the lack of transportation from the place where the conference was held to the city centre (we had to use cab’s all the time) as the information on RuPy official site were a bit misleading.

Photos are available in the Code Sprinters Flickr Pool.

Andy's Mind: Something is afoot

Bad things are happening on the markets, as everybody knows. One can read (and view talks) about all kinds or problems in the US economy, including imminent predicted collapse of the US dollar and some grim political theories behind it.

Putting the political dimension aside those predictions seem to be based on solid economical theories. In fact, opponents of the Keynesian economy have predicted exactly this kind of thing to happen for years. Just no one believed it will really happen, because said opponents (Hayek, Friedman, von Mises) have been saying that for such a long time people got used to it. But maybe the mechanisms involved required a long time to produce results we are about to see.

And here comes the startling analogy that occurred to me today: same thing happened with “real communism” in the Soviet Union - no one really believed it can fall apart within our lifetime, especially leading sovietologists. May it be so that Western-style socialism will collapse like communism - just a few decades later, because it is - after all - more efficient then communism?

Andy's Mind: Conferences

I just realized today why I like going to conferences so much. It allows me to think.

This is so because usually getting to a conference involves travel and traveling has always induced high quality thinking in my brain. I don’t know why, but I find the whole experience of moving very inspiring. No matter if this is by plane, train or car my mental gears spin faster. This is a creative time also because this is usually a when I can’t use my computer, I’m not answering phone calls and generally I have less distractions.

In any case I’m finding out, that without a few hours of travel every month I’m deprived of some of my quality thinking time I used to have.

But the conference experience doesn’t end there - it also gives me time to actually listen to people talking about subjects of interest with full concentration. And ask them questions. Not possible with some interesting lectures available on Google Video and other sites. First, because it is surprisingly hard to find a free hour within a day to listen to them. Second, because there is no interaction.

Finally, on a conference I get usually a few hours of very good work on my computer in the evening. Again, precisely for the reason outlined above - less distractions.

Seems like I have to cut down distractions to move faster, the problem is that Internet is just one huge distraction. And with a laptop and Wi-Fi it is almost everywhere now.

hauru.eu: Shiny new blog

Sceptical Point of View: Tracing and profiling Ruby code

Every child knows that premature optimization is the root of all evil, and even when optimization is necessary, we should concentrate on the bottlenecks. This is where profiling becomes crucial. Ruby includes a simple profiler in the standard library, so to generate a report of program execution you just have to invoke it with ruby -r profile or add require "profile" to the code. In fact the whole profiler is implemented in only 59 lines of Ruby and relies on set_trace_func method to register a callback tracing certain events during program execution (method calls and returns in this case). This tool should suffice for simple profiling, but if you need something faster and more powerful you should rather try ruby-prof.

The powerful introspection features of dynamic languages make tricks like this not only possible, but also straightforward. This gives me an idea that the same approach could be used to implement an aspect-oriented library for Ruby — but I’m almost sure somebody has already tried this.

ShareThis

Square Wheel: squarewheel

The company I work for - Code Sprinters - released three rails plugins I wrote under the MIT licence.

OutputStream and FlashMessages

Those two plugins allow safe mixing of unescaped text and html content. Strings are marked as being in either format and then are escaped appropriately in the view layer. This allows the developer to embed eg. links to other pages in flash messages without worrying about having some other piece of data unescaped.

Strings not marked explicitly as safe for html are escaped with default rails h() helper.

Although currently the plugin only knows how to escape text output to HTML, it can be easily extended to support other formats.

Expose

This plugin is inspired by CherryPy (http://www.cherrypy.org) - a simple yet powerful HTTP application server (and microframework).

This plugin changes the default policy of exporting all public methods of controllers via HTTP protocol to only exporting explicitly stated methods - and only to specified HTTP verbs. This helps prevent mistakingly exposing methods that should only be filters - or exposing via GET methods that should only accept POSTs.

Also, this is generally good practice to deny access by default - and allow access only when explicitly stated.

For actual downloads, go to the plugin page.

Andy's Mind: Some Scrum bits

To make my life easier with introducing people to Scrum I did prepare a short presentation. You can view it here.

And, BTW, I’m happy to report here that a new version of our Scrum tool - Banana Scrum - was released last Friday. Development is still going on, even though I can only put people who are not engaged in other projects on that work.

Right now I decide we’ll do a sprint concentrating on “cleanup” - mostly features or development that aims at improving overall usability and compatibility with different browsers. Even though IE 6&7 + Firefox + Safari work flawlessly we had some problems with Opera.

Sceptical Point of View: Thinner Ruby deployment

In the post on benchmarking HTTP performance I mentioned that according to my tests a cluster of Mongrels performs about 10-20% worse than the same number of FastCGI processes behind a reverse proxy. Recently I tried Thin, a new web server based on Mongrel libraries, and it turns out to be a solution that gets the best of both worlds. It is very easy to setup and manage (even easier than Mongrel), extremely flexible (mostly thanks to Rack) and really fast. It matches FastCGI in performance, without all the quirks, and can communicate through UNIX sockets too. I have to admit that I was impressed by the simple and clean design of Thin (which is based on existing quality modules). The only disadvantage is that it isn’t very mature yet — but in the near future Thin might become the best server for deploying Ruby web applications.

ShareThis

Sceptical Point of View: Benchmarking HTTP performance

Deployment of Rails application is a subject that tends to raise some hot discussions, leading to many misunderstandings. That’s why I decided to try different deployment strategies and check for myself how they perform.

To make any reasonable comparisons it is crucial to measure performance of different configurations. The most common metric is the number of requests processed per second (RPS). This metric (and many others) can be measured by HTTP benchmarking tools like ab and httperf.

The first tool, ab, comes bundled with Apache and is very easy to use, so it is a good option to start with. You can provide a total number of requests to perform (-n) and a number of concurrent requests (-c). If you like you can also give maximum time to wait for a response (-t), as real users won’t wait for a page to load more than just a few seconds.

For example to issue 1000 requests with concurrency of 100 you might run (remember about a trailing slash in the URL, it is necessary)

% ab -n 1000 -c 100 http://www.example.com/

httperf is a slightly more complex tool with more features. The most important is a possibility to issue multiple request per connection (--num-calls command line option) and support for replaying sessions that imitate real use cases. The tool is also believed to be more robust and give more reliable results. The basic use might look like

% httperf --server www.example.com --num-conn 1000 \
          --num-call 10 --rate 10

This will issue 1000 connections with a rate of ten connections per second (and no more), passing ten requests through each connection before it is closed. So the total number of requests will be 10000. Be sure to remember the distinction between connections and requests, otherwise this can lead to confusion when interpreting results. Another tricky part is the actual meaning of the rate command line option. Rate is not a number of simultaneous connections at a given time (like concurrency in ab), but rather a number of new connections made per second. This means your RPS cannot exceed rate given multiplied by number of requests per connection. So httperf has to be ran multiple times with increasing rate to find the saturation point of the server.^[1]

When benchmarking HTTP performance don’t just accept the first results blindly. Think for a minute what you are actually measuring. Check the status of the replies — if most of requests fail it is a sign that something is wrong, if you are getting 3xx redirects probably you should rather test the URL the redirects point to. If many requests have timed out the concurrency you requested might be too high.

Never perform such tests from your desktop machine far away from the server. In the perfect world you should run the benchmark from an independent machine in the same network segment as the server, and make sure the network is not saturated during the test. If you have to run the tests on local machine, remember that the load caused by the test itself can skew the results (note that from my experience ab causes considerably smaller load than httperf).

Finally consider where the URL you provided points to. If this is a static page or file, you can easily achieve thousands of RPS, as the performance is bounded mostly by disk operations. On the other hand if you measure a dynamic page running multiple SQL queries you might get very low results, as the database will be the bottleneck. Many recommend to benchmark a simple dynamic “hello world” application that doesn’t communicate with the database. But if you want to measure performance of the application, not a web server, you can measure and compare different URLs.

In my benchmarks I found out that three Mongrel instances load-balanced by Pound are about 10-20% slower than three static^[2] FastCGI processed running from a vanilla Apache installation. It is probably due to the fact that the front-end server communicates with Mongrels through TCP connections, which are considerably slower than UNIX sockets used by FastCGI. On the other hand this architecture makes scaling Mongrels easier, because one load balancer can proxy requests to multiple machines.

It looks like there are reasonable arguments for both strategies, and I find it a bit surprising that the whole Rails community is voting against FastCGI, calling it a legacy solution. It’s true that FastCGI can be tricky to setup correctly — but at the end of the day it performs better, and there are other benchmarks showing similar results (as shown on this chart).

—

^[1] More information on good HTTP benchmarking practices and the usage of httperf can be found in the Linux HTTP Benchmarking HOWTO.

^[2] Never use dynamic FastCGI processes for production purposes. Dynamic processes are killed when unused and due to timing issues users can get internal server errors. Moreover every request assigned to a fresh process is delayed, as it has to wait for the new process to boot.

ShareThis

Andy's Mind: New Banana and some mussings on tools

We have released a new version of our on-line hosted Scrum tool last week - the Banana Scrum. The most important addition is, of course, the automatic registration form, but we also improved the way in which the user interface works. We start to get ideas from our users, who generally like the tool but will undoubtedly help us make it better.

Which is good, since I think there is a definite need for a simple, on-line tool to assist agile teams in their work. Which leads me to another topic - resistance to any such tools. When someone asked about a tool for Scrum on the Yahoo Scrum group there was a bunch of answers advising not to use any tools - use a wall with velcro attached index cards or, at the very least, Excel.

Agile community in general and Scrum community in particular seems to be very attached to “good old” physical artifacts, like index cards, hand-drawn burndowns, hand sorted backlogs etc. I can respect that but I’m a completely different person - I’m a “paperless guy”.

The only thing I still prefer on paper is books. Hand drawn brundowns can be nice if everyone sits in the same room from 9 to 5, but we have a much more relaxed atmosphere - everyone has to be in for the Daily Scrum but otherwise people can work from where they want when they want. Whiteboard is great for sketching things or making notes during a meeting but I don’t think it is good to make it a permanent repository for anything. I think if we work with computers and systems and web apps we should use them. How credible we are telling other people our applications can save their business if we stick to index cards and velcro?

So, I don’t like it when some agile gurus look down us, paperless guys, when we confess we use software tools to manage our projects rather than walls, cards and boards. I don’t think it is a good idea to be too dogmatic about tools, I agree with that, but it also applies to the old paraphernalia of the paper age.

Sceptical Point of View: Subversion Scripts for Finder

Subversion is one of the basic tools in my daily work. I know, distributed version control is more en vogue those days, but I would argue that for personal use and small teams Subversion is still a reasonable choice^[1] — it is very popular, flexible and there are many additional tools available.

About a year ago, when I started to play with my first Mac, I was looking for Subversion tools that can integrate smoothly with Finder — the standard Mac OS X file manager. To my surprise I couldn’t find anything useful, only SCPlugin which didn’t work at that time, and as far as I know is still somehow buggy. So I decided to write my own set of scripts, as an excuse to play with AppleScript — a funny high-level scripting language that can speak with Mac applications (including Finder) over simple interfaces called dictionaries, not surprisingly consisting of nouns (objects) and verbs (methods). This custom set of scripts had been so useful to me (especially when invoked from Quicksilver) that I decided to release it publicly, starting a small open source project.

Recently I’ve released version 1.2 of the scripts, including support for Copy, Move and Checkout operations, with improved Leopard and MacPorts support. The release was also a good excuse to make some adjustments to the project page and publish a screencast showing how to use the scripts. Judging from the download statistics and the feedback I get, people find the project useful, so if you are a Mac user consider giving it a try!

—

^[1] But not for projects with many independent branches of development — branching and merging sucks in Subversion (it will be improved in Subversion 1.5, which is now in beta). Linus’ critical opinion on Subversion is well known, and I don’t claim it is the best choice for large open source projects (though many such projects use it).

ShareThis

Andy's Mind: The choice

On LinkedIn someone asked this question:

How do you deal with the project requesters that are asking you for a project estimate before you get all your questions answered? Would you just ignore them and loose the project or go ahead and earn a client?

This is indeed a problem all of us in the software development business face. The question is indeed what to do in such a situation and it boils down to following three choices a service provider has:

• rip the client off - add all the uncertainties, add the industry standard 25% and produce a bid - then do it as fast and cheap as possible and profit - that’s probably what _most_ companies do,
• risk your money - to win the bid do all the above but not add 25% but rather subtract it, so that you’re cheapest thus winning the bid, then kick the project as fast out of the door as you can compromising on everything your client being ignorant in technology won’t be able to tell - namely quality (do spaghetti code, do it ugly, test only positive paths and forget unit testing etc.),
• tell the truth. That is - the client he won’t get anything solid with what he has, just lies, assumptions etc. However, for whatever amount he can spend he can get the most important features he needs with solid quality. If his site would be a success he can add nice-haves later on. Ask the client to make a list of them, discuss them, tell him that in one short iteration one or two will be up & running. Be honest and cooperative. It’s not a guarantee of success, but well… this is what this option is all about.

Everyone who is approached by such a client makes this choice. The problem is that typically such clients don’t like the third option, so they fall for the people choosing the other two. The results we all known - poorly designed sites, unmaintainable code or - worst - lost time and money.

What we follow is of course the third option.

Andy's Mind: Banana Scrum

Another project that we have been working on is now available for preview. This time it is a simple tool for managing the Scrum process called “Banana Scrum”.

You can log in to the preview installation by following this link and using the name “admin” and password “test”. You can play with the application as much as you want - it works on a copy of the test database that is regularly refreshed, so you can move and delete and edit whatever you like.

This project was born out of our frustration with ScrumWorks. It was the application we have used for the most of last year, because it was the only free and decent Scrum software around when I went looking for it in January 2007. However, it had many serious limitations, exorbitant pricing for the “Pro” version which added basically access rights for user (not needed for small teams IMHO) and was done as a Java app - a design decision I couldn’t understand, since there is absolutely nothing in there that could not have been done with an interactive, AJAX web app.

That is exactly what we did - our little Banana is done fully in Ruby on Rails, works great with most browsers and supports interactive features like in-place editing, drag and drop or a nice, interactive burndown chart. As of now it is pretty useful and - as it supports multiple projects - we use it for our own work.

We hope to add more features soon and possibly some day make it available to the world. If you’d be interested in trying it already - drop me a line.

As for the name… well, there was a competition in the team and Tomek Paczkowski won it (and a bag of bananas) with this proposal.

Sceptical Point of View: Trivial accessors and uniform access

Some tend to think that Java is a synonym of object orientation done right, some even don’t know other alternatives. But it was always unnatural to me that most of Java classes start their existence with plenty of boilerplate code like this^[1]

public class Money {
    private double amount;
 
    public double getAmount() {
        return this.amount;
    }
 
    public void setAmount(double amount) {
        this.amount = amount;
    }
}

This is a lot of code to write just to define one single property, a code that is mostly meaningless. But in Java you have to introduce getters and setters from the very beginning, or it will bite you back in the future. It clearly contradicts with the DRY principle and a preference for evolutionary design, which discourages writing code that is useless right now, but may (or may not) be needed in the future. Things get even worse when such code is created automatically by some code generation tool.

In theory your methods should always have some meaningful behaviour, and your should avoid trivial accessors in the public interface. This is a good rule of a thumb, and when it’s broken, this is often a symptom of some wrong design decisions. Though in many practical situations you simply need trivial accessors without any behaviour, for example when mapping relational databases to objects^[2].

The whole problem boils down to the fact that in Java you can’t apply the Uniform Access Principle, which states that users of a class shouldn’t care whether a given service is implemented through storage (property) or computation (method). But the syntax for accessing a property and calling a method in Java is completely different, and you can’t start with a simple public property and change it into a method later when it becomes necessary, keeping the public interface intact. So you are told not to use public properties at all and always define trivial accessors just in case.

I would like to contrast this approach with two dynamic languages, Python and Ruby, each presenting a different point of view on the problem we discuss.

In Ruby — which has been inspired by Smalltalk — properties (instance variables) are always private, and the only way to interact with an object is by sending messages to it. This is similar to a method call, but the meaning is slightly different, and there are certain conventions to make the syntax nicer. You can’t access instance variables outside the class, so the following code

cash = Money.new
cash.amount = 10
puts cash.amount

is actually the same as sending messages amount= and amount to the instance, which can be written explicitly as

cash = Money.new
cash.amount=(10)
puts cash.amount()

This means that Ruby has an uniform syntax for attribute access, but you still have to write message handling methods inside the class. This is where attr_accessor comes in handy (along with its siblings attr_reader and attr_writer), avoiding duplication and making the code more terse. The following piece of code

class Money
  attr_accessor :amount
end

has the same effect as

class Money
  def amount
    @amount
  end
 
  def amount=(value)
    @amount = value
  end
end

When the class evolves and we would like to make accessors more complex (for example implement lazy load or caching) we can replace attr_accessor with real methods, keeping external interface intact.

Python takes a different approach than the message passing metaphor. It publicly exposes all attributes of an instance as slots you can access freely. Inside such slot can be any object (in Pythonic sense of the word), including standard objects (integers, tuples, etc.) and methods. The client simply fetches object from the slot and either invokes it (if it is a callable) or uses its value directly — so the access is not uniform.

To maintain an illusion of uniform access when refactoring a property into a method you can use the property() function, passing new getters and setters as arguments. This means you can start with a class as simple as

class Money(object):
    def __init__(self):
        self.amount = 0

Later, when you need some more complex accessors, you can refactor the class with property(), maintaining the same external interface

class Money(object):
    def _get_amount(self):
        # Getter code here
        return self.amount
 
    def _set_amount(self, value):
        # Setter code here
        self.amount = value
 
    amount = property(_get_amount, _set_amount)

As you see the code is not as clear as with Ruby, and there are some other problems with this approach, but it is possible to maintain uniform access in Python.

I believe obeying the Uniform Access Principle is the right way of solving the accessor problem, and both Ruby and Python handle this quite well. If you see trivial getProperty() and setProperty() methods in Python or Ruby code, stay aware. This probably means the code has been written by a programmer who is unable to change his mindset.

—

^[1] To convince you this is not a fake example I did a quick search on the Web, finding this piece of code.

^[2] Martin Fowler on page 155 of his PEAA book gives example of a class to map a simple person table. He writes it starts with data fields and accessors and then gives an example of over twenty lines of boilerplate accessor code.

PS. Thanks to Tomek for reviewing the first draft of this article.

ShareThis

Sceptical Point of View: Sharing knowledge inside a team

What I like about being a programmer is that you have to constantly learn new things — either new languages, tools and frameworks that make your job easier (and more fun), or interesting theoretical concepts that stretch your mind, a kind of mental yoga. Being a math graduate I can tell that even if this knowledge is not instantly useful, it will probably pay off in the future.

There are different ways for programmers to gather knowledge, most have something to do with reading. But, as skilled craftsmen have known for centuries, the best way to learn is from a person who is willing to share his real-life experiences, tricks and habits. For example skimming through a cryptic Vim reference sheet or reading even the best tutorial is so much different than seeing the actual usage patterns of a skilled user.

That’s why every week at Code Sprinters we organize Tech Talks, meetings during which one person speaks about about a topic of his (or her) interest. Those are not official presentations with slides, but rather discussions around the whiteboard, sometimes turning into hands-on workshop. The subject doesn’t have to be purely technical — a few weeks ago I spoke about Getting Things Done, a way to organize myself I use and find very useful. We don’t claim that we are experts on a given subject, but certainly each of us knows something that might be interesting to others.

I think holding such meetings is a great (and quite easy) way to spread the knowledge inside a team or a small company, and it’s also fun!

ShareThis

Andy's Mind: A search that defines

Yesterday we have decided to publicly show “Sprinters Search” - a research project we did at Code Sprinters in the recent month. It is a meta-search engine that attempts to provide the user with a short definition of the search term followed by unaltered results of a regular, traditional web search.

This project is a result of an idea that came to me last year. It occurred to me that today people are frequently using search engines not to find a well-sorted list of pages on something they do know, but rather they want to learn what a word or a name or a phrase they picked up somewhere (in a conversation, on the Internet, on TV etc.) means, what that is. Said well sorted list of pages helps only partially in getting the definition people are after. An attempt to provide a very succinct and correct description as a result immediately would be much better. Such a result could be followed with a traditional page list if the engine’s guess was wrong or the user wants to research further.

I thought that with lots of structured information now available on the Internet building something like that should be quite possible. After all, when current search engines were invented they were designed to parse just general web pages with no structure to them at all. Now we have all kinds of data and content bases that provide good quality information in a structured way. It seemed quite possible, so we gave it a try.

And here it is our fully working attempt at demonstrating that it is indeed possible. The aim was to provide - in most cases - a correct definition of the search term upfront, on the top of the page, possibly with an image, so that the user doesn’t have to scroll down or click through to get the definition he needs. I’d say for an early beta our meta-engine does pretty well - thanks in part to simple yet ingenious algorithms applied by Pawel Stradomski who designed that part of the code.

Of course, it is still just a research project. To make it robust and scalable resources we don’t have are be needed - much more computing power and fast storage + more time to fine-tune the algorithms. So for now we’ll leave it as it is - a good demonstration of our capabilities.

Andy's Mind: Why recruiters dont deliver

I know someone who recently applied for a job in a recruiting agency and learned quite a bit about their working methods. As it turns out a recruiter at that agency has to handle in parallel 14-16 cases - positions that they have to fill for the agency’s clients - and there is no industry specialization. So, one might have to find three accountants, two C# developers, one scrum masters and three floor cleaners - and five other people from other, completely unrelated fields. With this number of cases to handle and lack of focus on a given industry the recruiters they have can’t be good, even if they wanted to. It becomes a number game, hence retorting to database handling and everything really that can make the process faster. Hence I was not surprised when I’ve learned that on top of all that the recruiters at that agency were required to strictly follow company’s procedures.

And this is not a small agency, they employ some 60 people and have been recently acquired by an investment fund (who, btw, requires them to be more profitable - read increase the load of cases on recruiters).

This corroborates what I was long suspecting and explains why no recruitment agency I’ve worked with was able to deliver really good programmers, IT managers, routing specialists and the like. First - the best rarely ever look for a job or read ads in newspapers. You have to go after them and fish them out of the universities, this or that language users group etc. And you have to know that a typical geek is a completely different type of fellow than a sleek marketing graduate looking for a job. I bet fishing out good accounting & finance talent is equally hard and in this day and age requires much more effort than just shuffling CVs around as they flow in.

Somehow this - and most other recruitment agencies - don’t get it. Why? Well, because the truth is most jobs - especially many corporate jobs - don’t require exceptional talent and outstanding skills. Filling the seats with half-decent people is a success already so anyone who can deliver them in numbers has a business. That’s why I expect also this agency to grow along, congratulating themselves they do the right thing - and still missing the point completely.

Sceptical Point of View: Functional programming in Python

Below you can find the slides from my talk on functional programming in Python, which I presented a few days ago at the first meeting of Pythonistas in Kraków (the slides are in Polish only, sorry). Feel free to leave your comments!

Update 2008-04-24: Slides are also available in English.

ShareThis

Sceptical Point of View: Agile goes underground

About a year ago I had an idea to run an informal Open Space event here in Kraków, to gather a group of people interested in Agile methods of software development. At that time there was little or no interest, so my motivation slowly decreased. But a few weeks ago I met with Kuba, who had a very similar idea some time ago, and we decided to take a try. After a few days of quick (and quite dirty) preparations we are proud to invite you to the first iteration of Agile Underground, an event which will take place on Feb 28th here in Kraków.

ShareThis

Sceptical Point of View: Every journey begins with a first step

I’ve just moved to a new apartment, in a monumental building just a short distance away from the Old Town. Changes like this always give me a hit of fresh energy, so I thought it would be a perfect moment to take a second approach to the blog.

I decided to make some changes before the second iteration (”inspect and adapt”, like with Scrum). First of all, I will write only in English — it has become the universal language and is almost assumed in my profession. Societies on the Web concentrate around shared interests, not spatial location, and I don’t want to limit my potential audience without a good reason.

Secondly, I decided not to use Blogger anymore. It is a good solution to setup quick and dirty blog, but in the long run I wasn’t comfortable with a tool that is not pleasant to use, and imposes certain constraints on its users. I like to be in power, so after looking into different alternatives I’ve chosen WordPress, which convinced me with its maturity, extensibility, clean administrative interface, and overall attention to details (though I know the source code has its issues). Special thanks to Scott for blog.txt, a simple theme that I used.

I intend to keep this site alive, and I already have a long backlog of ideas. My plan is to post at least once a week, maybe even more frequently during the first month, to fill the site with content. So stay tuned!

ShareThis

Square Wheel: squarewheel

Everybody using Rails probably knows that ActiveRecord provides many methods for declaring validation of records - the validate method as well as validates_xxx methods which generate code for most common validation tasks. There is a problem though…

The validation can be easily skipped. Just use the update_attribute method and you can set any values without validation. Kinda scary if you’re used to encapsulating consistency checks in model - some other coder writing controllers for your model can just break your rules…

Just one more reason to ignore DHH’s words on moving validation to application layer and putting it back in database (where it should always be, even if you have the rules in your application layer too; skip consistency checks provided by the RDBMS only if they are too complicated to be effectively expressed with SQL or you’re using RDBMS with limited capabilites in this regard, like MySQL or SQLite)

Rambling about Software: Łukasz

New site content is coming soon.

Square Wheel: squarewheel

We’re working on a project using Ruby on Rails 2.0.2 currently.

As you probably know, RoR is not thread-safe. But even if you are not using ActiveRecord, ActionPack nor any other Rails part you still might run into some problems if you thread.

We use the probably most common setup - small loop launching threads to query some web services in parallel. This seemed safe, as it would not use any of RoR classes/modules.

Now here’s the gotcha - automatic constant resolver (part of ActiveSupport) that loads classes from properly named files. It’s a part of RoR too…. and has threading problems, just like the rest of RoR. If a constant is not known before the threads launched, every thread would try to load that constant on its own; only the first thread would succeeded, all the other would raise “constant X is not unknown” exception.

Currently we worked this around by requiring files defining all dependencies in the file that launches the thread before going parallel. It’s ugly and not easily maintainable, as threads use different functions, each with it’s own dependencies - but it works, for now.

Perhaps it’s time to port that project to python - it has proper thread support (not to mention the fact that I wouldn’t have to deal with RoR worst part, that is ActiveRecord - SqlAlchemy is way better)

Andy's Mind: New year - new challenge

The last year was quite successful - I’ve managed to change my development team into Code Sprinters and we did some interesting projects. We also have some ideas of our own that we work on - and here comes the challenge: how to sell a really great idea to a larger company without being ripped-off.

That’s what I’m thinking on when I’m not talking to clients and preparing a series of webcasts about our work methods.

Andy's Mind: Merry Christmas!

Andy's Mind: Not mainstream enough?

Some folks started an inter-blog discussion about why enterprise software is mostly bloated crap. The theories put forward range from a meaningless observation that enterprise software “is not sexy“, through really funny stuff like George Ou lamenting that they are typically not done with “lightweight” languages like C++ but instead with “programming shortcuts like Java” to gloomy statements that it has to be that way since it’s enterprise software - so what else could we expect? Nicholas Carr postis that solution might be in doing enterprise software as web applications (exactly what we offer), which sets him apart from the others. But even that voice included some key elements are missing from the corporate IT experts prospective: who produces enterprise software - and how.

I think you can’t understand why enterprise software mostly sucks by not taking into account that it is the area where most projects are done with ineffective waterfall methods under heavy bureaucracy of traditional project management methods inside corporations. SAP implementation, for example, is mostly expensive consultants doing all kinds of paperwork and producing tons of specifications before any programmer does anything.

All that further amplifies the already bad effects of developers working in a corporate environment (like at SAP or Microsoft), usually not very dynamic, with limited upward mobility and very limited influence a single employee can have on a product. Adding long release times, extensive paperwork and removing any link between developers and clients has to kill any excitement and emotion developers have for their work. Without it what they produce must be crap.

Not that clients or producers of these overpriced beasts care. Clients got educated through their lifetime that enterprise software has to be hard to use, buggy and expensive. Producers are more than happy to earn huge profits from cheaply made products. Until the clients won’t start to ask for more not much will change here.

During the Agile Development Practices conference last week Mary Poppendieck announced that agile is now mainstream. Well, apparently it is not mainstream enough yet for the IT experts I mentioned above - or for the enterprise systems buyers. But I believe we’ll get there eventually.

Andy's Mind: Day four: agile wolf in a sheeps skin

The last day of the Agile Development Practices kicked off with a keynote by Jutta Eckstein on scaling agile development in really big organizations (Deutsche Telekom I suspect). I have to say it wasn’t all that interesting - the delivery was not spirited and the way they did in Germany smelled of trying to make good all command&control chains sound agile. I used the time to jot down some notes before the Open Space session that I initiated.

It turned out few people were there, but there was another session started by Chris Spagnuolo on agile contracting - which was pretty much the same thing - so we merged the two and had a very good discussion with his other colleague, Zvonimir Durcevic and Rachel Weston of Rally Software. It turned out each of us were in a bit different position even though we all do outsourced agile. Chris’ clients are mostly government agencies, so sadly they are stuck with fixed bids - their clients even if sold on agile just can’t go around the straitjacket of regulations. Rally in turn does what I called “being agile wolf in sheep’s skin” - they first play their clients by the book, offering a bid etc. and then once they have them signed off try to sway them to agile modifying (amending) contract later on. That sounds like an interesting way of getting in, but there is always the risk they will have go against the fixed contract anyway - with known bad results (money lost, quality degraded etc.). Still, again, with some types of clients that the only thing you can do.

Next, I got immersed in a discussion with Richard Sharpe of Enerjy and another guy doing testing, measurement and qa analysis. Guys at Enerjy did some really cool research they are about to go public with, which will basically allow to programmatically find areas in software likely to have bugs based on certain metrics. Now, I’m a metric-skeptic but I think if they really have an algorithm that can do that it’s a big thing. In the course of that discussion I learned they have quite an interesting blog there - and a video blog too, for which Richard was making recordings there. I’d have to check it out.

We also discussed some pretty crazy stuff about how Europe is going more and more socialist - and in stupid ways too. It turns out, for example, that in Germany one is not allowed to track individual performance of employees, for example software developers. When they tried to sell some software that tracks different quality metrics in software they were not allowed to enable the part that does this per developer. Complete utter idiocy.

Because of that interesting chat I kind of missed one session - so I went to next one, which was about thing called Fit. Fit is of course an acronym (it’s US after all) for Framework for Integrated Tests and the idea behind it is to follow business rules and check whether software we build meets them. So it’s not unit test - and it’s not functional test, but rather something in between. James Shore presented all that in a rather dry, academic style which I think scared some of the audience away. I had hard time trying to stay focused as the room was so cold in the end my primary concern was staying warm. But anyway, that was interesting stuff - I’d have to point it to my team, even though I think right now we don’t do anything that we could use it for. And the tool itself is working mainly with .NET - or in other words crap we stay away from.

Last strong accord of the conference was Andy Hunt’s keynote, which he himself called “a dessert”. It was again a broad reflection on research into generations, their characteristics and the fact, there is certain repeatability to general trends in how generations view world. The catch is that this repeatability goes on every four generations more or less - so the one entering the world has no chance to meet their grand-grand-parents and see how much in common they have. That would explain a lot of the cycles that are clearly visible in history, though it obviously opens another question: what causes this pattern to occur.

In any case Andy’s point was that surprisingly - or not so - the what is considered a good method for developing software reflects very well the outlook on life and everything typical of the generation dominant in a given time. Which means that at some point even waterfall might make sense again. On the other hand, Andy was wise enough to point out that this cozy picture of a generations wheel turning predictably through history can and probably will be disrupted by a black swan event - or a huge event no one can possibly foresee.

Well, we’ll see how it all goes. For now let’s say that this conference was a great learning and networking opportunity. I’ll do my best to return next year.

Andy's Mind: Day three: the mainstream and the pragmatic mind

It’s day three and the proper conference opened after two days of tutorials with a keynote by Mary Poppendieck. She officially recognized that agile has become mainstream now, referencing a book about how ideas cross a chasm separating them from mainstream. But instead of cheering and encouraging the audience to indulge in feeling good about it she warned us about possible ways in which we could fail with agile. What I did carry out from that speech was that we shouldn’t become too attached to “agile methodologies”. We shouldn’t follow what others are doing but rather constantly inspect what we do and adapt to our own circumstances. Mary’s keynote was full of references to books and cases showing the progress continues - some even see iterations as a transitional stage to a state of perpetual release, constant improvement.

Mary’s overall message was important to me. I represent a small team working from a little country at the outskirts of Europe - and hence I have a tendency to look up to those great organizations at the bleeding edge of the new. But we can be great too - and in fact, as I already observed, my team is well ahead of most of the people I spoke to during breaks. And that there are teams that are even better is great too - at least we know we have to keep on improving. Our clients can only benefit from that.

Afterwards I attended a session led by James Waletzky from Microsoft, which was… well, boring. He had this great idea of using two of his friends as actors playing a pair of developers - a waterfall-ish one and an agile one - talking about emerging vs. upfront design while he commented on it. It was maybe a nice idea, but the actors were not in sync with his presentation and what they discussed was so basic I got bored quickly. I didn’t want to disrupt others by leaving so I opened my laptop and responded to some e-mails thanks to Wi-fi reaching into that particular room.

Luckily, that was the only disappointment of the day. The afternoon sessions by Andy Hunt about the workings of the mind and improving how we learn and handle things in our heads were quite interesting. Some practices - like GTD or mind mapping or the left-brain vs. right-brain thinking - I was already familiar with. Others were new or I’ve just read about them but didn’t use them.

I always like to listen to people who have broad knowledge, research many things and can talk about them in an interesting way. I feel I have lots in common with such people, whom I call “searchers”. It is great that there are others who don’t just use the mind, but try to understand what it is and how it works.

During the break there was a small discussion and it turned out some insights I gained from my meditation practice were quite interesting for the others. I was surprised that Andy - being interested in the workings of the mind - started some kind of meditation practice only recently. We also exchanged some ideas - I’m now researching what EMDR is, Andy will - I’m sure - go through the website of the Global Conscience Project. As it is frequently the case - questions and discussions during breaks can be as valuable as the sessions themselves.

The day closed with another keynote, this time by Mike Cohn. His presentation - as usual powerfully delivered - concentrated on reservations and fears people have against switching to agile. Not much new material for me, but again, hearing it in an organized fashion was quite refreshing.

Tomorrow I’ll be leading an open space session on agile in outsourcing. I don’t count on many people being interested in such a discussion but I thought it’s worth a try especially as the subject is really underrepresented in the conference (and most of agile literature).

Andy's Mind: Day two - estimating and retrospectives

I opened my second day of Agile Development Practices on a high note with Mike Cohn’s “Agile estimating and planning” session. Mike is one of the best speakers and trainers in the whole agile movement - he conveys ideas in a very powerful way, yet despite his experience and knowledge he remains friendly, accessible and laid back. I think the biggest value in his sessions is always in his replies to questions from the audience and the chance to talk to him during breaks - or listen in on others doing that.

Even though we have been estimating and planning with the team for quite a time I still found going through it all in an organized fashion very refreshing. It did rejuvenate my idea of moving the team from ideal days to story points in estimating the product backlog. Also quite interesting was Mike’s point that people are better at relative than absolute estimating yet favor precision over accuracy. A few exercises were designed so that participants could see how their minds make implicit assumptions to arrive at a number which is in fact less accurate and more misleading than a range.

One of the examples: how long will it take to drive from Orlando, FL to Seattle, WA? No one could answer it with a number, but most tried to making certain assumptions about speed, conditions etc. An accurate answer would be a range - let’s say 7 days +5/-3 days - but people would strive to give a single number. Precise seemed better than accurate. This is important to remember - as Mike underlined good decision making has to be based on accurate information. If precision is not possible at a given time it is better if the decision maker knows that. Hiding uncertainty behind a seemingly precise number denies decision maker important information hence leading to wrong choices.

It’s another point to what I and others have been saying about fixed bids being wrong. The whole concept of a fixed bid is requiring a precise answer (an amount, a date) at the point in the project where the certainty is lowest and the risk is highest. Is it ok if a client is served an illusion of certainty when there is in fact lots of uncertainty? Wouldn’t it be better if he was honestly informed that it is indeed at that stage a range rather than a precise number?

Afternoon I spent on a session about retrospectives led by Esther Derby and Diana Larson.

I read their book on the plane from Europe but I put it down a bit disappointed. The whole approach seemed a bit artificial. I imagined it would feel awkward and stupid to use the activities and techniques described with my team. I have to say this workshop changed my mind on this - seeing it all in action made the book kind of spring to life. Now it will become a resource to bring some fresh air to our retrospectives. I hope the team will enjoy that.

The day concluded with a reception (I was amazed that some were still able to eat after all the food we are served all the day during breaks) - a chance to chat with the others. Always an interesting experience - and uplifting, since I see that as a team we are much more advanced in using agile than most participants of this conference. I’m looking forward to another day - the format now will be less interactive, I’m afraid, but I’ll go to the open space - maybe I’ll have a chance to share some of my insights about using agile in outsourcing.

Andy's Mind: From conference: Agile requirements session

I attended Ken Pugh’s session on agile requirements management today on the SQE’s Agile Development Practices conference. What surprised me was that most participants were newcomers to agile - stuff like user stories, planing poker, story points etc. were a complete novelty to them. Judging from the questions & teamwork during the session and the chats during the breaks I have to say that we are doing pretty good as a team in this area. We have a pretty good process for starting projects and getting requirements from clients in an agile way. We get the most important user stories within one few-hour long initial backlog creation session - and then refine and evolve the backlog together with the client as the project develops.

Many of the techniques presented by Ken were applicable mainly to projects ranging from large to huge - and in corporate environment. But still I walked from that sessions with few ideas how we could improve what we do - and add to our toolbox for a bigger project when it comes our way.

One of the things Ken demonstrated was Business Value Points technique. The idea is to capture the relative business value of each user story as seen by the customer. It works pretty much the same as story points for effort or difficulty: stakeholders can do “planning poker” on perceived business value of each user story. As with each poker-style estimating only stories where there is a disagreement in the group get discussed at length and a consensus is achieved. The result can be then one of the factors in the planning sessions. Or it can be used along with story points to calculate how much value for the effort could be obtained. It is - I think - a great process for fostering some agreement on the priorities when instead of one product owner we have a group with different interestes.

Overall, this was an interesting day and I’m looking forward to tomorrow.

Square Wheel: squarewheel

Just a simple console session:

>> "Zażółć"[0,3]
=> “Za\305″
>> “Zażółć”[0,4]
=> “Zaż”
>> “Zażółć”.length
=> 10

OK, it quite works with Chars class:

>> "Zażółć".chars[0,3].to_s
=> “Zaż”
>> “Zażółć”.chars[0,4].to_s
=> “Zażó”
>> “Zażółć”.chars.length
=> 6

Not the most elegant solution, but works.

Andy's Mind: Dilbert meets Agile

In one of the recent episodes of Dilbert agile gets mentioned (not for the first time). One of my friends said that “Dilbert turned against us” but in fact I think this cartoon very pointedly shows how much a typical manager (Pointy-Haired Boss) knows about agile. Managers like the PHB from Dilbert cartoons do exists in large corporations for real, I’ve met some. But much more managers, accustomed to reporting, WBSs, Gantts, PMSs, 3 months requirements analysis, specifications documents, acceptance documents etc. do think that getting rid of all that means complete chaos and lack of all rules.

That the rules are scaled down it doesn’t mean they don’t work. In fact, rules have a much higher chance of working if there is just a few of them, not a whole 200 pages handbook. Same goes for documentation and almost everything else. Things have to be scaled down so that the rules and information are manageable to humans.

Take Scrum - it is just three roles (the Team, the Product Owner, the Scrum Master), two lists (the Product Backlog and the Sprint Backlog) and three meetings (the Sprint Planning, the Daily Scrum and the Sprint Review). That’s all, not much to tweak around, simple to introduce and then follow. Same goes for XP - simple ideas, easy to understand directions.

But simple doesn’t mean easy. Building unit tests for everything a