Crafting Software – avoiding the waste
In the last couple of months i have gotten attracted to a notion of crafting software which focuses on avoiding the waste. Allow me to explain :- Once we decide what a software should do and how it should behave then we give it to the software developers. Since software is so malleable you can create the required software in so many ways, using different components, databases, architectures, design patterns etc. The act of selecting the right components for the problem at hand and combining them in such a manner that the software is easy to extend and maintain is what i term as crafting software.
The problem is, the behavior of the software we are about to craft changes often. Especially in a product company you need couple of iterations before you get to know the exact nature of the software. You need to test it on a lot of different people to get a very good idea on what features to include,discard or change.
I guess the notion of software development should be such that you start with a vague idea of what you want to develop and in time with more testing and real world use you get more insights and things will get more clear. But current notion of software development goes more like this :- you release your first version after careful crafting and then as you observe people you keep on adding or changing features. After some time it comes to a point where you realize the choices you made initially before release weren’t appropriate now as there were so many changes in the assumptions you made during the first release. You throw out most of the code and start crafting the software which is apt for the insights you got. Then one more release happens and the same thing repeats. Users will have new set of problems, marketing will get some more ideas after observing user’s behavior etc etc. The software really smoothens out only after couple of iterations which might mean throwing out existing code and re-crafting it couple of times.
So you must be willing to iterate, test and then allow yourself to throw out the code and craft from scratch again if you are serious about crafting software. So much software doesn’t have that luxury today. Some things you can do to smoothen this process are – try and choose high level languages and frameworks with emphasis more on developer productivity compared to amount of computing resources it consumers. Computing infrastructure is relatively cheap while quality developer time is not. So optimize for developer productivity. While crafting software avoid premature optimization. you will be spending time optimizing something and after launch you will see that it really doesn’t matter. All in all if you see yourself as a craftsman of software try and reduce the amount of waste while you craft the software. Like many things in life you will get better at it with experience.
Insane hiring, young talent and marketing dreams
This is a cross post from our Slog Blog -
http://pagalguy.com/slog/2009/05/05/young-talent-insane-hiring-and-a-marketing-dream/
Backing up woes
You’ve got your website up and chugging along very well (shared hosting or dedicated server or a vps) and life is good – the traffic is increasing, you are hiring and one fine day the site crashes. You panic, check with your hosting provider and you are told that it was a server crash. The hard drives didn’t survive and your account will be restored from a backup.
The backup could be a day old, a week old or a month old. What also is of note is whether the backup was stored on the server itself, on a different server or on an offnetwork backup setup – or a combination of any of the above. You may wish to check with your provider about the backup setup because if your data really matters you, then you should be careful and proactive enough to find the best solutions that work for you.
Why o why?
Some of the above mentioned backup systems may not be helpful or be enough to save any of your data in case the server was compromised. If you backup all your stuff back onto the same server and a hacker finds a way into your machine – there is a chance he will destroy both your primary data and backups. You’re left with nothing after this, except if you had the foresight to do other forms of backups as well.
Lets say you backed up your stuff within another server in the data center, you are again not scot free. Say the DC goes down, or gets raided by the FBI (check HA post) or just a simple case wherein the server which was compromised is connected to the backup server using SSH keyless login (kinda required for easier setup of regular rsync backups) – here again you could have the possibility of losing all your data. Kinda scary isn’t it?
All this makes a case for Off network backups i.e DC down, FBI raid, earthquake, flash flood, or err.. an errant truck crashing into a pole and taking the DC’s electricity offline.
But wait, if you still allow your primary machine to login keylessly to your external backup system, a hacker can take out your data and backups as well. If you are doing an external backup (within DC) or outside the DC – then work with solutions where you can’t login to the backup systems without knowing the login/pass and the login/pass should never be stored on the primary machine. Take a look at solutions like Evaut or R1soft (we use this) to do backups of all your servers/accounts to an external provider.
We use R1soft because of a couple of features/advantages it allow us – first it does sector level incremental backups and therefore it doesn’t use too much outbound bandwidth as it transfers only the changed files (well, just like rsync), secondly it provides a control panel which lets us restores individual files, directories from any of our backups – we tend to maintain 30 snapshots of our servers at all times and in some cases over 240 snapshots. Finally the killer feature is bare-metal restore – say your box crashed – all you need to do is get a new box up, and specify the R1soft setup to restore stuff. It will replicate everything as per the last snapshot, including the OS. Kinda life saving if you ever need it. If you folks use any other backup setup, I’d love to hear 
While you might get all things right – I’ve seen cases where these backups were not verified and all the hardwork has gone down the drain because the integrity of the backups were not verified on a regular basis. Also you may want to try and restore your backups on a spare server sometime to ensure you have gotten it right. There are various backup options available today, opensource and commercial – but the above are some of the problems we take seriously with our data and work accordingly. It is never possible to have a 100% secure setup (someone just needs to find one loophole or exploit, while you have to continuously patch 100s of them) – but do spend the time and take the effort to build a backup system that appropriately reflects your value for the data. You can’t always spend a bomb to create a backup system, when you may be fine with losing 1 day worth of data
As you think critically of HA, your backup solutions too needs to be thought about in a critical manner that reflects the importance you accord it.
Why write Code ? Because i can test it :)
ok, I might have slightly exaggerated my motivation towards why I write code but my fascination towards test first development came from learning lessons the hard way. In later part of the post i shall explain test first development in more detail but before we get there we will come back to the question – why do we write code? While acknowledging the fact that we all might have myriad number of reasons ranging from “programming jobs are plenty in number” to “writing code is just so liberating” i will repeat what Frederick P. Brooks said :-
The programmer, like the poet, works only slightly removed from pure thought-stuff. He builds his castles in the air, from air, creating by exertion of the imagination. Few media of creation are so flexible, so easy to polish and rework, so readily capable of realizing grand conceptual structures.
If the above paragraph seemed like shakespearean language it simply means “When solving a problem, whatever solution you have in your mind,it is so easy(relative to other media) to put it into code. Since the implementation of your solution is trivial majority of the time will be spent on providing a solution to a problem than grappling with implementation details. The sheer smoothness of the medium to solve problems leads to the joy of coding(i added this 
)”.
But code has a tendency to grow beyond initial expectations, quickly surpassing our capacity to fully understand it. we have all experienced how end phase of any project looks like. Development slows down to a crawl. we are all busy taming many headed bugs. we wrestle with somebody else’s code just to give up or fully re-write in despair. we sit in frustration as we see countless hours of work go up in smoke while people from other departments can’t understand why programmers are taking so much time to finish off the last 1% of the project.
This doesn’t even remotely sound like the utopian land as described in above paragraphs. So if we still have to spend obscene amount of time grappling with implementation details then may be coding isn’t as utopian as some people suggest. ok, very sad truth but we have to move on. so give up coding, update your resume and try your hand at some other job. wait, i am just kidding. There are ways to bring back fun into coding. one of such ways is test first development. It is a simple process which might take some time to get used to but once you are there the process will be fun all the while solving most of the problems mentioned in the above paragraph. No i am not kidding this time – the process is very enjoyable and as a side affect it will solve most of the problems mentioned in the previous paragraph.
Test first development is a well documented process. It consists of very rapid and short cycles. In each cycle you write some tests, code enough to pass those tests and then improve the design. This is famously known as red-green-refactor cycle. Let me explain each step in a more detailed manner.
1. Red :- Suppose you are about to write down a class. First thing you need to know is get a very good understanding of what the class should do and what it should not do. Then you write a test to validate one of the behaviors. Don’t complicate anything. Take a simple behavior and write a small example as a test to validate that behavior. At this stage you have not yet written any code in the class itself. Now run the test and the test bar should turn red.
2. Green :- Now go to the class and write small code just enough to make the test pass. Nothing more. Don’t worry about reusing or repeating any code. We will get there soon. Just concentrate on writing enough code to pass the test. Now run again and see that the test bar turns green. Yahoo. Congratulations you have just written a well tested code.
3. Refactor :- Now that you have a test go back to the code and see if there are any duplications or anything that makes the code look ugly. Now that you have a test available go ahead and change the code to your hearts content making re-usable components and beautifying the code. Don’t worry about breaking anything. you have already written a test and it will tell you if you break anything. Now repeat the whole red-green-refactor cycle for the next behavior.
Each step will hardly take couple of minutes and usually when you are productive you can cover upto 20-40 such cycles per hour. let’s see how this process solve couple of problems we discussed at the start of the article.
First you are working in baby steps constantly checking whatever you have written(“The bar should turn red now…..now it should turn green…now it should still be green…now it should turn red again..”). If you made any mistake then you will catch it right away and since you know the mistake is within the couple of lines of code you have just written for the latest test it is very easy to identify and correct the mistake. No more frustration while reading through your code trying to figure out where you have made a mistake and what were you thinking when you were writing that code. One major hassle gone. Now once you commit code along with the tests, one of your team mates wrote some code in his own class which will break some of the functionality that you have written. Don’t worry the test will fail and test bar turns red. Time to stop whatever you are doing and huddle in to solve the mistake then and there itself until the test turns green. This itself will solve most of your frustrations which usually lead you to “wrestle with somebody else’s code just to give up or fully re-write in despair”. We all know that finding mistakes, not fixing them immediately, is the most expensive part of programming.
This process solves most of the common problems which are the major culprits in making you a bad programmer. Next time i shall come up with one more process which we are setting up now which should improve your code quality further. Hola and have a nice time trying out test first development. If you want more material to read on test first development, just google “Test Driven Development” and you will get enough material to quench your thirst. Any help needed please put it in comments and i will be glad to help.
High Availability
Every time we grow & traffic sets new records – we never fail to be happy at the need for more servers to be added to our rack. The happiness however very soon makes me cringe because I know it is going to cost more to add all these servers to the rack and what is even more tough is to keep these set of servers chugging along at a good speed, secure and easy to operate.
I’ll focus on the high availability (HA) challenges of such a setup and why the costs get very steep very soon in case you require HA. The challenge for a startup is to work on an architecture that allows you to start at the right scale and then extend it as painlessly as possible.
For an early stage startup with minimal traffic, you can get by with a single server handling files, databases, emails and the webserver as well. After you’ve grown for a while, the database has the biggest chance of becoming your bottleneck, unless you are serving tons of files really fast. Now you would need to put in a separate server which just runs your MySQL installation. Right after that you would realize you need more machines on the frontend to serve all the awesome goodies.
This process is vicious and very soon you will have a couple of machines upfront acting as the front end and a couple of database servers. To keep MySQL playing nice, you should have sharded your data across multiple machines and/or put up those slave MySQL servers to which you can force your application reads. Now while you need to pat yourself on the back for being able to generate this much traffic – the next set of challenges are just starting.
This is the time you start worrying about single points of failure. HA simply means your system stays up and alive even if certain parts of the infrastructure come down on you. Hopefully you put up a hardware load balancer or a failover system w/software load balancers (haproxy comes to mind) – because if you didn’t and used a single load balancer on a server, all you need for your entire site to go down is that one server to go down. All your front end/backend servers come to naught when you don’t load balance/failover your loadbalancer 
Here you were operating on a really tight budget and major chinks in the architecture start showing up. What if the master MySQL server goes down? Now usually these are fairly expensive beefy machines – but the fact is that hardware goes down and it usually does when you are the least prepared (hail, murphy!). Now are the choices you need to make – do you put up a similar beefy MySQL server and wait for the 1% time your server would go down and then failover the setup ? Or do you err. just chalk that 1% downtime to .. err.. you know .. keeping it cheap? Or you could HA MySQL by using a master + master configuration. However if you need to take frequent backups of your data, you may need a master + master + slave configuration so that you could shut down the slave for a brief period of time and do all those important backups and send them to your backup systems. While adding your set of machines to the setup, do ensure you have all the servers on a private vlan connected with gigabit cards/networks. Fast interlinkages, plus you don’t have to pay for the internal bandwidth transfer.
Lets also talk about backups – you do have backups dontcha!? Backing up your files/systems etc on the front end servers shouldn’t be too tough. You could rsync them over to another location and keep off network backups or use bare metal restore solutions like R1soft and have the ability to restore entire servers very easily. The costs of additional backup servers & the bandwidth to do snapshots at short time intervals become drivers of your decision as you continue to keep everything safe and secure. Explore multiple backup systems and do practice restoring them – that would be instrumental in getting you up back online fast. Keep a copy of your data backed up off network – this is critical – you don’t know when your DC can be raided by the FBI and you might lose all your servers. Allright, if its not the FBI, it could be an earthquake, bankruptcy, power surges, building fires – whatever. If your company means more to you than the montly server costs, backup – relentlessly.
As you continue to grow and think HA, make sure you have no single point of failure that jeopardizes all your hard work. One site I knew had a fairly comprehensive HA setup, but they kept their DNS on one server. Sadly, when that went down – nothing remained accessible. A gentle reminder that when going HA, work hard to isolate all such possibilities and remove such dependencies as soon as you can
Speed matters
If there is anything worse than being on the wrong side of the digital divide, it is being on it. A considerable part of India now has access to Internet, but so abysmal are the connection speeds in sub-urban towns and college networks that all efforts to increase user productivity in a web-app through intelligent UI can come to a naught.
We put PG Apply, our web-app for the larger mass of India seeking MBA admissions, to a small usability test at a Pune-based engineering college this weekend. Seven fresh graduates from a few private engineering colleges of Pune were put to a card sorting test and then a screen capture analytics test using Morae.
The pathetic Internet speed (download speeds of 5 kbps) on the college wi-fi was a welcome addition to our initial test environment of Internet Explorer 6 + Windows XP, making it all the more real.
- Contrary to beliefs, a user from a non-urban background (example: Marathi medium upbringing in Gondia, Eastern Maharashtra) does not have difficulty figuring out the function of complex UI patterns. As long as the flow has sufficient cues, a user very quickly figures out how a process funnel on a lightbox will work, or how to interact with a multi-select dropdown with checkboxes.
- What plays foul is a bad and unresponsive connection. It is not sufficient to just test the software on Internet Explorer 6. It is equally important to test it on realistic connection speeds. If your page load includes a CSS file, a few images and a JavaScript file, then on a slow and unresponsive connection they will all load asynchronously, causing the dreaded ‘Done, but with errors’ yellow icon to show up. Optimization of page loads, serving image sprites – they aren’t just cool developer fads of the day that can be done without. They are probably the deal clincher in deciding whether someone in a small town manages to see the face of your site or clicks the X button in frustration.
- In web forms, assisting cues or examples are required at the unlikeliest of places. When we asked why some users spent a few seconds hesitating to fill the ‘First Name’ and ‘Last Name’ boxes, they replied that they were not sure whether they ought to enter the name in capitals or smallcase!
- For all the advertising of ‘Free Laptops’ by private management colleges, MBA aspirants don’t hold it as an important parameter while joining a college. They also don’t deem the presence of hostel facility in an MBA institute as important. Reasons being – the vast landscape of private education in the country is based on optimized infrastructure. Hundreds of private engineering colleges across the country function without hostels; students stay in paying-guest accommodation. After having spent four years in such a system, hostels for post-graduate education are irrelevant to them.
The usability testing project has increased our curiosity to similarly look up user behaviour in different parts of the country. How do users in different cultures and markets of India interact with web artifacts? How do they differ in their need of information?
One of the things we are looking at is to build a comprehensive Design/UI Pattern Library for the Indian context, rigorously tested in real Indian conditions over a period of time. Do you know of such a project already existing? We’d love to know. But if you are a Graphic Designer / Interface Designer / Frontend Developer interested in such a challenge, do apply to us at jobs@pagalguy.com
Card sorting
More card sorting
Screen capturing on Morae
Gratification!
Hello world!
Welcome to Thought Crime! We are an set of opinionated technology, design, content, marketing and HR professionals working at PaGaLGuY.com. We’ve been in business since 2002 with a rather counterintuitive modus operandi. Here we’ll share how we do business, our small and big achievements and learnings thus derived. Hope you enjoy reading!
