lean software development (english version)

It was a sleepless night, when I first learned of the lean manufacturing idea. just arriving back from the US, I was fighting the jet lag by watching random programs on the history channel. then came a program about the car production lines, which overall is a fascinating subject (and if you think otherwise, then there is something is terribly wrong with you)

what caught my attention was the problems that Mr. Toyota encountered when studying Henry Ford’s production plant. while Mr. Ford could sell cars “painted any color he wants as long as it’s black”, then Mr. Toyota had to make small quantities of various types and models. My production manager used to frequently complain: “every day you ask me to make a product with different color pom-poms”. manufacturing people naturally like to create production line a-la Henry Ford (who actually took many of his ideas from Eli Whitney)

so I decided to look into what exactly they did in Japan. I found one of the leading experts in Israel. His name is Ofer, and he opened up our way of thinking with fascinating ideas, while sometimes difficult to digest. but it was worth it, and ever since I’m a big fan of the system.

at the end of the process, Ofer tried to convince me in the universality of the lean methodology, and it’s adaptation to other fields outside of the production plant, even product development.

we didn’t really succeeded in doing that. the R&D department has a hard time with the production line analogy. they are of course smart and creative. some of them graduated the best universities. the comparison to Charlie Chaplin’s bolt tightening scene was too much.

they were not right, but not entirely wrong. if you dig down many of the software technologies common today, you’ll find tool that enables what is knows as java-school programmers create software by following simple repetitive tasks, just like those in Henry Ford’s plant. for example you might look at the MVC books. while I really love the paradigm, many of the implementation is simple copy-paste of templates. on the other hand, one of the main ideas on the toyota production system is the recognition that the human being is not a robot, but the smartest a most flexible “tool” in the plant.

fast-forward a few years, while working on a different project I came to learn some totally different methodologies. while they originated from different set of needs, they reminded me of the same ideas. I’ll try first to explain the principles of lean manufacturing in order to show how they interact so nicely in software creation process, and will try to show how you can adapt physical production ideas into code and information development.

In lean production, everything you do is classified into one of three options:

  • value adding – every action that the customer will be willing to pay for. an action that advances the product in the production process.
  • necessary but not value adding – usually action that is enforced by management or regulatory authorities.
  • non value adding – everything that do not change our product (or information)

non value adding action are the greatest sin of lean production. they are usually referred to as waste, and there are 7 categories of waste:

  • overproduction – adding features that was not asked for. in physics, work equals power multiplied by time. in the modern workplace, I’ll replace force with productivity. many time we confuse “work” with “productivity”. we encourage workers with a high productivity, but fail to make sure that productivity is utilized into a profitable work. the textbook explanation of what’s wrong in overproduction is that it create inventory. those items need to be stored and managed, so we rent a building and hire people, which in turn needs to be serviced and managed. all this create a burden that drive our cost up. but what about too much data? excess information can definitely make decision making more difficult, confuse us, and sometimes make us take the wrong route.
  • waiting– while in the past the main constraint was availability of machines, even today we seek to find and solve bottlenecks, while mostly in human resource, like field experts or the QA queue.
  • transport – moving of our product from one place to another, trying to avoid any damage that can occur during transport, which is in his nature not value adding. but what can happen to our data while travelling in our super fast giga networks? admittingly, not much. but you should make sure the right data gets to the right people at the right time (more on that later on)
  •  Over Processing – resulting from poor tool or bad design. I’ll add “pushing” of “hot” technologies that excite the engineers but does not make a difference to the end users.
  • Inventory – one may argue that the cost of memory devices are so small, so we should not be concerned by it (any IT manager that needs to perform daily backups of an entire company may very upset by this comment). so what about code inventory? I really hope you allready use a source versioning system. managing that system do take time and effort, but they do not grow as your code base expands, so no big issue there. what does grow are the risks of bugs and security holes.
  • (unneeded) motion – every action of the engineer that is not directly contributing to the product. I’ll mention setting up the work environment, getting needed equipment and devices, meetings and bureaucracy.
  • defects – anything that will not function as expected. we call it bugs.

so we’ve identified our problems. let’s see what solutions do we have for them. the “two pillars” of lean management are “just-in-time” and “autonomation” (smart automation).

autonomation means letting machines do the boring stuff, and use people to oversee everything is running smoothly. this idea originated by the design of a automated textile loom by Sakichi Toyoda. these looms stopped themselves when a thread broke. it then was implemented at the production line where workers we encouraged to stop the line immediately as the spotted something is wrong, thus not letting the problem spread. the motivation is that fixing a problem early on cost less. after each event, the root cause was checked and fixed. in this way faulty products do not leave the production line and quality is increased (and you can ask any fleet manager about reliability of Japanese cars). In software engineering we have unit testing and automated testing. by using these we can decrease dramatically the repetitive tedious tasks while still making sure we provide a tested and functioning product. it also eliminate the recipient of your code to verify your work.

Just In Time means making only what is needed, when needed and in good quality. pursuing this goal, there are several things we can do

  • small batch size – which is a fancy name to making one thing at a time and passing it down the line. in Scrum methodology it is referred to as “sprints” which enable us to pipeline our work, and also making sure we do not “bite more than we can chew”. it also makes are more flexible to changes. many time software developers complain when they are thrown out of a task to fix some other issue. but when each task is a short time limited task, it is often possible to wait until it ends and only then reprioritize upcoming work.
  • pull processing – meaning we do not create code for the sake of coding but only when our customer (external or internal) needs it. this happens when using behavioral driven development. you define exactly is required by the user and then you code what is needed to implement that, and only that.
  • canban – maybe one of the most famous terms out of the lean methodology. it literally means a “sign” or “card’ in Japanese. the idea is to have a simple visual que that a work needs to be done. using bug tracking systems and continuous integration servers gives us that dashboard.
  • 5S – Sort, Set, Shine, Standardize, Sustain. the idea of a clean and tidy workplace may not be unique to anything you do

any of these ideas ring a bell? if you ever heard of the agile manifesto, I guess it does. many of their ideas were actually derived from lean. this is why the solutions also remind each other. but I think it is worth a while to study the original ideas to better understand these solutions.

if you even tried to implement lean or embrace agile, you know it is a daunting task. conventions and lifelong habits are hard to change. and once a problem occurs, it’s people tendency to return to their tried and true way of doing stuff. which brings us to one more idea

  • kaizen – continuous improvement. it is not enough to teach once, but the process must be followed and improved all the time.

so what have we learned so far ?

I think the most important lesson is to keep an open mind. in the hightech industry (especially the Israeli one) there is a lot of arrogance. too many people are sure they invented a new world regarding technical issues, management, human resources or other subject. it does not hurt to learn from others experience.

there are many other lean tools and ideas I did not mentioned. it would be very interesting to dig dipper and find which other analogies we can find and borrow to make better software.

“if all you have is a hammer, everything looks like a nail”. lean manufacturing is just one methodology which try to solve certain situations . there are many other philosophies that are very interesting. you always need to understand your requirements and find the best solution to them. it like one Microsoft engineer once told me: “we use the traditional waterfall development flow, and it works great for us”