Freelancing in India – The oriental carpet shop

According to an ancient saying in India, the secret of long and fulfilling life lies in avoiding four types of people at all costs - bankers, doctors, lawyers and police. There is only way to avoid these four types – don’t earn too much, take care of your health, don’t get into disputes with anyone, and don’t commit any crime. This must be true in most countries, but here in India - this is the most practical thing to do.

In India, the laws are very favorable to the weaker party - it is impossible (at least legally) to fire an employee or avoid payment. But, the unfortunate part is that if you go to court - only your great grandson may hear the final ruling. This practically rules out any legal recourse for many people. Therefore, any legal documents that you sign are practically useless - at least as far as YOU are concerned. The advantage is you don't need to consult a lawyer. But, the disadvantage is you are on your own. You have to evolve your own way of getting your payments on time, getting your deliverables accepted and work out the complexities of the inner workings of your client organizations.

First, let me tell you a couple of stories from the field. The first story is about the need to read carefully all legal agreements and more importantly the price of arrogance and individualism that one pays for in this country. The second story is about the need to understand the subtlety of the eastern mind.

I know of a consultant who worked for a billionaire. There was a disagreement between them on some issues - the issue was not about money, but the working styles of the client and the consultant. This particular consultant was very arrogant - one day he had a major fight and walked out of the assignment. According to the agreement he signed - he could leave with one day notice, so he thought everything would be fine, and at best his client wouldn't release the pending payments.

Two days later, a battery of lawyers descended on his house - and practically sealed everything - his wristwatch, cell phone, microwave, TV, credit cards, computer - and practically every single tiny electronic item he owned was sealed. The reason: he signed an intellectual property document, and the document says that in the event of termination of the assignment, the client has a right to verify that no confidential data was stored in any electronic medium!! Technically even a wrist watch is an electronic medium.

This poor chap couldn't even make a phone call, and his client was out of contact for a week. Finally, he had to swallow his pride and beg for pardon.

The important lesson is that be careful of the documents you sign with your clients. Any condition in the agreement must be something that you yourself should be able to establish.

The second story is my own first hand experience. In India, your clients will never dispute anything - especially about payments. They fully agree that they have to pay and they will always tell you that they have every intention to clear all the pending payments as soon as they can. They will never get into an argument with you on payments. If you ask about payments - they will talk to you for two hours, tell you how much they value your work and relationship, take you out for lunch, make you very comfortable - but they always find some way of delaying the actual payment. In the west, people are in general have an 'on your face' attitude, but here it is very subtle. It is like going to an oriental carpet shop and negotiating for price - everything is discussed except the price.

Once, I completed my assignment, and sent all documents to my client, and sent them the invoice. There is a thirty day period for all payments - so, I waited for about three weeks. Nothing happened. I contacted the finance department - they told me that they did not receive any approval from the VP, Engineering - who is the person in charge for my consulting work. I met him again - he apologized profusely for the delay for about half an hour, and in the meanwhile asked me to 'help' them with a few small things and so on and so forth. I obliged - it took another one week. I sent the invoice again to the VP and the finance department. This time, the VP took a printout of the invoice - and he wrote on it the three magic words: "please pay" and signed. I handed over the invoice to the finance department. The accountant gave me a charming smile and said everything will be done in a week's time. Well, nothing happened for another two weeks.

Again, I met the VP. This time he cursed their finance department - he said they are always very slow, don't do their job on time etc., etc. He again took a printout and again wrote the three magic words and signed. I took the invoice to the finance department. This time the accountant asked me to wait - he wrote the check and gave it to me within half an hour.

I asked the accountant that why they did not clear my payments the first time the bill was passed by the VP. The accountant said "Sir, the first time he cleared your invoice, but wrote it in blue ink. That is an instruction to us not to pay. If he writes it in green ink, then it means send the check after two weeks, and if he writes it in red ink - it means that we should pass it immediately".

This is how subtle the process could be here. I use some simple rules to deal with the eastern complexity and it works.

First, I always do one week of gift work before I formally engage with any clients. I don't charge for the first one week, and I work without any formal contract. This allows me to understand several things - most importantly I get to understand the problem I am expected to solve, I get to assess whether I could do it or not, whether the team is capable enough to solve it themselves - how much guidance they need from me and so on. Most importantly, I can estimate the culture of the organization. Metaphorically speaking - I get to know in which color the invoice has to be signed!!

A consultant has to deal with three different people in the client organization. The person who employed you, generally it is a senior manager - most often, the head of development, delivery or technology. The second person is the one who actually needs you and works with you - generally it is the development team, represented by a project manager, and finally - some one from the finance department. These three people can form a nice little triangle and play a perpetual football with you.

What I do is to try and break the triangle. I insist that one person from the development team - either a project manager, team lead or a programmer be assigned as my primary client contact, who will have all the authority to decide on the deliverables, to sign off on my time sheets and who has to ensure that payments are cleared. This means that even though the agreement is signed between me and a senior manager of the company, the agreement explicitly nominates one person by name.

The advantage is that - programmers do not switch off their cell phones, they do not go on extended foreign tours, and they don't have secretaries. You can always call them up and go for a lunch. And, they understand which color of ink their boss has to sign. Basically, I make my internal client contact run around to clear all my payments. Once they accept this responsibility - they are in general very sincere and prompt. Contrary to all generally accepted beliefs, I found that the finance department tends to be the most efficient and prompt of all other divisions of the organizations. Once they get clear instructions, they act very promptly.

The second part of my consulting agreement is to clearly identify three types of deliverables. In India, the clients do not draw a clear boundary between the consultant and the organization. The consultant - especially if he is a freelancer - is always treated as one of their own. This has several advantages and disadvantages.

The disadvantages:

• The scope always expands.
• No clear idea of what the deliverables are - the distinction between the work and the deliverable are always blurred. You are asked to do work but get paid based on certain concrete deliverables that you have to produce.
• Confusion about who is owner for a certain area of work.

Normally, what I do is to identify three types of responsibilities and document them as part of my 'work contract':

• Primary responsibilities: I have the primary ownership for these deliverables, and I have control on the project plan.
• Collaborative responsibilities: My expertise is required for these deliverables, but someone else has the overall ownership and control on the pace of the activity.
• Participatory responsibilities: Areas where my inputs are sought - but, I have a right of refusal to participate.

Finally, it is important to get the payments on time. Suppose I expect the payment to be done on 30 of the month. If I send the 'deliverable' document on 25th, I can never get the payment on time. This is because, the deliverable is not completed when I send it, but when it is read and understood by the receiver. So, if I want my payments on 30th, I send my documents on 10th of the month - and give them two weeks time to accept the deliverable, and also I use the two weeks to ensure that they indeed make the effort to understand it clearly.

I wish all of you and your families a very happy and successful new year. May the Lord bless you to find your heart's desire.

Freelancing in India

Many young people from India are writing to me asking about freelancing. If you are also ‘bitten’ by the freelance bug – and you are thinking of writing to me, read this before you shoot that email to me.

I believe self employment will take off in a big way in India in the next two to five years – therefore, there will be lot of scope for people to be on their own. There is a need for specialized knowledge, speed and flexibility in our industry. Many people are now well connected, well travelled and they know their bearings well. All this means that the situation is ‘cooking’. It is the right time to jump in for the early mover advantage.

However, freelancing is a difficult thing to make it work – anywhere in the world, and especially in India. It is not going to be straightforward.

First, let me dispel a few myths – freelancing is not financially lucrative in the long run. People who work as freelancers do so because they love to be independent, have some time on their hand to pursue various other interests and generally be the trailblazers. It is not a road to make wealth. If you are a Peter Drucker – it may be different, but even for Drucker – he would have made lot more money if he started his own company or worked for some other big company.

There is never any ‘job’ security if you are on your own – it is always a struggle to find the next project, and do it well and have some regular income.

There are a few tax advantages – but the overall expenses far out weigh any tax advantages. For example, you have to invest in your own training, and in your own career. If you work for any large IT-Company, they take care of all your training needs. If you work for a large IT company in India, this amounts to about 50K per year (including the training expenses + the salary you get even though you are attending training). Similarly, there are many other expenses if you are on your own. These days many IT companies provide many perks – which are tax free incentives, you don’t get anything more if you are a freelancer.

Part time job, second income is completely different from being on your own. There is no comparison at all.

There is also a big difference between contracting and consulting. One is getting some work and executing it, and the other is lending your expertise to produce results. These two ways of engagement work very differently.

So, think carefully before making a move. You can always get the first couple of contracts – but can you get business for the rest of your life? I do not want to discourage anyone – but only trying to throw some light on a few dark areas that people generally don’t see.

Now, here are a few tips:

To be successful you have to:

• Establish your credentials – credentials are not your resume, but your expertise and demonstrating that you can deliver. What you can do is to first try and get some work from internet. There are now some websites like rentacoder.com, elance.com etc., - which help freelancers find work across the globe. You can register yourself on these websites. It works pretty much similar to e-bay. Some companies post their requirements – and you can bid for projects, execute them from home. I suggest that you find out more about sites like this, and see if there is some work there that fits your skills and what you are looking for.
• Transition from ‘a skilled person’ to ‘an expert’ – you have to develop certain techniques, a way of doing your work that establishes your expertise. You have to learn to write well, communicate well, publish some papers, have a website of your own and carve out for yourself a niche. There are tons of open sources projects these days – if you participate in these projects, you get to learn the world class standards that are used today, and also you get to meet other people who are freelancers and create a network of your own.
• Have a network of contacts, friends and others from the same area. If you are working in a company – even though you don’t realize, you are part of a community which ensures that you get to know a lot of things from the environment. If you are a freelancer, you have to cultivate different kinds of relationships. There are many ways of accomplishing this – membership in professional bodies, attending conferences, teaching in universities, making contact with a group of people who work in your area (see point no: 2).
• Understand how consulting works. There are some excellent reference books on this subject. Flawless Consulting by Peter Block, Consultant’s calling by Bellman, Soloing are some books that I recommend. Read them, understand them and then think about how to go about it.
• Read a lot. You have to read many different kinds of books – not just technical books. The general thumb rule is that you have to be able to read at least 500 pages of material every week and absorb it. If you claim to be a knowledge worker – you have to have knowledge. Right? Some books I suggest to begin with: Gerald Weinberg’s books on Software Engineering, Quality Software Management and his books on consulting. Also books on agile techniques. Books on systems thinking, history and some poetry. Definitely you have to know a great deal about accounting, managing your own finances and legal aspects of self employment.
• Invest in your own career. This means that you have to have a definite idea of how much free time you will have, and how you will use the free time. If you do billing work all through the year – you will become stagnated very fast, and you can’t get business the next year. Our field evolves very fast, and for a freelancer it is very important to be on top of the technology. Therefore, it is mandatory that you have a definite plan how to stay ahead – this will include, reading a lot, attending workshops and training programs, write and communicate, teach and so on. In financial terms, you have to invest at least 30% of your revenue in your own learning at least in the initial years.

That’s pretty much it. If you are committed to it – it is not hard. Believe in yourself, make a small begining, do it well and don’t give up. That’s all there is to it.

Secrets of Successful Freelancing

Many people ask me about freelancing – how it works, how do I make it work and so on. Most often I duck the question because it is a difficult question. This article is a short guide on how to survive as a freelancer.

It is not easy to be a freelancer. Most freelancers want work, not a job – which makes it tough for them in the long run. They want freedom, they value competency, they think they bring real expertise to the table and want to continue to enjoy the challenge of work.

I always suspect the people who tell me that they enjoy learning. Learning is a painful process, why would anyone want to learn in the first place? We learn best only if we need to. Children learn so fast because it is a question of survival. Psychologists define learning as a permanent change in behavior due to a prior experience. Genetically, we are designed to resist any change – whether it is for good or for worse. Apparently in the same way we have an internal thermostat which controls our temperature – we have another instrument whose function is always bring us back to the original state. This is the reason why changing anything in ourselves is such a massive effort.

Learning is a lot like being in love – and being love is a painful experience. The separation from one’s beloved is a suffering and the lover secretly craves for that rupture involved in that longing. This is a mysterious process – much like the drug addicts cravings for a drug.

In the initial phases of my career as a freelancer, I deluded myself that my clients hire me because of my expertise, knowledge or the skill I possess. I deluded myself that I am expensive because I am very valuable. I deluded myself that doing the best job is the most important thing in the world. I deluded myself that doing the best job on time will get me my payments on time and also bring me fame and respect, and even an extension of the contract.

Well, people value those traits. As freelancers we are respected for our expertise, our knowledge and our skill. Eventually, these qualities might even allows us differentiate ourselves in the market place and allow us to quote a premium for our services. We may be valued for these traits, but we are not hired for these reasons. And, we never retain our contracts for these reasons. We retain a contract only if – as people – we demonstrate very high levels of personal integrity. If we have no personal integrity – then we cannot sustain ourselves as freelancers even for a month.

Clients hire freelancers primarily for business, economic and operational reasons. We are too expensive to be on their employee roles. We are temperamental prima-donnas who are fiercely independent and therefore in the long run we are an expensive maintenance problem that nobody wants to have on their hands. We are addicted to problem solving, and therefore we cannot stay in one organization long enough.

These are the real reasons why we are ‘allowed’ to be freelancers.

It took me a long time to understand this. Most problems between consultants and their clients arise because of ‘expectation’ mismatch. We expect to be valued for our contributions and for our love of work. Organizations value and reward loyalty – not loyalty to work, but loyalty to themselves.

It was a difficult learning. But, after I realized it, it is not too hard to internalize it. The following eleven rules are a direct consequence of this realization. If you practice these rules – you may not become a world class freelancer, but you may be able to survive as one – which is no mean accomplishment in itself.

1. If you are a Socrates, be prepared for a painful exit. Remember - Socrates did not complain about painful exits.
2. Do not blame the stimulus for the response. You are the one who seeks problems – so, never blame the client for giving you the problems. How you experience the problem and your (in)ability to solve it belongs only to you.
3. The estimation of the self must be tempered by the knowledge of the self.
4. A King will never ask his general whether he is ready to defend the country. The fact that the general is still in his job means that he is ready to do so. Therefore, the King will only give marching orders.
5. As a consultant, the only right you have is to do your duty. Always remember that it is your right. Fight for it, protect it but never allow anyone/anything to deny this right to you. Most important – don’t deny it yourself.
6. A freelancer works for cash - not for kind. Therefore, we don’t get paid twice. You can either take a check or receive gratitude. Learn to prefer the check to gratitude.
7. Always do a thankless job. The only thank-you note you can expect is the payment made on time.
8. Be thankful if you get some work to do, and you get regular payments. These are the only two objective measures that you can use to assess yourself. The rest are delusions.
9. The Shareholders control the company, the founders operate the company, the employees are the company, and a consultant is associated with the company. All data modelers know that association is the weakest relationship – it has no integrity constraint imposed on it. It is the easiest and most convenient relationship to terminate. No reason or rationale is required to terminate it – because such a termination does not break down the underlying system.
10. The success of a consultant is directly proportional to how fast he/she makes himself/herself redundant to the client.
1. Solve that problem as quickly as you can and get out.
2. If there is no problem to be solved, get out.
3. If you cannot solve the problem – get out.
11. Luxuries that a freelancer cannot afford:
1. Anger – never walk out of the contract in anger. Don’t leave especially after you have had a fight with your client over some issue.
2. Resignation – an employee can resign. A freelancer cannot. There are only two ways to exit: either complete the assignment, or get fired. Sometimes getting fired is not as bad as we imagine it to be.
3. Procrastination – you sell your time, so it does not belong to you.

The Weinberg touch to Rule No: 11:

I had some difficulty and even used to get angry with the last rule. I felt that there was something wrong with it – it did not sound right. But, I kept it – because, it served me very well in my initial years as a freelancer. My discomfort also meant that there is something deeper lurking inside that rule. I was confident that I will eventually discover what it is.

Recently I posted these rules to Gerald Weinberg’s Shape Forum. Jerry disagreed with this rule and suggested a very subtle shift to this rule. The nudge he provided suddenly threw light in the dark corners of the rule and the full beauty of the rule suddenly became evident to me. So, I dedicated the Rule Eleven to Gerald Weinberg.

I call it the Weinberg Touch:

11a. never leave anything behind – especially your anger.

11b. An employee has an option to resign – a freelancer has an option to renegotiate.

11c. Your non billing time is more valuable than you’re billing time.

11d. Getting fired is never as bad we imagine it to be and it is more often than not a blessing.

Effective application of these rules however requires some tact. Here are two Nasruddin stories that capture the essence of these rules and how to practice them:

******

Whose Servant Am I?
Mulla Nasruddin had become a favorite at Court. He used his position to show up the methods of courtiers.

One day the King was exceptionally hungry. Some aubergines had been so deliciously cooked that he told the palace chief to serve them everyday.

‘Are they not the best vegetables in the world, Mulla?’ he asked Nasruddin.
‘The very best, Majesty’

Five days later, when the aubergines had been served for the tenth meal in succession, the King roared: ‘Take these things away. I HATE them!’

‘They are the worst vegetables in the world, Majesty’ agreed Nasruddin

‘But, Mulla, less than a week ago you said that they are were the very best’

‘I did. But, I am a servant of the King, not of the vegetable’

The Gold, the cloak and the horse

‘I cannot get a job’ said the Mulla, ‘because I am already in the service of the All-Highest’
‘In that case’, said his wife ‘ask for your wages, because every employer must pay’
Quite right, thought Nasruddin
‘I have not been paid simply because I have never asked’, he said aloud
‘Then you had better go and ask’

Nasruddin went into the garden, knelt and cried out: ‘O Allah, send me a hundred pieces of gold, for all my past services are worth at least that much in back pay’

His neighbor, a moneylender, thought he would play a joke on Nasruddin. Taking a bag of hundred gold pieces he threw it down from a window.

Nasruddin stood up with dignity and took the money to his wife. ‘I am one of the saints,’ he told her. ‘Here are my arrears’

She was very impressed.

Presently, made suspicious of by the succession of delivery men carrying food, clothing and furniture into Nasruddin’s house, the neighbor went to get his money back.

‘You heard me calling for it, and now you are pretending it is yours’ said Nasruddin. ‘You shall never have it’.

The neighbor said that he would take Nasruddin to the court of summary jurisdiction.

‘I cannot go like this’, said Nasruddin. ‘I have no suitable clothes, not have I a horse. If we appear together the judge will be prejudiced in your favor by my mean appearance.’

The neighbor took off his own cloak and gave it to Nasruddin, then he mounted him on his own horse, and they went before the judge.

The plaintiff was heard first.

‘What is your defense?’ the magistrate asked Nasruddin.

‘That my neighbor is insane’.

‘What evidence have you, Mulla?’

‘What better than from his own mouth? He thinks that everything belongs to him. If you ask him about my horse, or even my cloak, he will claim them, let alone my gold’

‘But they are mine!’ roared the neighbor.

Case dismissed.

Service Oriented Computing and Tech Shamans

The Design and the Designer – Part 8

There is a lot of talk about Service Oriented Computing these days – mostly by people who don’t seem to understand what it really stands for.

Until now, we have very good models of infrastructure engineering and applications engineering, but we are yet to see a reasonable model for process engineering and structural engineering in the computing science. This could be because of the unfolding of services as a primary business model has just begun and we have not yet understood what service really is.

A very successful billionaire once told me that the Services Automation is several orders of magnitude more significant than the industrial revolution. The explanation he offered for this significance was very striking. It has the beauty of simplicity and the brilliance of Truth to it. According to him, the basis of industrial revolution is control of resources and centralization of knowledge and power. Who ever can control and marshal resources and centralize and guard knowledge becomes wealthy.

But, the services revolution is exactly opposite. Its basis is decentralization of information, power and knowledge. Its basic premise is that we should use resources as when required without really possessing them. Look at the business models of today, especially the Internet. Its basic principle is making information equally available to everyone, and make use of resources as when required without possessing any resource at all.

But, how is this possible? Why would the wealthy people - who became wealthy because of the exact opposite principle - allow such a thing to happen? Why would they want to let go of their control and let go of their wealth?

Well, the answer is that it was not made possible by the wealthy and rich business people. Most of the computing science and the consequent business models are the creations of enlightened hippies. They were the only people capable of such subtlety of thought and action. They even managed to get vast amounts of funding from the very same people that they resolved to dethrone them from their seats of power!!

Let's look at the evolution of computing science, its absorption by the 'industry' and the unseen hand of the hippy.

On one hand, there is IBM and all its 'industrial' representatives and on the other hand there were tech-shamans.

What did IBM and its representatives contribute to the computing science? They tried big, fat main frame machines, resource hogging, hierarchical operating systems like MVS, they tried unusable programming languages like COBOL and they tried to push for data modeling rooted in their three hundred years of arrogance. Now they are trying Java and J2EE. In the software engineering side, they perpetrated process centric methodologies. All these technologies, methodologies, disciplines have one thing in common - they all aim to make the process expensive, aimed to clear the "owner's" inventory and make their clients/customer pay a very heavy price. And, more importantly - they have some kind of 'lock-in' of the customer built into all of them. In the long run - all of them are very expensive - the price paid by people who adopted COBOL during the Y2K issue must be an eye opener. And, who spent the money to resolve the Y2K bug, and who earned it? The people who bought COBOL paid the price, and the people who invented COBOL made the money. The same thing is currently happening with Java and J2EE - all in the name of open computing. The Java technology is an invention of the large computer hardware companies - their aim is to push their hardware, and consequently - the Java Technology demands more and more of 'centralized' computing power. As a language, it is doesn't maintain a clean separation between the virtual machine, the environment and the programming constructs. It looks like SmallTalk, but its philosophy is that of ADA - a big, fat language with too many features and no simplicity. And, today these are the same people who are talking about service oriented architectures and design methodologies. How will it ever work? When they talk about service - they have only one thing in their mind, the customer serving them!! One critical look at the current SOA will reveal that it is a tiger disguised as a holy cow - the SOA wants more and more resources, introduces more and more technologies, centralizes all information and knowledge in the system and very cleanly hands over the customer to the corporations. It is nothing but the mainframe machines, COBOL and process centric methodologies put together in a new disguise.

On the other hand, what was the invention of the tech-shamans of computing science? They invented workstations, powerful user interfaces, networking, distributed computing, cryptography, artificial intelligence, gaming, multi-media, Internet, Unix Operating system, C-like programming languages, hypertext, grid computing and relational data bases, agile programming, extreme programming, design centric processes and problem solving and so on. The list goes on and on. More importantly, they invented open sources programming, making freely available their ideas and designs. Is there any real programmer on earth who learnt his programming from IBM and what it stands for? Is it even possible to learn programming in such closed circuit, big brother environments? All programmers learn programming from the open sources - by contributing to open sources, by using open sources, by reading and following the open sources programming models. One thing common in all their contributions - is a clean separation of concerns, extraordinary simplicity of design, an approach to very clearly communicate the problem-solving techniques, and all these above inventions do not make it heavy on the users - they take responsibility for their own mistakes, they are all designed to tolerate errors from the environment, all of them are extraordinarily reliable and consequently very cheap to own. Neither the inventors nor their inventions charge the customer for their own mistakes. The SOA is known to this community decades ago. In fact, all the systems and technologies built by were all inherently "service-oriented". They are very economical in their resource usage, and they all freely disseminate knowledge and information. I call these people the tech-shamans and hippies of computing science.

I need to explain what I mean by Shaman and Hippy.

In general, the public image of a hippy is a person who is a drug addict and a general misfit in the society, who cannot live in the usual demands of the societal pressures. But, in reality - the hippies are people who are "free" - they are free from the usual tendency to acquire material possessions, they are free from the usual conventions of linear thought processes, and they are free from fear, greed and the usual trappings. They only desire for ecstasy - a few degenerated hippies may get this ecstasy from drugs - but the majority of them get it from problem-solving, from the joy of work. They do not demand any other form of payment. For them, money, power, fame etc., are all byproducts. Their main interest is in the 'kick' they get in understanding, the suffering they go through in the effort required to assimilate and internalize information and in problem-solving - the application of knowledge. A recent psychological survey revealed that problem-solving – working on difficult problems – releases the same chemicals that are involved in drugs – it produces a ‘kick’ very similar to drugs and sex.

Shamans are evolved ‘hippies’ – in a way hippies of the highest order. He could sit with a plant for a month, talk to it to find out whether its herbs could be eaten. Finally he may eat the herb and die – that would be his contribution to humanity. On the other hand, there are priests – the organized institutional worker - working for control and centralization as opposed to the Shaman who acts as a bridge between two worlds.

In no other field of human Endeavour, the fight between the free spirited hippy and the controlling industrialist are more marked and visible than in the computing science. The history of computing science is littered with records of such fiercely fought battles - the failure and success of Multix and UNIX, the great debate on relational and hierarchical data models, between what is ugly and beautiful and so on. The hippies had Dijkstra, Knuth, Ken Thompson, Tony Hoare, Writh, and E.F. Codd as their keepers of faith and the industrialists had large bank accounts. The history of human beings recorded rather well that it is fearlessness and truth that win the war, not power, money and ignorance. The result is well known even before the war started.

But, this time the hippies learned their lessons from history. They were no longer interested in fighting useless battles. By definition, their knowledge and inventions are available to every one - including the control minded industrialist. The evolution of computing science really nothing but a statement of how the industry assimilated the contribution of the hippies and shamans, and is slowly and painfully learning to respect them and their contributions.

Some examples and illustrations are probably required at this stage.

First, let's take Object Oriented Programming. For any normal human being, objectivity is the hardest thing to understand. Many philosophers, over the last few millenniums devoted their entire lives to understand and explain to us objectivity. The aim of science is an objective understanding and explanation of the reality - did it succeed?

Some tech shamans like Marvin Minsky and others - unleashed in the early sixties - some thing called Frames as a knowledge representation scheme, and some other shamans developed SmallTalk programming language. Even the names give their Shamanistic origins. So, object oriented programming was born. This was a very original idea - for those who understand it, it appeared very simple, and for those who did not understand it - it was an alluring mystery, whose secrets must be cracked at any cost. This division continues even till today - very few programmers really understand object oriented programming - the rest of them use UML. The object oriented analysis, modeling and the modeling languages are nothing but the industry's attempt to assimilate a very hippy concept. After all the literature, tools, systems, training courses that the industry developed for Object Oriented Programming - no one can still explain 'how to' identify and design an 'Object'. This is still the best kept secret of programming - passed on from the Master to his most accomplished disciple.

Again in the Sixties, the patron saints of tech-shamans - Dijkstra and Tony Hoare - unleashed some thing called concurrent or non-linear programming. The concept simply is that it is more efficient use of resources if programming is relieved of its tiresome linearity. How many people can think non-linearly? The shaman hardware designers went ahead and fundamentally altered the way the machine is constructed - they made the machines interrupt driven, in other words, the behavior of the machine is non-deterministic and thus non-linear. Countless coding soldiers wasted probably billions of lines of code trying to understand how to configure such a non-deterministic machine. If you look carefully at the core of the unreliable software - you can trace it to the so called 'dynamic' aspects of the environment - aspects that are not known apriori. How can you design a program for an environment that you cannot predict it at the time of writing the program? For the shamans - it is a simple problem with an obvious solution, and for a control minded industrial freak - it is unthinkable.

One thing is very obvious. More than seventy percent of the programming projects undertaken by the industry are failures - they run into too many technical and budget problems, they run into performance and reliability problems. But, name one open sources project that has any of these problems?

The industry tried to tackle this problem using their age old technique and very old, useless thinking tools. They created software engineering methodologies and they write tons of documentation of their technical designs and programs, they spend millions of dollars in testing. But, for some mysterious reason - open sources systems do not have any such things. They don't come with tons of specification documents, high level designs, pseudo code etc.

Anywhere you care to look - this division is obvious. There are two streams of computing science and two streams of people. Let me call them programmers and coders, or if you don't mind - shamans and priests. Even computing literature can be divided into two classes - the shaman literature and the priest literature. This division is very marked especially in Software Engineering.

Software Engineering is perhaps the only discipline that aims to teach problem-solving. Its interest is not in providing some specific solutions to specific problems, but to teach problem-solving. But, the big, fat books on Software Engineering written by priests do not get the central idea. They only confuse it even further – they describe methodologies and processes, basically they preach what people should be doing, without even giving a hint of how to do something.

The best software engineering books were written by Shamans – people like Dijkstra, Knuth, Weinberg, Rob Pike, and Bertrand Mayer and so on. Knuth’s 70 page paper on Errors of Tex is perhaps the best description of the software engineering process ever published. These are works that are full of insights and techniques – not some religious preaching.

But, what is the secret of the shaman?

The secret is - as all best kept secrets in the history are - is very simple and very obvious. The tech-shamans are basically builders and engineers. The metaphorical links between Free Masons and Free Software Foundation is hard to miss. They understand the process of design, they understand engineering as a discipline, and they understand that the process of making something and its final object of creation are two completely different things altogether.

The Alchemy of Design and the Sound of Music

The Design and the Designer – Part 7

In the last post, I made some observations about the process of Architecture – very briefly I stated that Architecture is the process of bringing meaning to space, and expressing the meaning in terms of certain engineering transformations, which transform the problem into an engineering problem.

The process of design – transforming ‘meaning’ into a set of engineering problems is very analogous to music composition. The difference between music and sound is the same as the difference between a sweet home and a concrete building. Therefore, let’s try and understand how music is composed. Later, we will define what engineering really is, and then I shall describe the relationship between design and engineering.

It is very simple to explain the physics of Sound. Sound is produced when two objects make contact with each other. Sound consists of a series of waves that ‘move’ the air in a particular way. The two major characteristics of Sound are its amplitude and frequency plotted on a time scale. That’s pretty much it.

What is Music? What makes certain sounds musical and certain other types of sounds non-musical? Can we say that Music is the sound we like? What do we ‘mean’ by liking? We use many different ‘esoteric’ words associated with Music, but not with Sound – words like melody, harmony, rhythm, tempo, beauty and balance.

A music composer somehow ‘transforms’ these musical attributes into a set of sound waves. The music theory build up over thousands of years provides some guide lines of how this transformation could be achieved. The music theory – whether it is western music, or Indian music theory – is a body of knowledge that codifies how music can be produced using sound. This body of knowledge can be divided into two distinct categories – the first category enables a music composer to ‘recognize’ his/her creative impulse. This is the body of knowledge that deals with recognizing music. This body of knowledge does not talk about frequencies, amplitude or rhythm, but it deals with characterizing the emotional qualities of music, and tries to categorize them.

In Indian Music – the concept of Raga for example deals with systematizing the ‘melody’ into a set of information structures. A Raga is therefore nothing but a shared understanding of melody and how to express it objectively. For those who are not familiar with Indian Classical music, here is a very brief explanation of Raga.

A Raga defines a set of melodic patterns and a melodic structure. For example, if you take a Raga like Mohanam – it is used to express the feelings of love and romance. Love and Romance are very ‘semantic’ concepts. What the Raga manages to do is to transform these concepts into a melodic structure and a set of melodic patterns. So, a composer whose creative impulse connects him with love or romance, he can then use the codified body of knowledge of Raga Mohanam, and compose music in that Raga.

Melodic Structure: This deals with the ‘rasa’ or ‘experience’ of music. There are about nine primary ‘rasas’ and another nine sub categories. There are all kinds of permutations and combinations of these primary experiences are possible, but any composition’s melodic aspects can be explained using these ‘rasas’. Apart from the Rasas, there is a ‘presentation’ aspect of the experience. The presentation is further divided into three categories: Majestic, Mellifluous, Simple (the Sanskrit Terms are Ghana Ragas, Rakti Ragas and Desi Ragas).

Melodic Patterns: The human ear is apparently very logarithmic. All music uses a concept of Octave. An octave is a predefined set of ‘notes’ or ‘frequencies’. The most frequently used octave all over the world uses 12 notes. Basically, you start from a certain base frequency, and keep increasing the frequency for every subsequent note, and by the time the octave is completed, you would have doubled the original frequency. Basically a Raga is a particular subset from an octave (7, 6 or 5 notes). It defines how to move back and forth along the octave (called Ascent and Descent). It also defines certain relationships between the notes – these are called ‘sangati’ and ‘sancharam’. In other words, a Raga defines – purely theoretically – a certain ‘musical movement’ (chalan), and also gives a set of ‘musical phrases’ that could be used to create a certain type of experience. A ‘raga’ therefore defines both the melodic structure and the melodic pattern.

The beauty of a raga is that it is just a pattern – without any ‘fixed’ compositional aspect to it. The same pattern can be used in millions of different ways – giving rise to a wide variety of compositions. A raga is thus a design pattern or a ‘semantic abstraction’. In the same way that a bird does not exist any where in the physical world, but there are millions of manifestations of the bird, a raga has no physical existence – it is just a useful abstraction.

The second category of Music Composition is closer to the characteristics of sound – and this acts a bridge between the musical characteristics and sound characteristics. For example, almost all musical traditions have a concept of an octave and beat cycles. An octave is an information structure and a very useful abstraction. (Please refer to my earlier posts for a detailed discussion on abstraction, semantic types and concepts). This body of knowledge of basically deals with the structural aspects of music composition.

Music composition therefore is very similar to design and engineering. There is an understanding of meaning and transforming that understanding into a set of structural transformations. Perhaps this is the reason why architecture is followed by structural engineering, and there is a very strong connection between architecture and structural engineering.

In the next post, we will define structural engineering, and its relationship to design and architecture.

A Disturbing question

Recently a very close friend of mine asked me a very simple question. He heard from many radical fundamentalists that there is a sanction in Islam to kill infidels – meaning the Koran specifically grants the permission to do so. He wanted to know from people who read Koran whether such a statement does indeed exist. Basically his point is very simple – how can other religions coexist with Islam if such a provision does indeed exist in Koran? Of course, this friend of mine is a very fine scholar, so he believed that such claim by fundamentalists must be totally false because no true religion would ever make such a statement.

At a spiritual, metaphysical and philosophical level, such a statement can have many inner meanings, but if it were true – how can other religions co-exist with Islam at a theological level? If every non-Muslim is considered as a non-believer, and if Islam ‘mandates’ its followers that such non-believers should either be converted or killed, how can we even have a dialogue with them?

This is a widespread belief among many people and perhaps their source of discomfort with Islam. My friend told me that someone who read Koran thoroughly must come out strongly against such a widespread belief and make a public statement, providing the relevant references. How many people would have read Koran? If you are a Hindu, or a Christian or a Buddhist – you are not expected to read the holy book of Islam. May be half the Muslims do not know Arabic and so they cannot read Koran themselves.

What use is another newspaper article in any case?

Any sane person like my friend knows intuitively that such a belief has no basis and must be false. Islam apparently means peace and the Prophet called himself the messenger of peace. Therefore, any such belief must be self contradictory.

But, sanity today is an endangered species.

Our knowledge of history comes largely from various propaganda machines and is constructed by what we read about events in the newspapers. Newspapers report only events. Can they report progress, can they report ten thousand years of human evolution, can they even make an attempt to report what a religion and culture is all about?

But, it is a fact that we live in the age of short term memories and information overload. There is just too much information these days – most of it is pure junk – traveling at the speed of light. Short of asking everyone to read Koran – is there any other sane solution to such a wide spread belief?

There is overwhelming evidence in history that kings, leaders and clergy use their power with disastrous effects. Some societies – like some of the Polynesian islands like Easter Island and the Maya Kingdoms completely destroyed themselves largely because of the attitudes of their leaders. In all such cases, Religion was invoked by the leaders and rich people to justify their own ends, and to make people firmly hold on to certain belief systems.

Religion is a very deep rooted value system, and it can be and almost always is exploited by people in power to their advantage, to promote their own vanity or as an instrument to keep people ignorant.

However, religion and culture do not live in history books or buried in historical monuments. They exist for and through the common people. If we meet an average American, we can estimate the American value system, their religion and culture in five minutes of interaction – we may not be able to write detailed analytical essays, but we understand it intuitively. What a religion and culture stands for is represented by its living monuments – the common, average people. If you interact with an Indian – even casually – you get a ‘feel’ of Indianness, if you meet a Buddhist you get a ‘feel’ for Buddhism.

The average, common people – not people in influential positions – like scholars, clergy and leaders – but just average common people – internalize their culture so deeply and radiate their value systems all the time. They do it unknowingly, without any pretence. There is no drama, no cultivated, politically correct or incorrect responses. Their responses are natural.

If you meet a Muslim autoriksha driver in Hyderabad or a Muslim carpet merchant in Bangalore, a Muslim antique dealer in Delhi – what impression do you carry about them? I believe that impression is always correct. My impression is they are very tolerant, graceful, genuinely honest and nice people. To me, that is what Islam is.

Is there a better answer than that? Can any amount of scholastic reports answer such questions better than our own first hand experiences? Why don’t we trust what we observe, what we feel about our own fellow human being? Why do we place an overwhelming reliance on the media and other propaganda machinary?

There is so much of “manufacturing of our concent” that goes on in the world today by our governments, by our leaders and by the multi-billion dollar corporate houses. They have vested interests. Therefore, in today’s world – it is much better to trust our own first hand experience. If we don't then there is a danger that we lose our own innate ability to discern, and become unwitting instruments of manipulation by the powerful.

God is one when religion is dead.

Abstraction, Pattern Recognition and Problem Solving

The Design and the Designer – Part 6

A gap of more than two weeks interrupts the flow. But, the gap is probably required because from here – the approach is very different. Until now, I largely covered a lot of breadth, and was able to give examples in one paragraph – an approach that may not be sufficient from now onwards. It is like getting into the second chapter of a book – by nature, it gets a little deeper and a bit more boring.

The difference between Architecture and Engineering is like the difference between Music and Sound. Architecture is a word that is associated with many disciplines today – we talk of architects of nations, architects of enterprises and corporations, architects of cities, plantation forests, gardens, buildings, computers and information systems. Sometimes, this word is also used in relation to effects produced by an individual or a group of people – architects of success or failure, architects of war and of destiny. After the last Great War, this discipline seemed to have become main stream.

Even though, these days we use the term in its verb form, actually it exists only as a noun. The architect produces architecture – what he does is not architecture, but design. Therefore, the process used is the process of design, but the end product is “Architecture”. All architects are designers of a particular kind.

What does really an architect do?

Let’s consider the traditional architecture of buildings as an example. Even though the structural engineers deride architects as people who basically add unnecessary embellishments that do not have serve any functional purpose, in reality the job of an architect is one of transformation much like the work of an alchemist.

What the alchemist is to science, the architect is to engineering.

His work involves understanding the essence or meaning of space and transforming this understanding into an engineering problem.

Sounds abstruse even to me. We need some detailed explanation.

An architect basically brings meaning to space. An engineer does not make any distinction between kitchen and living room – both are basically enclosed spaces as far as the engineer is concerned. But, for people living in a house, kitchen and living room have very distinct functions, how they relate to these spaces is very different, the time they spend in these spaces is very different. It is the job of the architect to understand the function of a kitchen or a living room, and their relation to the people who use them and then somehow be able to express it in terms of certain ‘spatial’ characteristics – their geometry, placement, the equipment provided in that space. Basically, the architect defines what a kitchen is in certain engineering terms.

In short, the mysterious craft of architecture is very simple to express in words. The architect performs a series of semantic transformations; the last transformation achieves the feat of converting the problem into a structural engineering problem. If the alchemist transforms lead into gold, the architect transforms gold into lead – in a manner of speaking.

The reason why structural engineers do not respect architects is perhaps because of the fact that over the years, the community of architects was successful in creating a set of transformations that are universally accepted and understood, and the engineers themselves understand the difference between the kitchen and living room without the help of an architect. So, they do not see the necessity of an architect for the most common problems like the construction of a house, or a bridge any more.

In short, architects create new abstractions. An abstraction is an information structure – that codifies meaning objectively, an information structure that is universally understood. The power of abstraction is such that – if we have to state in one single line the difference between human being and an animal – we can say that the difference is the ability to come up with an abstraction.

What is an abstraction? A “tree” is an abstraction. ‘A Tree’ does not exist anywhere in the physical world – there are mango trees, there are lemon trees, there are all kinds of other trees, but there is no such entity called ‘tree’ – it is a pure abstraction that exists only in our consciousness. It helps us to group all mango trees, all lemon trees and all other trees in one sweep, and capture the ‘essential treeness’ that all of them share. Similarly, ‘A Man’ does not exist any where except in our consciousness – there is a John, there is a Joe, there is an Adam – but where is ‘Man’?

It took several thousands of years for us to come up with some very simple abstractions that we take it for granted today. One such example is the number system and the simple arithmetic of additions and subtractions. In ancient times, when people did not know the basic math, the methods they used for trading were very funny. If one goat equals two bags of rice (purely a concept of quantity decided by its value, there was really no such thing as ‘one’ goat and ‘two’ bags of rice) and if you want to want to exchange two goats, then what do you if you do not understand what is one and what is two? You give one goat, take two bags of rice, and then give one more goat again and take another two bags of rice. A very cumbersome process indeed – isn’t it?

But, what can people do if they do not know what is one and what is two? The numbers are very powerful abstractions.

An abstraction is valid only if it is completely objective. This condition means that an abstraction must mean exactly the same thing to everyone. The difference between an abstraction and a concept is in how strictly this condition is applied. A concept is not very strict in imposing the condition of objectivity. For example, “world” is a concept – but what I mean by world and what you mean by world can be somewhat different.

Abstractions, concepts or semantic types – are all information structures. It is a unit of information that retains the meaning of something, and is made accessible to everyone.

What is interesting is how we as humans acquire concepts and abstractions. We do it so naturally – we do not even realize that we are dealing with abstractions and concepts all the time. But, it is an amazingly complex process.

When as children we learn our alphabet – all we are shown is one particular shape of the alphabet. Your teachers writes ‘a’ in your notebook, and you practice recognizing ‘a’ and writing it yourself. That’s about it – after that you can recognize ‘a’ in what ever shape, size and form it appears – and you can recognize ‘a’ instantaneously. What you as a child accomplished is an amazing feat – by understanding one particular ‘a’ – you somehow extracted the ‘a-ness’ and then can apply that understanding of ‘a-ness’ like a flash. You can recognize ‘a’ in which ever font it is written, in millions of different handwriting samples. You do not need to be taught to recognize ‘a’ by going through several thousand samples, and various complex rules of recognition.

Such a simple act of extracting the meaning by understanding from one single illustration and then apply it on the fly to recognize millions of variations is an amazingly human ability – something very difficult for machines to do. Computers cannot even recognize their own hand writing!!

It is easy for a computer to produce millions of different ways of writing – you can type a word into a document, and then change the font any number of times. Computers are quite good at such manipulations. But, they are miserable when it comes to ‘recognition’ – at least until now. Suppose you take a printout of a document – and then use a scanner to scan the same document, connected to the same computer, and then try and reproduce the document in its original form ( a process called optical character recognition) – almost always you get a document with too many ‘errors’. The OCR programs are very complex where as the document creating programs (like Microsoft Word) are relatively simple. The OCR programs use several thousand ‘sample’ character sets, and ‘train’ the computer to recognize the ‘a-ness’. But, till now there is no foolproof method to accomplish this.

What makes us so good at pattern creation, extraction and recognition? Let’s leave that question to psychologists. However, this particular ability is at the heart of problem solving. In a most generalized sense, architecture, design, engineering, diagnosis are all a form of problem solving. Problem-Solving and Design involves an innate ability to recognize and categorize patterns, abstractions and concepts. Patterns, abstractions and concepts require associative thinking.

In the earlier posts, I mentioned about two kinds of thinking, two different processes of problem solving and so on. I tried to make a distinction between demonstrative reasoning, and heuristic reasoning. It is well recognized that the art of problem solving involves heuristic methods – for example, using analogy, generalization, reduction, specialization and so on. We also described in some detail about various forms of non-linear thinking – be it creativity, right brain thinking, lateral thinking, out of box thinking and so on.

Associative thinking and heuristic/plausible reasoning are the most generalized forms of non-linear thinking and problem-solving.

How does associative thinking work really and how can we use it systematically?

Here is a small exercise:

Let’s pick up some word randomly – let’s say it is Sun. Write down ten other words that come to your mind (without thinking) when you think of Sun – for example, star, heat, light, earth, round, fire, red, day, night, energy, Egypt, Japan and so on. Now, how is Sun connected with Egypt? The relationship between these two very different concepts is one association and it is association of some meaning. For example, Egyptians used to worship Sun God. Therefore, the relationship between these two concepts is “worshippers of”.

In the last post, I discussed about four structural relationships – hierarchy, network, hypertext and set. The relationship between Sun and Egypt is none of the four structural relationships. It can – at best – be represented in terms of a hypertext relationship, but it is in fact a “semantic relationship” – it is the inherent meaning that connects these two concepts.

Nasruddin – while solving a logic puzzle uses his associative network to come up with a completely out of the box solution. We need such an associative mental network to work with analogies, to work with induction, to generalize something, and from generalization to specialization and so on.

Another advantage with the associative thinking is that – you can recall your entire memory starting at any point. I suggest doing this as a fun exercise. Start with Sun – and write down all the words that come to your mind, and then for every word connected with Sun, write down all the words you can think of. For example, expand Egypt – something like – desert, pyramids, Africa, Nile, Pharaohs, Moses and so on.

Unfortunately, we are never taught how to make and keep an associative memory consciously. We are taught how to practice organizing our knowledge using the structural relationships, and we are taught how to use demonstrative (logical) reasoning, even though our brain uses an associative structure to organize its own memory and experiences. We are naturally gifted with the ability to organize and relate to our experience using associative networks. But, we are never taught how to practice it formally. As a consequence, our memory is a very randomly formed associative, semantic network that we cannot use efficiently.

The real trick of problem solving is nothing but consciously organizing our knowledge and experiences as an inner semantic network of concepts and relationships. In other words, if we can make and keep associative networks of our knowledge and experiences, then we have a better chance of becoming better problem solvers.

More on this in the next post.

An Apology

First – my apologies to all the regular visitors of my blog. Last two weeks have been very hectic. I made a promise to myself that I would upload at least two articles every week. Promises are like spoilt children – as soon as you make a promise - somehow, it brings along situations with it and is always bent upon proving that you are not worthy of it.

There are about seventy articles that I planed to write on various topics – I even made a brief abstract about each one of them – but the actual process of writing them down demands a kind of mental and physical discipline that is very hard to command all the time. The natural laziness is always waiting round the corner to catch up with its old friend, and its company is so enjoyable, one tends to forget everything else and spend all the time with that best of friends.

There is a wonderful Sufi story on this theme, published by Idries Shah in his Seeker After Truth. Here is the gist of the story:

A dervish visited a small town in India. He used to give a talk every night, and many people would go to listen to him. The discourse would begin some after dinner and usually lasted for about an hour and half.

An old man would attend the talk everyday along with his grandson. As soon as the dervish starts to speak, the old man would go to sleep. The dervish watched this for a couple of days, and then he called the grandson to him and promised to give him one rupee if he promises to wake up the old man every time he falls asleep.

The next two days, the boy would pinch his grand father whenever the old man dozed off. But, on the third day, the dervish noticed that the old man was snoring as usual.

The dervish called the boy aside and asked him what had happened – “I thought we had a deal. Why did you not keep your grandfather awake today?”

“That’s right. But when I told my grandfather about our deal, he offered me three rupees not to wake him up”.

The first rupee is our desire to do something productive. The other three are – our habits, our natural laziness and the unobserved opposition to truth.

******

Writing is very difficult. I know what to write, it is very simple to make up the outline of the article. The difficulty starts from the time I sit down to write the actual article. As I am writing the first line, the second line has to ‘spring up’ in the mind simultaneously. Otherwise, the flow is lost. I also have to think of the actual sentence at the same time, and the mind has to recall its typing experience. All this is just too much for the mind. It prefers to drift off, my old friend is back – and we go for a smoke break after every paragraph.

Sometimes the exact opposite happens. I can’t catch up with the speed of my thought – I can’t type as fast as I think. Too many thoughts flood my mind, and I just can’t type any more. So, I take a break and have a dialogue with myself. Before I realize, five hours are gone! By that time, there is too much content piled up in the head – I get enough content for another ten articles. I end up with a perpetual backlog.

I also have another difficulty. English is still a foreign language to me. The Indian languages and English are not natural friends. They are in fact quite the opposite. All Indian Languages are free order languages - it is so easy to construct phrases in Indian Languages. In fact, most words are not single words – most of them are phrases.

English does not have that kind of flexibility. So, when most Indians write English – they tend to make up wrong kind of phrases. For example, the meaning between “I am John’s friend” and “I am a friend of John” is quite significant depending on the context. For some strange reason, I always write the sentence in the reverse order, and then I have to do a lot of editing and rewriting.

Take the previous paragraph as an illustration. An Englishman would write almost every line in the reverse order: Most Indians tend to make up wrong kind of phrases when they write English. Depending on the context, the difference between “I am John’s friend” and “I am a friend of John” could be significant.

Can you see my difficulty? The previous paragraph was my first attempt. So, I have to apply the principle of inversion even to my English!! I always look for which sentences need to be reversed to make them right!!

I have a difficulty with the usage of tense as well. I still do not get the difference between the simple past and present perfect tense. The difference between I read the article and I have read the article eludes me all the time. For some strange reason, many Indians have a preference for “have, has, had” – they prefer “I have written a letter” to “I wrote a letter”. I am sure without the help of a grammar book, most of us can’t tell the difference between the two.

So, I end up doing a lot of editing on each article. I pray for the day when I can write effortlessly.

Forgive me for the delay. I will be back tomorrow again and continue the series on Semantics and other topics.

Non-Linear Thinking and Structures

The Design and the Designer – Part 5

In the last post, I made a brief mention of Nasruddin, Heuristics and Non-Linear Thinking. I shall continue in this post on the same theme and establish the role of semantics in Design and Non-Linear Thinking, and explore some techniques of practicing non-linear thinking.

First, here is a very brief synthesis of the previous posts on this subject: In the previous posts, I applied law of 2 and tried to demonstrate the underlying unity of the theme in various disciplines – which is referred to as Plausible and Demonstrative, Creative and Logical, Synthesis and Analysis, Right Brain and Left Brain, Linear and Non-Linear, Heuristic and Rational and so forth. Largely, the conclusion that one can reach is that there are two distinctive types of thinking – which are interdependent on each other, but at the same time have very different functions and applications. We termed the functions of these two types as – recognition of problems and finding solutions to problems. We also presented a few laws of synthesis – from various fields – particularly societies, management, computing science and consciousness.

Before we get into Semantics, let’s examine linear thinking and non-linear thinking in some detail – with some examples and techniques of how to switch from one to another.

Logic – at least the way Socrates and his disciples developed it – is largely linear. This is carried over into mathematics as well. Logic mostly depends on reasoning from evidence. The early Artificial Intelligence (which is neither artificial nor intelligence) applications ran into some problems with the linear reasoning and there are now several attempts to develop non-linear reasoning systems. Such logics in AI are called “non-classical” reasoning systems. Here is a classic example of linear and non-linear reasoning – it involves a very well known logic puzzle.

Three monks and three cannibals were traveling together because of some strange circumstances – even though naturally there cannot be any friendship and trust among such a diametrically opposite groups. If at any time, the cannibals outnumber the monks – they will eat them. Now, they came to a river bank and they had to cross the river. There was only one boat. The boat can only carry two people at a time. Now, your problem is to devise a strategy to transport the monks and the cannibals safely across the river.

The usual linear reasoning system will start with River Bank-1: 3M, 3C, River Bank2: 0M, 0C as the initial state of the system, and will try to devise a series of moves that will establish the desired final state - River Bank1:- 0M, 0C, River Bank2: 3M, 3C. It is a simple puzzle – you start with something like first two cannibals will travel, one will come back, two monks will then go etc., and you can easily arrive at the final state.

Suppose you give the problem to Nasruddin, what do you think he will do? He will listen to your descriptions, and even before you have completed your narration, he will jump from his chair and shout with the excitement of a child “I know, I know - the monks will take the bridge, and the cannibals will take the boat”.

It is a perfectly valid solution to the problem – isn’t it? Then you tell him “look Nasruddin, I did not say that there was a bridge”. Nasruddin will reply, “Well – you did not say that there was no bridge either”. Now, you modify the problem description, and you will add another constraint to the problem – there are no bridges. “Well, in that case, the monks take the helicopter” will be Nasruddin’s answer. By this time, you get very frustrated and shout back at him – “I want you to solve the problem – not avoid it”. But, for Nasruddin, there was no problem there to be solved. Can you see how he eliminates the problem instead of trying to solve it?

No matter, how hard you try, you cannot contain Nasruddin. He will always think of exploiting some constraint that is not included in the problem definition and use that as a means to eliminate the problem.

The AI community in the last thirty years tried and discovered several techniques for “programming Nasruddin type behavior” into the computer programmers – with some reasonable success. I have no intention of boring you with the Non-Monotonic Reasoning systems here. But what is the technique of Nasruddin?

One, Nasruddin does not depend on “evidence” as the only way of establishing the existence of something. All formal logical systems depend on evidence as the only means of establishing “truth”. For example, if I tell you that all visitors from Canopus are fools, and you know that Mr. X is a visitor from Canopus, you can confidently establish the truth that this particular Mr. X is a fool. But, there is a catch. Suppose, if this particular Mr. X is not a visitor from Canopus, then what are you going to do? You cannot establish anything. He may or may not be a fool. You can conclude Mr. X’s foolishness, only if you know that he is a visitor from Canopus.

We need facts to recognize truth, but Nasruddin does not rely on such petty things like facts to recognize truth. According to Nasruddin, some thing does not have to be a fact for it to be true. Kurt Gödel proved this mathematically – it is known as Gödel’s incompleteness theorem. The incompleteness theorem states that in any formal system, there always exists a statement which is “known” to be true, but cannot be proved.

Rabindranath Tagore – with the brevity that a poet alone can command – says it beautifully.

“If I say that the earth is flat – you inform me that my near sight is false.
If say that the stars are the fireflies attracted to the moon, the candle of the sky – you inform me that my far sight is false. My dear logician – in your presence, I prefer to be blind”.

*****

The second principle that Nasruddin mastered is the System’s Principle – understand the problem by studying its relationship to the larger environment in which it is only but a part. I wrote about this principle in part 2 of this series of articles.

In the specific case of monks and cannibals – all Nasruddin does is to bring his knowledge of the larger environment to solve the problem. The minute you tell him about the river and the boat, he can think of a bridge. He can bring his knowledge of traveling in the air, or swim in the water. Or, he will say that the monks – using their magical techniques – will vanish and reappear on the other side. Or, he will say that the monks will use their secret herbs and put the cannibals to sleep. His possibilities to come up a solution are infinite.

The study of system level relationships requires knowledge of semantics – or the “meaning” of things. There are basically two types of relationships – structural and semantic. It is important to distinguish between these two types of relationships. Though, in logic and other sciences, we are mostly taught how to study the structural relationships, in fact, our mind works naturally very well with semantic relationships.

A few examples from various fields may be in order here.

In language, grammar defines the structural relationships. In English, a valid sentence must have a subject, a verb, an orbject/predicate. And, there are various rules that define the relationships between various parts of speech. These relationships are basically structural – they govern the structure of the sentence. We can say that water is triangular – it is a grammatically valid sentence, but completely meaningless.

In information systems there are basically four types of structural relationships. By information systems I do not mean just software applications. Any system that organizes knowledge and depends on that organization is an information system. By this definition, societies, cultures, various disciplines of study are all information systems.

The first – and simplest structural relationship is hierarchy. This relationship in information systems is called “IS-A” relationship. A “IS-A” B. This could be interpreted in several different ways:

• B is the parent of A
• B is the boss of A
• B is part of A
• B is subsumed by A
• B is the root of A

Entire systems, organizations, societies are modeled based on this one particular relationship. Most hierarchical structures use this relationship as the primary relationship. The different sections in this blog-page have a parent-child relationship with the master page. There are basically five sub-sections – header, footer, posts, author information, archives – each of them is a “child” of the main blog page. Similarly, many corporations have a hierarchical structure – with the owner of the corporation at the top. The “semantics” of this relationship connotes ownership and protection.

The second structural relationship is network relationship. In such a structure, elements, or objects at the same level can be connected. This is called “sibling” relationship. A “IS-A brother of” B. The relationship basically means that two things are at the same level. It connotes competition and/or collobaration.

There are two other types of structural relationships in information systems hyper-text and groups. Both these relationships are somewhat non-linear relationships, but nonetheless, they are structural relationships.

The Set, and Hyper-Text are more complex structural relationships than a simple hierarchy and a Network. A set uses a function to create a structure. For example, you can say that all integers belong to a set. You can then define a function that determines an integer and therefore, the membership to the set. The members in the set may not have any particular relationship – they have something in common – that’s all. This is how most social groups are formed. The function gives an identity to the set and therefore to all its members. All people “born” in US are “Americans”. “Being born” in the US determines the identity of the set called Americans.

A Hyper-Text relationship is much more interesting relationships. It does not relate two nodes, but instead it relates content from one node to another node. All web pages use this structure extensively. For example, I can provide a link to an article in wikipedia in one of my articles in this blog. This does not mean that this blog as a whole and wikipedia share any relationship; it also does not mean that this article and the wikipedia share any relationship, and it also does not mean that this article as a whole has any relationship to that article in wikipedia. This relationship connotes friendship. There is no function, there is no responsibility, and there is no ownership in this relationship.

Using these four relationships, we can explain almost all “order” in our world. We defined the basic family relationships (parent-child, sibling), groups and friendships.

So, where is the place for Semantic Relationships?

We conveniently left out the most important, most complex, wonderful, problematic and the most beautiful relationship of all– that of the lover and the beloved. None of the above relationships explain that relationship – do they? What is the relationship of a husband and wife? What is the relationship of a teacher and a student? Friendship comes closest, but these relationships transcend friendship.

This is the realm of semantics – which is the topic of next post.

Non-Linear Thinking, Nasruddin and Polya

The Design and the Designer – Part 4

In these series of articles on Creativity and Design, my intention is to explore if there is a “formal” model of creativity. I am not interested in the psychological aspects of creativity, how it works and so on. The Psychology of discovery and invention is wonderfully described by Mihaly Csikszentmihalyi in his classic book called Creativity.

My interest is to discover a method of consciously practicing it. In the last few posts, I tried to provide a basic framework that I am working on. Basically I am trying to bring together all my experience under one unified theme. Design is the name I gave it.

I studied mathematics and more importantly I studied how to ‘do’ mathematics. I studied computer science and information systems – I designed some very large and complex software systems. I studied design theory – product design, appliance design, communication design, aesthetics, user interfaces, ergonomics etc. I studied philosophy – Western Philosophy, Indian Philosophy and Sufi Philosophy. I studied systems thinking. Basically I am a problem-solver. Given any problem, I can come up with some kind of a solution. I met many people who are fantastic problem solvers. Many of these people can almost instantaneously identify a line of attack and can come up with a solution almost immediately.

In my experience, problem-solving can be taught and can be learned – even though we generally think some people are gifted with this ability. But, teaching problem-solving has been one of the toughest problems for the teaching community. There does not seem to be a discipline for teaching problem-solving. Even the most beautiful subject – mathematics – has not addressed this problem.

How are we taught mathematics? The mathematics teacher presents the proof of a theorem in a step by step manner. First, he defines the theorem, and he has the proof in front of him, and all the teacher does is to explain the ‘logical flow’ of the proof. In the course of several years, various problems and solutions are demonstrated as examples, and somehow the student is expected to ‘understand’ problem-solving from these examples. There is no conscious attempt at teaching problem-solving.

The ‘doing’ of mathematics is not taught. This is what makes Mathematics difficult to learn for many people. The best reference works ever written about this are Polya’s books on mathematical method: How to Solve It and Mathematics and Plausible Reasoning. For some strange reason – these books are not part of the mathematics curriculum.

Polya makes a very useful observation. According to him – rightly so – the process of problem solving is a heuristic process. The difficult part of problem solving is to ‘recognize the problem’ correctly. This recognition is not a “rational”, “logical” process. It is very irrational – if you are good at it – you can immediately recognize what the problem really is in an instant, and you decide on the line of attack. How do we normally recognize the problem? We generally use a set of heuristics. For example, you may recognize that the problem is “similar” to another problem you solved before, or you may recognize that the problem belongs to a class of problems that are already solved and so on. These are all heuristics.

Here is an illustration of how heuristics are applied. Archimedes first discovered the formula for calculating the area of the circle. The area of the circle is PI*r2. In Mathematics texts, the proof is presented using some complex coordinate geometric equations. Archimedes proof is rather very simple and can be explained in one line. He drew several lines from the center of the circle to its circumference. Now, the area enclosed by two such lines and the arc of the circle looks like a triangle. The area of the triangle is (base*height)/2. Now, the height in this case is r. And, the base of all the triangles put together is the circumference of the circle – which is 2*PI*r. Therefore, the area of the circle is PI*r2. Beautiful – isn’t it?

A mathematician will not agree to a proof like this – because, the way Archimedes divided the circle into different regions are not exact triangles – they only look like a triangle. Archimedes used a heuristic.

Now, let’s look at the process that Archimedes used to solve this problem. First, for Archimedes, the problem of calculating the area of a circle is a real problem; it is not a theoretical problem. He had to calculate the area of agricultural land for calculating taxes. Second, given a particular circle, he knew how to come up with answer. So, he has some data in hand. He has the areas of different circles in front of him and he was looking for a common “pattern” that explains all the answers. Third, he knew that the two fundamental properties of the circle are its radius and its circumference. The last step is the important one. He asked himself a question – can I formulate this problem in terms of problems that I already know how to solve? In other words, he knew how to calculate the area of a square, a rectangle and a triangle. Now, he is trying to “reduce” this problem to the problem of a square, a rectangle or a triangle. He also knew another important fact about areas – they are always expressed as a product of the two different lengths. He used four different heuristic techniques – generalization, analogy, reduction and induction.

Polya calls this type of reasoning “plausible” reasoning as opposed to “demonstrative reasoning”. Different disciplines may have different names for this. Psychologists call it right brain thinking, designers call it creativity at work, scientist call it intuition at work, systems people call it “systems thinking”. In popular literature, this is called “non-linear thinking”, “out of box thinking” and by many other similar names.

No matter what name you give it – the process of recognizing the problem, and the process of actually solving it involve two different kinds of thinking altogether.

Polya’s books are the best reference works on this subject. He tried to formulate a systematic method of “problem-recognition”. Unfortunately – thanks largely due to the American experiments with modern mathematics – the world today suffers from cultural isolation of mathematics. Polya’s achievement is remarkable. With the precision of a mathematician, he formulated a “dictionary of heuristic”. This means – if you have some patience – you can learn the heuristic thinking like learning the vocabulary from a dictionary.

Another dictionary of heuristic that is particularly enjoyable to read is the collection of Nasruddin Stories. These stories help us to break our usual linear and cause-effect thinking. Even just remembering the stories has the remarkable effect of constructing a different kind of associative memory. The stories ‘just’ pop up by themselves when we need them most.

More than 700 tales of Nasruddin are collected by Idries Shah and are published by Octagon Press. Here is one story of how Nasruddin describes the “two modes of thinking” that we have been talking about:

One day Nasruddin stormed into the tea house and announced with lot of excitement that he discovered a very important truth.

People asked him what it is.

Nasruddin said – “The moon is more useful than the Sun”

Everyone was taken aback, and asked him why he thinks so.

“Because we have more need for light in the night”

Nasruddin Stories make sense at many levels, and it is said that there are at least seven interpretations for each story. Here is one interpretation that is relevant in our context. Moon is the agent of synthesis. He synthesizes the sun light and reflects it back to earth. Sun is the source of light – when he is there – we have no need for light because it there everywhere. Your need is more when something is not available – right?

There are people who are experts at synthesis. Polya synthesized mathematical method, Will Durant synthesized history, Ackoff synthesized management, and Christopher Alexander synthesized Architecture. I am trying to synthesize Design. In order to synthesize a domain, you need to be a super-specialist in that domain and you need to be able to absorb the entire domain completely.

Both modes of thinking co-exist and compliment each other. I am not indicating that synthesis is more “important” or “superior” to analysis. Like the Sun and Moon, they are interdependent on each other. The linear and the non-linear, the creative and the logical, the plausible and the demonstrative co-exist together.

In order to synthesize Design – we need an understanding of Semantics. We have to understand the meaning of things, and how different things relate to each other.

This will be the topic of the next post in this series of articles.

****

Nasruddin on cause and effect:

One day Nasruddin was waling along a street with his students. As Nasruddin was walking past a two story building, a man fell down from the first floor – and he fell on Nasruddin. Nasruddin’s neck was badly hurt.

His students asked him what lesson they can draw from this incident.

Nasruddin was in pain and he was now angry at such stupid questions. He shouted – “fools – can’t you see? He falls from the building, and it is my neck that is broken”.