Now, the canopus archives blog is moved to my own domain. The new blog url is:
http://canopusconsulting.com/canopusarchives
The new blog has a completely new interface - which is more suited to the long posts that are a norm of this blog.
Please visit the new blog and leave your comments.
And, for those of my good friends who provided a link to this blog from theirs, may I request them to update their links please?
Wednesday, April 11, 2007
Monday, March 12, 2007
The Dark side of the Internet
This article is about security and privacy issues that the new generation of Internet poses to individuals, groups and societies at large. For some strange reason, the power, benefits and the positive aspects of the Internet are very widely known, but there seem to be very little public awareness - especially among the average users of the Internet - about some of the dark elements and inherent risks in making all information publicly available on the Internet. The Internet itself has changed drastically over the last few years, and it is evolving very rapidly. There is a tremendous difference between the first generation Internet and the new semantic web which is evolving - both in terms of its power, and the risks it poses if we are not careful of how we use it.
I will first introduce very briefly the three different phases of the Internet in the first section of this article, and later discuss the various privacy and security issues connected with these different phases.
Web 1.0: Web 1.0 was the first generation of the Web. During this phase the focus was primarily on building the Web, making it accessible, and commercializing it for the first time. Key areas of interest centered on protocols such as HTTP, open standard markup languages such as HTML and XML, Internet access through ISPs, the first Web browsers, Web development platforms and tools, Web-centric software languages such as Java and Javascript, the creation of Web sites, the commercialization of the Web and Web business models, and the growth of key portals on the Web.
Web 2.0: According to the Wikipedia, "Web 2.0, a phrase coined by O'Reilly Media in 2004, refers to a supposed second generation of Internet-based services — such as social networking sites, wikis, communication tools, and folksonomies — that emphasize online collaboration and sharing among users."
I would also add to this definition another trend that has been a major factor in Web 2.0 — the emergence of the mobile Internet and mobile devices (including camera phones) as a major new platform driving the adoption and growth of the Web, particularly outside of the United States.
Web 3.0: Using the same pattern as the above Wikipedia definition, Web 3.0 could be defined as: "Web 3.0, a phrase coined by John Markoff of the New York Times in 2006, refers to a supposed third generation of Internet-based services that collectively comprise what might be called 'the intelligent Web' — such as those using semantic web, microformats, natural language search, data-mining, machine learning, recommendation agents, and artificial intelligence technologies — which emphasize machine-facilitated understanding of information in order to provide a more productive and intuitive user experience."
The key here - in the Web3.0 - is the "machine-facilitated understanding of information". The word "semantics" means understanding the essence of something. In Web3.0, this understanding is accomplished by the machines. This is a tremendous advantage as well as potentially very dangerous and very risky - because, ultimately machines are controlled and operated by humans.
The risk comes from the following:
• Our conversations are no longer ephemeral
• No individual is an isolated dot any more – we are all part of a connected scheme.
• Machine facilitated understanding of information (or Semantic Modeling) makes it possible to discover information and relationships that were previously impossible.
• The Internet allows us to drop our inhibitions - and many people "upload" very personal information about themselves, their lives, their families and friends. This information can potentially be used to generate a complete psychological profile of people. Such information can be potentially be misused by governments, by corporations, by mafia and other dark elements.
• You may buy your computer – but you don’t own it any more. Spyware, Malware, Corporate, governmental espionage
Our Conversations are not ephemeral
Imagine you are traveling in a taxi, a bus or a train with your spouse or a colleague or a friend. Generally, in such situations we talk, and we talk about personal and intimate subjects - forgetting that people around us can hear our conversations. You may probably discuss about a weekend vacation, or a travel plan with your partner. When you come back home after your vacation - you discover that your house was robbed over the weekend.
These things have happened in the past, and they still happen. People talk very private and intimate things in public and sometimes they face unpleasant consequences of such 'inadvertent' public conversations. But, in general, in the past - everyday conversations - whether face-to-face, conversations in public - evaporated as soon as they were said. You could be reasonably sure - that your talk simply vanished in the ether, no trace was left. Of course, organized crime bosses in Movies used to worry about their phones being tapped - but that was an exception. Privacy was the default assumption.
Well, this is no longer true. Our conversations on the Internet - emails we send, our messages on social networking sites like Orkut, Myspaces, the comments we leave on our friends' blogs, the blogs we write, the chat programs we use - all of them are stored, analyzed and recorded. They are stored permanently - for ever. Nothing can be deleted from the Internet - at least not by us.
Every single thing you type on your keyboard - is stored. We know this intellectually, but we never internalized it properly. There are tools that allow both businesses and government agencies to monitor and log IM conversations. E-mail can be saved by your ISP or by the IT department in your corporation. Gmail, for example, saves everything, even if you delete it. More over, these conversations are saved by many different organizations - not just one. Your employer may be storing everything you type on the net, your ISP may be storing everything you type, governments and defense organizations may be storing everything you type. We don't even know who else may be storing our conversations and our messages. We don't know who they are, and we certainly do not know what they do with the information.
There are no laws that regulate who can 'tap' into the information that we send over the Internet. Phone tapping is illegal in most countries - even if the police want to tap some one's phone, they still need a legal order from the magistrate. But, on the Internet, anyone can store anything. There is simply no law and no legislation.
The implications of this to personal safety, liberty and freedom are enormous. You may lose your job - because your employer finds out that you are spending too much time on the Internet and they can now 'prove' it. There may be criminal prosecutions that one has to face, divorce proceedings or simply one may have to face some unpleasant public embarrassment. Such things have happened already - former U.S. Rep. Mark Foley sent salacious instant messages to a young boy and found himself arrested very soon. IBM fired an employee because he was found to be chatting in internet chat rooms. Many people were bullied, kidnapped and robbed because of the information they keep on the internet.
If you find this disturbing, you should. Fewer conversations are ephemeral, and we're losing control over the data. We trust our ISPs, employers and cellphone companies with our privacy, but again and again they've proven they can't be trusted. Identity thieves routinely gain access to these repositories of our information. Paris Hilton and other celebrities have been the victims of hackers breaking into their cellphone providers' networks.
If you don't find this disturbing - read on. Ask yourself what information about you is already on the Internet, and how much information you have kept already. Do you have a blog? Do you have an email-id? Do you leave comments on other's blogs? Do you have your photograph on the Internet somewhere? Do you have membership in some news groups? Do you have an account in social network sites like Orkut? Do your friends write public 'scrapes' in your Orkut profile? Do you regularly chat on the Internet? Do you have am online access to your bank account? Do you buy anything on Internet using your credit card? That's a lot of data that you are putting on Internet, and most of this information is very personal and can be misused.
Not convinced yet? Here are some examples:
I have a blog in my local language - Telugu. The number of Telugu bloggers are still very few - by the Internet standards they don't even count. Because they are few, they are all friends and they know each other very well. Some of these bloggers have 'pseudo-names' and they don't keep their photographs and any other information on their profile pages. But, it is very easy to find out who they are, where they live and what companies they work for. For example, if they visit by blog - by looking at the IP-address records, I can figure out where they are from, where they live and their ISP or company name. Well, the IP-statistics still do not tell you the name of the person. But, suppose if they leave a comment on my blog - then it is very easy to correlate the IP-statistics with the person. I still do not know their name (if they choose a pseudo-name), but I know where they live, and which companies they work for. Once you figured out their IP-address, you can then generate lot of statistics about them - like their usual 'Internet Visiting Hours', whether they browse from home or from work, how long they usually spend time on the Internet, what are their favorite blogs are and so on. It takes only a couple of minutes for an average human being who does not posses any powerful tools like governments, crime syndicates, corporations and defense departments. Imagine how much information about you can be learned by those 'control' minded groups?
"You are innocent until proven guilty" was the old maxim - with the new 'security madness' by governments all over the world - you are guilty until proven innocent is the new heuristic.
The moral is clear: If you type it and send it, prepare to explain it in public later.
This was web1.0. Now, enter web2.0:
No individual is an isolated dot any more – we are all part of a connected scheme
I had a friend - we lost touch with each other for a long time, almost 15 years. I don't know much about her personal details - since we were mostly pen friends. All I knew about her was her name, her family name, her brother's name and her husband's name - and where they were from. I don't even know where all of them are living now.
Since she was a very good friend of mine - I tried several times to find out where she is and how she is, but couldn't locate her - until the social networking sites like Orkut came. Let's say her brother's name is Suresh, and her husband's name is Adarsh. I searched for Suresh in Orkut, and his 'native city' together. I find quite a few people by that name. How do I know whether this particular Suresh I was looking for is one of them or not? All I do is look whether any of these people have a friend called Adarsh. Bingo -- they are there. It took ten minutes to locate them. Rest is easy - you make a contact, get the phone number and call her.
I called her - after the initial joy and excitement of meeting an old friend was over, she suddenly remembered that it would be impossible for me to find her phone number. So, she asked how I got her numbers.
I told her that I found it on the Internet. She was surprised. I told her that I also happen to know her residence address, what her husband is doing, that they recently went for a vacation to Chennai, her in-laws live in such and such a place, that she had a five year old son and that the boy had some health problems recently, that she is doing a part time job and that they bought a computer recently and so on and so forth. BTW, she never touched a computer in her life, and she doesn't even have an email address.
She was totally dumbfounded and flabbergasted. She asked whether I work for CBI. Well, I don't work any intelligence agency. All the information was there on Orkut - for anyone to see. It is surprising how much information we keep unintentionally on the net.
The point here is something even more serious. Today, our presence on the net is not isolated - we are - by our own volition - connected to almost everyone else in the world. We expose our friends, colleagues, associates, classmates, relatives, family and everyone else we know. We don’t stop with that - we also expose how we came to know each of them, we tell the entire world what our interests are, which causes motivate us. How much more can you advertise about yourself to the world?
If her husband and brother were not linked together on the net, it would have been very difficult for me to locate them. It is the 'connectedness' that allows even an average person to quickly find out so much information - even about someone who does not have any internet presence.
We have several such social networking sites - there are social networking sites like Orkut, there are professional networking sites like Linked-IN, there are specialty networking sites and so on. We have a presence in each of them. And, we are connected with so many of our friends and colleagues through these networks. When I log on to Orkut - I get a message right at the top - "you are connected to 39million people through 40 friends". The question is do I want to be connected to 39 million people?
There is an interesting application developed by University of Virginia called Oracle of Bacon. This application can connect any movie actor from any country - past or present to the Hollywood actor Kevin Bacon with in 8 steps. It can connect somebody like S.V.Rangarao with Kevin Bacon. Surprised? How can S.V. Ranga Rao - a telugu actor who acted mostly in telugu movies be connected to Kevin Bacon? Well, he is - and he is connected with in 6 steps.
There is a theory that anyone in the world can be connected to anyone else in the world with in a few steps. This is called 'semantic' distance between two people. Using Orkut and other social networking sites - you are the Kevin Bacon. You can be connected to anyone in the world. What is our distance to the most wanted man in the world? May be six or seven? We don't know - all we know is we are connected. Suppose, if some idiot from a government agency decides to 'round-up' everyone who is connected with the infamous B.L of A.Q fame - we might be on the list. All because, you have some forty friends on Orkut and you don't know most of them.
Is this information available to governments, mafia, drug-traffickers, terrorists, extortion agents, thieves and all other dark forces? Sure it is. Do they have more powerful tools to analyze this information? Sure - they do. Can they and will they misuse this information - sure they can, they will and they do.
Recently FBI issued a parental warning to monitor the internet usage of their children. The reason - many children are lured by sex workers and mafia using the social networking sites.
Recently a young boy killed himself because he was cyber-bullied and he couldn't take it any more.
This is Web2.0. It only gets worse with Web3.0.
Semantic Modeling, psychological profiling, spheare pshing
Earlier, I stated that "understanding" and "extracting" meaning out of information can now be done by machines. Let me explain this in layman terms. Most of you may have read a book called 'Blink'. In the first chapter of this book, the author explains how a trained psychologist can listen to five minutes of everyday conversation between a couple, and figure out very accurately whether marriage lasts, and if does - for how long.
When we meet a person, spend time with them - we generally know something about that person. All of us do some extent of 'psychological profiling'. This is a normal human trait. We do it for our self defense, to know more about the other person, to make friends with them, to understand how to conduct with other person, to gain some favors and so on.
But, it is in general impossible for a human being to remember the entire conversation the other person had with everyone in the world, and analyze it later at leisure. Most of our 'estimate' of a person happens in a split-second, it is - in general - not a planned, conscious activity. Suppose, all our talk along with a lot of information about ourselves is available electronically - and suppose machines are capable of 'analyzing' this information and draw up a psychological profile of this person - isn't that a very powerful information to possess?
Recently, Amit Seth and others published a very interesting paper called "Semantic Analysis of Social Networks - Experiences in addressing the problem of conflict of interest detection". Interestingly, the project was sponsored by ARPA - which is the US defense department's advanced research agency. The researchers used information publicly available about authors of scientific papers from two different sources, and using the semantic modeling techniques they developed, they discovered that many authors - to put it in layman terms - cheat to get their papers published. Suppose there are two authors called X and Y. Let's say that X and Y worked together in some research institution in the past and published a joint paper together. Later, X left the institution and joined some other organization and Y continues to work for the same institution. Now, when X sends a paper for publication, Y acts as the reviewer and accepts the paper for publication. This is called conflict of interest - because people who know each other are not supposed to be reviewers of each other.
Earlier, finding out such information was practically impossible. How does one journal know that Y and X know each other? But, with social network sites, and professional network sites - such information can be very easily detected by computers today. Now, this is a harmless, useful research paper. But, what are its potential applications?
A super Google can be easily developed which can 'extract' very meaningful and powerful information about people and their lives. If there was ever any complaint was filed against you - that information can be 'fed' into such an application. It can easily discover what is your financial net worth by analyzing your online purchase history, or your online bank account transactions. It can easily generate a psychological profile of yourself from your online information. It can easily draw inferences about a person of various kinds - whether a person is anti-government, he/she is likely to be a political activist, whether they can be easily be made to submit themselves to some kind of brain washing and so on.
If the Internet adopts the RDF standard which is pending - developing such super 'Google' becomes a relatively easy task. Till now, such programs required very expensive and specialized computing models. But, with all 'information' and 'meaning' available simply as part of the attribute of an object - we no longer require any complex computational models to generate inferences, it can be done by any normal database query. This means that anyone who has access to the data - everyone - can easily develop applications to 'discover' information about people.
Implications of RDF and Semantic Databases
After the 9/11, there was a massive initiatives by the US government and several other governments to collect massive amounts of centralized data about people and store it in one central place. The Bush Administration started an initiative called Total Information Awareness - which was later scrapped because of the unprecedented public outcry. This TIA act gave absolute powers to the security agenecies to collect any information about anyone. Though this 'act' was cancelled - it is now part of the defence department's unclassified projects. The name changed, but the work continues.
With RDF, there is no need to develop any centralized databases anymore. Because, every object on the Web is capable of answering some interesting questions about itself - one can simply get the information from multiple sites and run any query on a distributed database.
If this sounds too technical - here is an example:
These days, a typical Internet usage by an average user can be summarized as:
1. An Email-Id with webbased email services like Google, Yahoo etc.
2. Probably and official or corporate Email-Id that is used for business and professional communication
3. Several Login-Ids and some activity at several e-commerce sites - like Amazon.Com, e-Bay
4. Some history of online purchases using credit cards - airline tickets, music and books, groceries
5. A login-id with at least one social networking site like Orkut, Linked-IN, Myspaces.com and so on
6. Probably a blog and a personal website.
Add to this, lot more online data that is created without our knowledge and permission - if you traveled in an Airplane - your travel history is recorded by that Airline company. If you visited some countries either as a tourist or on work - your entire travel history is recorded by many governments, travel companies and so on. If you booked your hotel reservations online - this history is recorded by various organizations. If you passed through various security zones at several places - your photographs and videos have been captured and stored.
With Semantic-Web, all this information - can be potentially linked together without any significant effort. There is no need to centralize all this various information sources into one centralized database anymore. This is the power of semantic databases. With the advances in multi-media technologies - searching through multi-media objects like image search, video search, voice search will also be possible quite soon. The technology has not yet reached this stage of evolution - but it will soon get there.
This basically means that we leave an enormous amount of electronic audit trail of ourselves, some of it willingly created by us, and most of it created without our knowledge.
How can such information be misused? One example is spear phishing. Here is an excerpt from an article published in Newsweek International called "A Dangerous Game of Phishing":
"Spear phishers gather information, usually on the Internet, about an individual, and then craft a personalized e-mail more likely to dupe the mark. According to the FBI, the personalization method has proved so profitable that a significant number of spear phishers, principally located outside the United States, began applying it to death-threat extortion e-mails for the first time last December. FBI spokeswoman Cathy Milhoan says the problem is "huge."
Here's how it works: A spear phisher collects information on an (often wealthy) individual, then writes a chilling e-mail. The sender, posing as a hit man, offers to spare the recipient in exchange for a large sum of money. If the ploy doesn't work, the target receives a second e-mail, purportedly from the police, explaining that his or her name and address were found on a recently arrested murder suspect. "The victim gets scared, gets paranoid, he gets a lot of things," says Alan Paller, a cybercrime expert with the Bethesda, Maryland, SANS Institute who has testified before the U.S. Congress on the matter. The target provides personal details—including financial data—to aid the investigation.
Traditional extortion often involves tailing targets and staking out their homes to obtain the particulars—such as the appearance of a victim's daughter—that render threats credible. Today much of that information is easily gleaned from the 'Net. Dan Vogel, an Edmond, Oklahoma, former FBI profiler, says social-networking Web sites such as MySpace are "fueling" the trend."
In an article entitled "Governments research to track online networking", Christopher Dela Cruz and Megan Carr say the following:
"The Department of Homeland Security is paying Rutgers $3 million to oversee development of computing methods that could monitor suspicious social networks and opinions found in news stories, Web blogs and other Web information to identify indicators of potential terrorist activity.
The software and algorithms could rapidly detect social networks among groups by identifying who is talking to whom on public blogs and message boards, researchers said. Computers could ideally pick out entities trying to conceal themselves under different aliases.
It would also be able to sift through massive amounts of text and decipher opinions - such as anti-American sentiment - that would otherwise be difficult to do manually.
The program is designed to sift rapidly through huge amounts of data. It has also been described as a sort of "Super Google" researchers such as Eduard Hovy at The University of Southern California, to explain the scope and quickness of the technology."
It is definitely scary. I don't even know what is possible with Web4.0.
You bought your computer, but you don't own it any more
If the above article sounds like one big conspiracy theory written by a paranoid individual - think twice. How many different entities are vying to control your computer? You bought your computer - but do you really "own" it?
In the words of internet security expert, Bruce Schneier - there's a battle raging on your computer right now -- one that pits you against worms and viruses, Trojans, spyware, automatic update features and digital rights management technologies. It's the battle to determine who owns your computer. Malicious software like spyware, malware, trojans, worms and viruses are basically agents that somehow slip into your computer and "serve" the interests of someone else - they steal your passwords, make fraudulent bank transactions, collect data about yourself and your internet usage, send spam emails using your email-id and so on. Estimates area that there are some millions of computers that are part of this "bot" network.
But, now things are not that simple. There are attempts by "legitimate" software programs - by many different media and other companies that determine what you can and cannot do with your computer. Recently, Sony released a "rootkit" program that attempted to block the user from doing something with Sony's music that Sony "considered" illegitimate. There are automatic update programs and digital rights management software that control what you can do and cannot do. They collect a lot of data about your machine and what you do with the machine that you probably do not even know. Here are some examples:
• Entertainment software: In October 2005, Sony had distributed a rootkit with several music CDs. This rootkit installed itself without the knowledge of the user. It simply installs itself when you play the music CD. Its purpose was to prevent people from doing things with the music that Sony didn't approve of: It was a DRM system. It would have been a "virus" if it were installed by a hacker. But Sony believed that it had legitimate reasons for wanting to own its customers’ machines. Interestingly, most commercial antivirus programs did not detect Sony's rootkit - simply because Sony asked them not to. Who are they serving? You? Or someone else?
• Application software: Internet Explorer users might have expected the program to incorporate easy-to-use cookie handling and pop-up blockers. After all, other browsers do, and users have found them useful in defending against internet annoyances. But Microsoft isn't just selling software to you; it sells internet advertising as well. It isn't in the company's best interest to offer users features that would adversely affect its business partners.
• Spyware: Spyware is nothing but someone else trying to own your computer. These programs eavesdrop on your behavior and report back to their real owners -- sometimes without your knowledge or consent -- about your behavior.
• Internet security: It recently came out that the firewall in Microsoft Vista will ship with half its protections turned off. Microsoft claims that large enterprise users demanded this default configuration, but that makes no sense. It's far more likely that Microsoft just doesn't want adware -- and DRM spyware -- blocked by default.
• Update: Automatic update features are another way software companies try to own your computer. While they can be useful for improving security, they also require you to trust your software vendor not to disable your computer for nonpayment, breach of contract or other presumed infractions.
So, I am not so paranoid after all - and most of it is not a conspiracy theory. We may not recognize it, but the world is fast becoming a gaint airport security area - where every single movement is recorded and everyone is suspected.
The Net and the Web - these two words always had a ring of "conspiracy and control" around them, may be they are not a coincidence after all.
Conclusion
What can we do? We cannot turn the clock back. Paraphrasing Oscar Wilde - we cannot live with the Internet and we cannot live without it. There is lot of debate by different people, groups, experts on whether the Internet should be controlled. My own view is it cannot be controlled. There is no legislation governing what can happen on Internet, and there is no possibility that such a legislation would ever emerge. As individuals - historically and culturally - we always thought that our governments and societies will take care of our security. With the Internet, that is not possible. We created the Internet to empower the individual and it did that exceedingly well. There are always individuals and groups who intend to misuse that power. It cannot be stopped.
There is very little public awareness about the dangers of power-shift into individual hands. Hundreds of years ago the power and control rested in the hands of the kings and aristocrats who supposedly took care of their subjects. With the Industrial revolution, the power and control shifted into the hands of the corporations, businesses and media companies. Now, the power is shifting into the hands of individuals. On one hand, this is a positive development and on the other hand - this has implications on our lives that we haven't even begun to assimilate.
Most of these privacy and security articles - sound like conspiracy theories - so people do not believe in them. It is very difficult to get factual support - because by definition it is a dark-side, underworld activity - it is impossible to get scientific data. Therefore, we do not see any 'publications' about these issues in any reputed scientific journals. So, a debate between the two sides - those who argue that the Internet should be left as a self regulating mechanism and those who argue for some kind of legislation and privacy control is not possible.
There is another factors that worries me personally. As human beings - we are genetically engineered to absorb information and train ourselves at a particular pace and speed. It takes several generations for us to 'internalize' certain information. For example, we all know how to take care of our security in a society - it has been internalized into us for hundreds of years. But, the Internet evolution is too fast for us to internalize. How do you retrain your "sub-conscious" in a matter of a month? When you login to your social networking site - in order to be careful of what information you put in there - you need to develop "automatic" safe-guards, a king of sub-conscious filter similar to the one that prompts you check whether you locked your door or not. This takes a lot of time - and we don't get that kind of time.
But, we have to retrain ourselves and conscious structures to take care of ourselves. We first need to educate ourselves that in this age - information is valuable and anything that has value carries certain risk with it. We have to have some idea of this risk and learn to act accordingly. I do not see any other possibility. My advise - do not bring your living rooms and bedrooms onto the Internet. Do not expose your friends, colleagues if there is no "specific" value you obtain in doing so. Be careful of expressing yourself and your opinions on the Internet, on blogs and other places. More importantly - it is our children who are at an immense risk, and we have to educate them about the risks of Internet - as we educate them about safe sex. Sex by itself is not dangerous - but, there are certain limits and a moral code that has to be followed. Same is the case with the Internet.
References and Further Reading:
Bruce Schneier’s Security Blog: Bruce Schneier is one of the world's most respected security and privacy expert. He writes some very insightful articles about these issues in his blog. A must read for everyone.
Kingsley Dennis' blog: Kingsley is a post doctorial research scholar at University of Lancaster, UK. His blog is one of the most interesting blogs on Internet, sociology, security and privacy issues, mobilities, mind-control and many other topics.
Global Guerrillas: Very interesting site and insightful articles.
Information War: The name says it all.
Spy Blog: Don't miss this one!!
Information warfare: U.S. Department of Defense's policy documents on Information Warfare.
Surveillance Issues: We live in a world of surveillance. It is amazing how many such technologies and devices surround us.
Electronic Frontier Foundation: An organization that works on several issues related to privacy, security and other matters. A must visit for all bloggers.
I will first introduce very briefly the three different phases of the Internet in the first section of this article, and later discuss the various privacy and security issues connected with these different phases.
Web 1.0: Web 1.0 was the first generation of the Web. During this phase the focus was primarily on building the Web, making it accessible, and commercializing it for the first time. Key areas of interest centered on protocols such as HTTP, open standard markup languages such as HTML and XML, Internet access through ISPs, the first Web browsers, Web development platforms and tools, Web-centric software languages such as Java and Javascript, the creation of Web sites, the commercialization of the Web and Web business models, and the growth of key portals on the Web.
Web 2.0: According to the Wikipedia, "Web 2.0, a phrase coined by O'Reilly Media in 2004, refers to a supposed second generation of Internet-based services — such as social networking sites, wikis, communication tools, and folksonomies — that emphasize online collaboration and sharing among users."
I would also add to this definition another trend that has been a major factor in Web 2.0 — the emergence of the mobile Internet and mobile devices (including camera phones) as a major new platform driving the adoption and growth of the Web, particularly outside of the United States.
Web 3.0: Using the same pattern as the above Wikipedia definition, Web 3.0 could be defined as: "Web 3.0, a phrase coined by John Markoff of the New York Times in 2006, refers to a supposed third generation of Internet-based services that collectively comprise what might be called 'the intelligent Web' — such as those using semantic web, microformats, natural language search, data-mining, machine learning, recommendation agents, and artificial intelligence technologies — which emphasize machine-facilitated understanding of information in order to provide a more productive and intuitive user experience."
The key here - in the Web3.0 - is the "machine-facilitated understanding of information". The word "semantics" means understanding the essence of something. In Web3.0, this understanding is accomplished by the machines. This is a tremendous advantage as well as potentially very dangerous and very risky - because, ultimately machines are controlled and operated by humans.
The risk comes from the following:
• Our conversations are no longer ephemeral
• No individual is an isolated dot any more – we are all part of a connected scheme.
• Machine facilitated understanding of information (or Semantic Modeling) makes it possible to discover information and relationships that were previously impossible.
• The Internet allows us to drop our inhibitions - and many people "upload" very personal information about themselves, their lives, their families and friends. This information can potentially be used to generate a complete psychological profile of people. Such information can be potentially be misused by governments, by corporations, by mafia and other dark elements.
• You may buy your computer – but you don’t own it any more. Spyware, Malware, Corporate, governmental espionage
Our Conversations are not ephemeral
Imagine you are traveling in a taxi, a bus or a train with your spouse or a colleague or a friend. Generally, in such situations we talk, and we talk about personal and intimate subjects - forgetting that people around us can hear our conversations. You may probably discuss about a weekend vacation, or a travel plan with your partner. When you come back home after your vacation - you discover that your house was robbed over the weekend.
These things have happened in the past, and they still happen. People talk very private and intimate things in public and sometimes they face unpleasant consequences of such 'inadvertent' public conversations. But, in general, in the past - everyday conversations - whether face-to-face, conversations in public - evaporated as soon as they were said. You could be reasonably sure - that your talk simply vanished in the ether, no trace was left. Of course, organized crime bosses in Movies used to worry about their phones being tapped - but that was an exception. Privacy was the default assumption.
Well, this is no longer true. Our conversations on the Internet - emails we send, our messages on social networking sites like Orkut, Myspaces, the comments we leave on our friends' blogs, the blogs we write, the chat programs we use - all of them are stored, analyzed and recorded. They are stored permanently - for ever. Nothing can be deleted from the Internet - at least not by us.
Every single thing you type on your keyboard - is stored. We know this intellectually, but we never internalized it properly. There are tools that allow both businesses and government agencies to monitor and log IM conversations. E-mail can be saved by your ISP or by the IT department in your corporation. Gmail, for example, saves everything, even if you delete it. More over, these conversations are saved by many different organizations - not just one. Your employer may be storing everything you type on the net, your ISP may be storing everything you type, governments and defense organizations may be storing everything you type. We don't even know who else may be storing our conversations and our messages. We don't know who they are, and we certainly do not know what they do with the information.
There are no laws that regulate who can 'tap' into the information that we send over the Internet. Phone tapping is illegal in most countries - even if the police want to tap some one's phone, they still need a legal order from the magistrate. But, on the Internet, anyone can store anything. There is simply no law and no legislation.
The implications of this to personal safety, liberty and freedom are enormous. You may lose your job - because your employer finds out that you are spending too much time on the Internet and they can now 'prove' it. There may be criminal prosecutions that one has to face, divorce proceedings or simply one may have to face some unpleasant public embarrassment. Such things have happened already - former U.S. Rep. Mark Foley sent salacious instant messages to a young boy and found himself arrested very soon. IBM fired an employee because he was found to be chatting in internet chat rooms. Many people were bullied, kidnapped and robbed because of the information they keep on the internet.
If you find this disturbing, you should. Fewer conversations are ephemeral, and we're losing control over the data. We trust our ISPs, employers and cellphone companies with our privacy, but again and again they've proven they can't be trusted. Identity thieves routinely gain access to these repositories of our information. Paris Hilton and other celebrities have been the victims of hackers breaking into their cellphone providers' networks.
If you don't find this disturbing - read on. Ask yourself what information about you is already on the Internet, and how much information you have kept already. Do you have a blog? Do you have an email-id? Do you leave comments on other's blogs? Do you have your photograph on the Internet somewhere? Do you have membership in some news groups? Do you have an account in social network sites like Orkut? Do your friends write public 'scrapes' in your Orkut profile? Do you regularly chat on the Internet? Do you have am online access to your bank account? Do you buy anything on Internet using your credit card? That's a lot of data that you are putting on Internet, and most of this information is very personal and can be misused.
Not convinced yet? Here are some examples:
I have a blog in my local language - Telugu. The number of Telugu bloggers are still very few - by the Internet standards they don't even count. Because they are few, they are all friends and they know each other very well. Some of these bloggers have 'pseudo-names' and they don't keep their photographs and any other information on their profile pages. But, it is very easy to find out who they are, where they live and what companies they work for. For example, if they visit by blog - by looking at the IP-address records, I can figure out where they are from, where they live and their ISP or company name. Well, the IP-statistics still do not tell you the name of the person. But, suppose if they leave a comment on my blog - then it is very easy to correlate the IP-statistics with the person. I still do not know their name (if they choose a pseudo-name), but I know where they live, and which companies they work for. Once you figured out their IP-address, you can then generate lot of statistics about them - like their usual 'Internet Visiting Hours', whether they browse from home or from work, how long they usually spend time on the Internet, what are their favorite blogs are and so on. It takes only a couple of minutes for an average human being who does not posses any powerful tools like governments, crime syndicates, corporations and defense departments. Imagine how much information about you can be learned by those 'control' minded groups?
"You are innocent until proven guilty" was the old maxim - with the new 'security madness' by governments all over the world - you are guilty until proven innocent is the new heuristic.
The moral is clear: If you type it and send it, prepare to explain it in public later.
This was web1.0. Now, enter web2.0:
No individual is an isolated dot any more – we are all part of a connected scheme
I had a friend - we lost touch with each other for a long time, almost 15 years. I don't know much about her personal details - since we were mostly pen friends. All I knew about her was her name, her family name, her brother's name and her husband's name - and where they were from. I don't even know where all of them are living now.
Since she was a very good friend of mine - I tried several times to find out where she is and how she is, but couldn't locate her - until the social networking sites like Orkut came. Let's say her brother's name is Suresh, and her husband's name is Adarsh. I searched for Suresh in Orkut, and his 'native city' together. I find quite a few people by that name. How do I know whether this particular Suresh I was looking for is one of them or not? All I do is look whether any of these people have a friend called Adarsh. Bingo -- they are there. It took ten minutes to locate them. Rest is easy - you make a contact, get the phone number and call her.
I called her - after the initial joy and excitement of meeting an old friend was over, she suddenly remembered that it would be impossible for me to find her phone number. So, she asked how I got her numbers.
I told her that I found it on the Internet. She was surprised. I told her that I also happen to know her residence address, what her husband is doing, that they recently went for a vacation to Chennai, her in-laws live in such and such a place, that she had a five year old son and that the boy had some health problems recently, that she is doing a part time job and that they bought a computer recently and so on and so forth. BTW, she never touched a computer in her life, and she doesn't even have an email address.
She was totally dumbfounded and flabbergasted. She asked whether I work for CBI. Well, I don't work any intelligence agency. All the information was there on Orkut - for anyone to see. It is surprising how much information we keep unintentionally on the net.
The point here is something even more serious. Today, our presence on the net is not isolated - we are - by our own volition - connected to almost everyone else in the world. We expose our friends, colleagues, associates, classmates, relatives, family and everyone else we know. We don’t stop with that - we also expose how we came to know each of them, we tell the entire world what our interests are, which causes motivate us. How much more can you advertise about yourself to the world?
If her husband and brother were not linked together on the net, it would have been very difficult for me to locate them. It is the 'connectedness' that allows even an average person to quickly find out so much information - even about someone who does not have any internet presence.
We have several such social networking sites - there are social networking sites like Orkut, there are professional networking sites like Linked-IN, there are specialty networking sites and so on. We have a presence in each of them. And, we are connected with so many of our friends and colleagues through these networks. When I log on to Orkut - I get a message right at the top - "you are connected to 39million people through 40 friends". The question is do I want to be connected to 39 million people?
There is an interesting application developed by University of Virginia called Oracle of Bacon. This application can connect any movie actor from any country - past or present to the Hollywood actor Kevin Bacon with in 8 steps. It can connect somebody like S.V.Rangarao with Kevin Bacon. Surprised? How can S.V. Ranga Rao - a telugu actor who acted mostly in telugu movies be connected to Kevin Bacon? Well, he is - and he is connected with in 6 steps.
There is a theory that anyone in the world can be connected to anyone else in the world with in a few steps. This is called 'semantic' distance between two people. Using Orkut and other social networking sites - you are the Kevin Bacon. You can be connected to anyone in the world. What is our distance to the most wanted man in the world? May be six or seven? We don't know - all we know is we are connected. Suppose, if some idiot from a government agency decides to 'round-up' everyone who is connected with the infamous B.L of A.Q fame - we might be on the list. All because, you have some forty friends on Orkut and you don't know most of them.
Is this information available to governments, mafia, drug-traffickers, terrorists, extortion agents, thieves and all other dark forces? Sure it is. Do they have more powerful tools to analyze this information? Sure - they do. Can they and will they misuse this information - sure they can, they will and they do.
Recently FBI issued a parental warning to monitor the internet usage of their children. The reason - many children are lured by sex workers and mafia using the social networking sites.
Recently a young boy killed himself because he was cyber-bullied and he couldn't take it any more.
This is Web2.0. It only gets worse with Web3.0.
Semantic Modeling, psychological profiling, spheare pshing
Earlier, I stated that "understanding" and "extracting" meaning out of information can now be done by machines. Let me explain this in layman terms. Most of you may have read a book called 'Blink'. In the first chapter of this book, the author explains how a trained psychologist can listen to five minutes of everyday conversation between a couple, and figure out very accurately whether marriage lasts, and if does - for how long.
When we meet a person, spend time with them - we generally know something about that person. All of us do some extent of 'psychological profiling'. This is a normal human trait. We do it for our self defense, to know more about the other person, to make friends with them, to understand how to conduct with other person, to gain some favors and so on.
But, it is in general impossible for a human being to remember the entire conversation the other person had with everyone in the world, and analyze it later at leisure. Most of our 'estimate' of a person happens in a split-second, it is - in general - not a planned, conscious activity. Suppose, all our talk along with a lot of information about ourselves is available electronically - and suppose machines are capable of 'analyzing' this information and draw up a psychological profile of this person - isn't that a very powerful information to possess?
Recently, Amit Seth and others published a very interesting paper called "Semantic Analysis of Social Networks - Experiences in addressing the problem of conflict of interest detection". Interestingly, the project was sponsored by ARPA - which is the US defense department's advanced research agency. The researchers used information publicly available about authors of scientific papers from two different sources, and using the semantic modeling techniques they developed, they discovered that many authors - to put it in layman terms - cheat to get their papers published. Suppose there are two authors called X and Y. Let's say that X and Y worked together in some research institution in the past and published a joint paper together. Later, X left the institution and joined some other organization and Y continues to work for the same institution. Now, when X sends a paper for publication, Y acts as the reviewer and accepts the paper for publication. This is called conflict of interest - because people who know each other are not supposed to be reviewers of each other.
Earlier, finding out such information was practically impossible. How does one journal know that Y and X know each other? But, with social network sites, and professional network sites - such information can be very easily detected by computers today. Now, this is a harmless, useful research paper. But, what are its potential applications?
A super Google can be easily developed which can 'extract' very meaningful and powerful information about people and their lives. If there was ever any complaint was filed against you - that information can be 'fed' into such an application. It can easily discover what is your financial net worth by analyzing your online purchase history, or your online bank account transactions. It can easily generate a psychological profile of yourself from your online information. It can easily draw inferences about a person of various kinds - whether a person is anti-government, he/she is likely to be a political activist, whether they can be easily be made to submit themselves to some kind of brain washing and so on.
If the Internet adopts the RDF standard which is pending - developing such super 'Google' becomes a relatively easy task. Till now, such programs required very expensive and specialized computing models. But, with all 'information' and 'meaning' available simply as part of the attribute of an object - we no longer require any complex computational models to generate inferences, it can be done by any normal database query. This means that anyone who has access to the data - everyone - can easily develop applications to 'discover' information about people.
Implications of RDF and Semantic Databases
After the 9/11, there was a massive initiatives by the US government and several other governments to collect massive amounts of centralized data about people and store it in one central place. The Bush Administration started an initiative called Total Information Awareness - which was later scrapped because of the unprecedented public outcry. This TIA act gave absolute powers to the security agenecies to collect any information about anyone. Though this 'act' was cancelled - it is now part of the defence department's unclassified projects. The name changed, but the work continues.
With RDF, there is no need to develop any centralized databases anymore. Because, every object on the Web is capable of answering some interesting questions about itself - one can simply get the information from multiple sites and run any query on a distributed database.
If this sounds too technical - here is an example:
These days, a typical Internet usage by an average user can be summarized as:
1. An Email-Id with webbased email services like Google, Yahoo etc.
2. Probably and official or corporate Email-Id that is used for business and professional communication
3. Several Login-Ids and some activity at several e-commerce sites - like Amazon.Com, e-Bay
4. Some history of online purchases using credit cards - airline tickets, music and books, groceries
5. A login-id with at least one social networking site like Orkut, Linked-IN, Myspaces.com and so on
6. Probably a blog and a personal website.
Add to this, lot more online data that is created without our knowledge and permission - if you traveled in an Airplane - your travel history is recorded by that Airline company. If you visited some countries either as a tourist or on work - your entire travel history is recorded by many governments, travel companies and so on. If you booked your hotel reservations online - this history is recorded by various organizations. If you passed through various security zones at several places - your photographs and videos have been captured and stored.
With Semantic-Web, all this information - can be potentially linked together without any significant effort. There is no need to centralize all this various information sources into one centralized database anymore. This is the power of semantic databases. With the advances in multi-media technologies - searching through multi-media objects like image search, video search, voice search will also be possible quite soon. The technology has not yet reached this stage of evolution - but it will soon get there.
This basically means that we leave an enormous amount of electronic audit trail of ourselves, some of it willingly created by us, and most of it created without our knowledge.
How can such information be misused? One example is spear phishing. Here is an excerpt from an article published in Newsweek International called "A Dangerous Game of Phishing":
"Spear phishers gather information, usually on the Internet, about an individual, and then craft a personalized e-mail more likely to dupe the mark. According to the FBI, the personalization method has proved so profitable that a significant number of spear phishers, principally located outside the United States, began applying it to death-threat extortion e-mails for the first time last December. FBI spokeswoman Cathy Milhoan says the problem is "huge."
Here's how it works: A spear phisher collects information on an (often wealthy) individual, then writes a chilling e-mail. The sender, posing as a hit man, offers to spare the recipient in exchange for a large sum of money. If the ploy doesn't work, the target receives a second e-mail, purportedly from the police, explaining that his or her name and address were found on a recently arrested murder suspect. "The victim gets scared, gets paranoid, he gets a lot of things," says Alan Paller, a cybercrime expert with the Bethesda, Maryland, SANS Institute who has testified before the U.S. Congress on the matter. The target provides personal details—including financial data—to aid the investigation.
Traditional extortion often involves tailing targets and staking out their homes to obtain the particulars—such as the appearance of a victim's daughter—that render threats credible. Today much of that information is easily gleaned from the 'Net. Dan Vogel, an Edmond, Oklahoma, former FBI profiler, says social-networking Web sites such as MySpace are "fueling" the trend."
In an article entitled "Governments research to track online networking", Christopher Dela Cruz and Megan Carr say the following:
"The Department of Homeland Security is paying Rutgers $3 million to oversee development of computing methods that could monitor suspicious social networks and opinions found in news stories, Web blogs and other Web information to identify indicators of potential terrorist activity.
The software and algorithms could rapidly detect social networks among groups by identifying who is talking to whom on public blogs and message boards, researchers said. Computers could ideally pick out entities trying to conceal themselves under different aliases.
It would also be able to sift through massive amounts of text and decipher opinions - such as anti-American sentiment - that would otherwise be difficult to do manually.
The program is designed to sift rapidly through huge amounts of data. It has also been described as a sort of "Super Google" researchers such as Eduard Hovy at The University of Southern California, to explain the scope and quickness of the technology."
Spy Blog: Don't miss this one!!
Information warfare: U.S. Department of Defense's policy documents on Information Warfare.
Surveillance Issues: We live in a world of surveillance. It is amazing how many such technologies and devices surround us.
Electronic Frontier Foundation: An organization that works on several issues related to privacy, security and other matters. A must visit for all bloggers.
Thursday, December 28, 2006
Freelancing in India – The oriental carpet shop
In
First, let me tell you a couple of stories from the field. The first story is about the need to read carefully all legal agreements and more importantly the price of arrogance and individualism that one pays for in this country. The second story is about the need to understand the subtlety of the eastern mind.
I know of a consultant who worked for a billionaire. There was a disagreement between them on some issues - the issue was not about money, but the working styles of the client and the consultant. This particular consultant was very arrogant - one day he had a major fight and walked out of the assignment. According to the agreement he signed - he could leave with one day notice, so he thought everything would be fine, and at best his client wouldn't release the pending payments.
Two days later, a battery of lawyers descended on his house - and practically sealed everything - his wristwatch, cell phone, microwave, TV, credit cards, computer - and practically every single tiny electronic item he owned was sealed. The reason: he signed an intellectual property document, and the document says that in the event of termination of the assignment, the client has a right to verify that no confidential data was stored in any electronic medium!! Technically even a wrist watch is an electronic medium.
This poor chap couldn't even make a phone call, and his client was out of contact for a week. Finally, he had to swallow his pride and beg for pardon.
The important lesson is that be careful of the documents you sign with your clients. Any condition in the agreement must be something that you yourself should be able to establish.
The second story is my own first hand experience. In
Once, I completed my assignment, and sent all documents to my client, and sent them the invoice. There is a thirty day period for all payments - so, I waited for about three weeks. Nothing happened. I contacted the finance department - they told me that they did not receive any approval from the VP, Engineering - who is the person in charge for my consulting work. I met him again - he apologized profusely for the delay for about half an hour, and in the meanwhile asked me to 'help' them with a few small things and so on and so forth. I obliged - it took another one week. I sent the invoice again to the VP and the finance department. This time, the VP took a printout of the invoice - and he wrote on it the three magic words: "please pay" and signed. I handed over the invoice to the finance department. The accountant gave me a charming smile and said everything will be done in a week's time. Well, nothing happened for another two weeks.
Again, I met the VP. This time he cursed their finance department - he said they are always very slow, don't do their job on time etc., etc. He again took a printout and again wrote the three magic words and signed. I took the invoice to the finance department. This time the accountant asked me to wait - he wrote the check and gave it to me within half an hour.
I asked the accountant that why they did not clear my payments the first time the bill was passed by the VP. The accountant said "Sir, the first time he cleared your invoice, but wrote it in blue ink. That is an instruction to us not to pay. If he writes it in green ink, then it means send the check after two weeks, and if he writes it in red ink - it means that we should pass it immediately".
This is how subtle the process could be here. I use some simple rules to deal with the eastern complexity and it works.
First, I always do one week of gift work before I formally engage with any clients. I don't charge for the first one week, and I work without any formal contract. This allows me to understand several things - most importantly I get to understand the problem I am expected to solve, I get to assess whether I could do it or not, whether the team is capable enough to solve it themselves - how much guidance they need from me and so on. Most importantly, I can estimate the culture of the organization. Metaphorically speaking - I get to know in which color the invoice has to be signed!!
A consultant has to deal with three different people in the client organization. The person who employed you, generally it is a senior manager - most often, the head of development, delivery or technology. The second person is the one who actually needs you and works with you - generally it is the development team, represented by a project manager, and finally - some one from the finance department. These three people can form a nice little triangle and play a perpetual football with you.
What I do is to try and break the triangle. I insist that one person from the development team - either a project manager, team lead or a programmer be assigned as my primary client contact, who will have all the authority to decide on the deliverables, to sign off on my time sheets and who has to ensure that payments are cleared. This means that even though the agreement is signed between me and a senior manager of the company, the agreement explicitly nominates one person by name.
The advantage is that - programmers do not switch off their cell phones, they do not go on extended foreign tours, and they don't have secretaries. You can always call them up and go for a lunch. And, they understand which color of ink their boss has to sign. Basically, I make my internal client contact run around to clear all my payments. Once they accept this responsibility - they are in general very sincere and prompt. Contrary to all generally accepted beliefs, I found that the finance department tends to be the most efficient and prompt of all other divisions of the organizations. Once they get clear instructions, they act very promptly.
The second part of my consulting agreement is to clearly identify three types of deliverables. In
The disadvantages:
- The scope always expands.
- No clear idea of what the deliverables are - the distinction between the work and the deliverable are always blurred. You are asked to do work but get paid based on certain concrete deliverables that you have to produce.
- Confusion about who is owner for a certain area of work.
Normally, what I do is to identify three types of responsibilities and document them as part of my 'work contract':
- Primary responsibilities: I have the primary ownership for these deliverables, and I have control on the project plan.
- Collaborative responsibilities: My expertise is required for these deliverables, but someone else has the overall ownership and control on the pace of the activity.
- Participatory responsibilities: Areas where my inputs are sought - but, I have a right of refusal to participate.
Finally, it is important to get the payments on time. Suppose I expect the payment to be done on 30 of the month. If I send the 'deliverable' document on 25th, I can never get the payment on time. This is because, the deliverable is not completed when I send it, but when it is read and understood by the receiver. So, if I want my payments on 30th, I send my documents on 10th of the month - and give them two weeks time to accept the deliverable, and also I use the two weeks to ensure that they indeed make the effort to understand it clearly.
Wednesday, December 27, 2006
Freelancing in India
Many young people from India are writing to me asking about freelancing. If you are also ‘bitten’ by the freelance bug – and you are thinking of writing to me, read this before you shoot that email to me.
I believe self employment will take off in a big way in India in the next two to five years – therefore, there will be lot of scope for people to be on their own. There is a need for specialized knowledge, speed and flexibility in our industry. Many people are now well connected, well travelled and they know their bearings well. All this means that the situation is ‘cooking’. It is the right time to jump in for the early mover advantage.
However, freelancing is a difficult thing to make it work – anywhere in the world, and especially in India. It is not going to be straightforward.
First, let me dispel a few myths – freelancing is not financially lucrative in the long run. People who work as freelancers do so because they love to be independent, have some time on their hand to pursue various other interests and generally be the trailblazers. It is not a road to make wealth. If you are a Peter Drucker – it may be different, but even for Drucker – he would have made lot more money if he started his own company or worked for some other big company.
There is never any ‘job’ security if you are on your own – it is always a struggle to find the next project, and do it well and have some regular income.
There are a few tax advantages – but the overall expenses far out weigh any tax advantages. For example, you have to invest in your own training, and in your own career. If you work for any large IT-Company, they take care of all your training needs. If you work for a large IT company in India, this amounts to about 50K per year (including the training expenses + the salary you get even though you are attending training). Similarly, there are many other expenses if you are on your own. These days many IT companies provide many perks – which are tax free incentives, you don’t get anything more if you are a freelancer.
Part time job, second income is completely different from being on your own. There is no comparison at all.
There is also a big difference between contracting and consulting. One is getting some work and executing it, and the other is lending your expertise to produce results. These two ways of engagement work very differently.
So, think carefully before making a move. You can always get the first couple of contracts – but can you get business for the rest of your life? I do not want to discourage anyone – but only trying to throw some light on a few dark areas that people generally don’t see.
Now, here are a few tips:
To be successful you have to:
That’s pretty much it. If you are committed to it – it is not hard. Believe in yourself, make a small begining, do it well and don’t give up. That’s all there is to it.
I believe self employment will take off in a big way in India in the next two to five years – therefore, there will be lot of scope for people to be on their own. There is a need for specialized knowledge, speed and flexibility in our industry. Many people are now well connected, well travelled and they know their bearings well. All this means that the situation is ‘cooking’. It is the right time to jump in for the early mover advantage.
However, freelancing is a difficult thing to make it work – anywhere in the world, and especially in India. It is not going to be straightforward.
First, let me dispel a few myths – freelancing is not financially lucrative in the long run. People who work as freelancers do so because they love to be independent, have some time on their hand to pursue various other interests and generally be the trailblazers. It is not a road to make wealth. If you are a Peter Drucker – it may be different, but even for Drucker – he would have made lot more money if he started his own company or worked for some other big company.
There is never any ‘job’ security if you are on your own – it is always a struggle to find the next project, and do it well and have some regular income.
There are a few tax advantages – but the overall expenses far out weigh any tax advantages. For example, you have to invest in your own training, and in your own career. If you work for any large IT-Company, they take care of all your training needs. If you work for a large IT company in India, this amounts to about 50K per year (including the training expenses + the salary you get even though you are attending training). Similarly, there are many other expenses if you are on your own. These days many IT companies provide many perks – which are tax free incentives, you don’t get anything more if you are a freelancer.
Part time job, second income is completely different from being on your own. There is no comparison at all.
There is also a big difference between contracting and consulting. One is getting some work and executing it, and the other is lending your expertise to produce results. These two ways of engagement work very differently.
So, think carefully before making a move. You can always get the first couple of contracts – but can you get business for the rest of your life? I do not want to discourage anyone – but only trying to throw some light on a few dark areas that people generally don’t see.
Now, here are a few tips:
To be successful you have to:
- Establish your credentials – credentials are not your resume, but your expertise and demonstrating that you can deliver. What you can do is to first try and get some work from internet. There are now some websites like rentacoder.com, elance.com etc., - which help freelancers find work across the globe. You can register yourself on these websites. It works pretty much similar to e-bay. Some companies post their requirements – and you can bid for projects, execute them from home. I suggest that you find out more about sites like this, and see if there is some work there that fits your skills and what you are looking for.
- Transition from ‘a skilled person’ to ‘an expert’ – you have to develop certain techniques, a way of doing your work that establishes your expertise. You have to learn to write well, communicate well, publish some papers, have a website of your own and carve out for yourself a niche. There are tons of open sources projects these days – if you participate in these projects, you get to learn the world class standards that are used today, and also you get to meet other people who are freelancers and create a network of your own.
- Have a network of contacts, friends and others from the same area. If you are working in a company – even though you don’t realize, you are part of a community which ensures that you get to know a lot of things from the environment. If you are a freelancer, you have to cultivate different kinds of relationships. There are many ways of accomplishing this – membership in professional bodies, attending conferences, teaching in universities, making contact with a group of people who work in your area (see point no: 2).
- Understand how consulting works. There are some excellent reference books on this subject. Flawless Consulting by Peter Block, Consultant’s calling by Bellman, Soloing are some books that I recommend. Read them, understand them and then think about how to go about it.
- Read a lot. You have to read many different kinds of books – not just technical books. The general thumb rule is that you have to be able to read at least 500 pages of material every week and absorb it. If you claim to be a knowledge worker – you have to have knowledge. Right? Some books I suggest to begin with: Gerald Weinberg’s books on Software Engineering, Quality Software Management and his books on consulting. Also books on agile techniques. Books on systems thinking, history and some poetry. Definitely you have to know a great deal about accounting, managing your own finances and legal aspects of self employment.
- Invest in your own career. This means that you have to have a definite idea of how much free time you will have, and how you will use the free time. If you do billing work all through the year – you will become stagnated very fast, and you can’t get business the next year. Our field evolves very fast, and for a freelancer it is very important to be on top of the technology. Therefore, it is mandatory that you have a definite plan how to stay ahead – this will include, reading a lot, attending workshops and training programs, write and communicate, teach and so on. In financial terms, you have to invest at least 30% of your revenue in your own learning at least in the initial years.
That’s pretty much it. If you are committed to it – it is not hard. Believe in yourself, make a small begining, do it well and don’t give up. That’s all there is to it.
Saturday, December 16, 2006
Secrets of Successful Freelancing
Many people ask me about freelancing – how it works, how do I make it work and so on. Most often I duck the question because it is a difficult question. This article is a short guide on how to survive as a freelancer.
It is not easy to be a freelancer. Most freelancers want work, not a job – which makes it tough for them in the long run. They want freedom, they value competency, they think they bring real expertise to the table and want to continue to enjoy the challenge of work.
I always suspect the people who tell me that they enjoy learning. Learning is a painful process, why would anyone want to learn in the first place? We learn best only if we need to. Children learn so fast because it is a question of survival. Psychologists define learning as a permanent change in behavior due to a prior experience. Genetically, we are designed to resist any change – whether it is for good or for worse. Apparently in the same way we have an internal thermostat which controls our temperature – we have another instrument whose function is always bring us back to the original state. This is the reason why changing anything in ourselves is such a massive effort.
Learning is a lot like being in love – and being love is a painful experience. The separation from one’s beloved is a suffering and the lover secretly craves for that rupture involved in that longing. This is a mysterious process – much like the drug addicts cravings for a drug.
Clients hire freelancers primarily for business, economic and operational reasons. We are too expensive to be on their employee roles. We are temperamental prima-donnas who are fiercely independent and therefore in the long run we are an expensive maintenance problem that nobody wants to have on their hands. We are addicted to problem solving, and therefore we cannot stay in one organization long enough.
These are the real reasons why we are ‘allowed’ to be freelancers.
It took me a long time to understand this. Most problems between consultants and their clients arise because of ‘expectation’ mismatch. We expect to be valued for our contributions and for our love of work. Organizations value and reward loyalty – not loyalty to work, but loyalty to themselves.
It was a difficult learning. But, after I realized it, it is not too hard to internalize it. The following eleven rules are a direct consequence of this realization. If you practice these rules – you may not become a world class freelancer, but you may be able to survive as one – which is no mean accomplishment in itself.
Effective application of these rules however requires some tact. Here are two Nasruddin stories that capture the essence of these rules and how to practice them:
Whose Servant Am I?
Mulla Nasruddin had become a favorite at Court. He used his position to show up the methods of courtiers.
One day the King was exceptionally hungry. Some aubergines had been so deliciously cooked that he told the palace chief to serve them everyday.
‘Are they not the best vegetables in the world, Mulla?’ he asked Nasruddin.
‘The very best, Majesty’
Five days later, when the aubergines had been served for the tenth meal in succession, the King roared: ‘Take these things away. I HATE them!’
‘They are the worst vegetables in the world, Majesty’ agreed Nasruddin
‘But, Mulla, less than a week ago you said that they are were the very best’
‘I did. But, I am a servant of the King, not of the vegetable’
The Gold, the cloak and the horse
‘I cannot get a job’ said the Mulla, ‘because I am already in the service of the All-Highest’
‘In that case’, said his wife ‘ask for your wages, because every employer must pay’
Quite right, thought Nasruddin
‘I have not been paid simply because I have never asked’, he said aloud
‘Then you had better go and ask’
Nasruddin went into the garden, knelt and cried out: ‘O Allah, send me a hundred pieces of gold, for all my past services are worth at least that much in back pay’
His neighbor, a moneylender, thought he would play a joke on Nasruddin. Taking a bag of hundred gold pieces he threw it down from a window.
Nasruddin stood up with dignity and took the money to his wife. ‘I am one of the saints,’ he told her. ‘Here are my arrears’
She was very impressed.
Presently, made suspicious of by the succession of delivery men carrying food, clothing and furniture into Nasruddin’s house, the neighbor went to get his money back.
‘You heard me calling for it, and now you are pretending it is yours’ said Nasruddin. ‘You shall never have it’.
The neighbor said that he would take Nasruddin to the court of summary jurisdiction.
‘I cannot go like this’, said Nasruddin. ‘I have no suitable clothes, not have I a horse. If we appear together the judge will be prejudiced in your favor by my mean appearance.’
The neighbor took off his own cloak and gave it to Nasruddin, then he mounted him on his own horse, and they went before the judge.
The plaintiff was heard first.
‘What is your defense?’ the magistrate asked Nasruddin.
‘That my neighbor is insane’.
‘What evidence have you, Mulla?’
‘What better than from his own mouth? He thinks that everything belongs to him. If you ask him about my horse, or even my cloak, he will claim them, let alone my gold’
‘But they are mine!’ roared the neighbor.
Case dismissed.
It is not easy to be a freelancer. Most freelancers want work, not a job – which makes it tough for them in the long run. They want freedom, they value competency, they think they bring real expertise to the table and want to continue to enjoy the challenge of work.
I always suspect the people who tell me that they enjoy learning. Learning is a painful process, why would anyone want to learn in the first place? We learn best only if we need to. Children learn so fast because it is a question of survival. Psychologists define learning as a permanent change in behavior due to a prior experience. Genetically, we are designed to resist any change – whether it is for good or for worse. Apparently in the same way we have an internal thermostat which controls our temperature – we have another instrument whose function is always bring us back to the original state. This is the reason why changing anything in ourselves is such a massive effort.
Learning is a lot like being in love – and being love is a painful experience. The separation from one’s beloved is a suffering and the lover secretly craves for that rupture involved in that longing. This is a mysterious process – much like the drug addicts cravings for a drug.
In the initial phases of my career as a freelancer, I deluded myself that my clients hire me because of my expertise, knowledge or the skill I possess. I deluded myself that I am expensive because I am very valuable. I deluded myself that doing the best job is the most important thing in the world. I deluded myself that doing the best job on time will get me my payments on time and also bring me fame and respect, and even an extension of the contract.
Well, people value those traits. As freelancers we are respected for our expertise, our knowledge and our skill. Eventually, these qualities might even allows us differentiate ourselves in the market place and allow us to quote a premium for our services. We may be valued for these traits, but we are not hired for these reasons. And, we never retain our contracts for these reasons. We retain a contract only if – as people – we demonstrate very high levels of personal integrity. If we have no personal integrity – then we cannot sustain ourselves as freelancers even for a month.Clients hire freelancers primarily for business, economic and operational reasons. We are too expensive to be on their employee roles. We are temperamental prima-donnas who are fiercely independent and therefore in the long run we are an expensive maintenance problem that nobody wants to have on their hands. We are addicted to problem solving, and therefore we cannot stay in one organization long enough.
These are the real reasons why we are ‘allowed’ to be freelancers.
It took me a long time to understand this. Most problems between consultants and their clients arise because of ‘expectation’ mismatch. We expect to be valued for our contributions and for our love of work. Organizations value and reward loyalty – not loyalty to work, but loyalty to themselves.
It was a difficult learning. But, after I realized it, it is not too hard to internalize it. The following eleven rules are a direct consequence of this realization. If you practice these rules – you may not become a world class freelancer, but you may be able to survive as one – which is no mean accomplishment in itself.
- If you are a Socrates, be prepared for a painful exit. Remember - Socrates did not complain about painful exits.
- Do not blame the stimulus for the response. You are the one who seeks problems – so, never blame the client for giving you the problems. How you experience the problem and your (in)ability to solve it belongs only to you.
- The estimation of the self must be tempered by the knowledge of the self.
- A King will never ask his general whether he is ready to defend the country. The fact that the general is still in his job means that he is ready to do so. Therefore, the King will only give marching orders.
- As a consultant, the only right you have is to do your duty. Always remember that it is your right. Fight for it, protect it but never allow anyone/anything to deny this right to you. Most important – don’t deny it yourself.
- A freelancer works for cash - not for kind. Therefore, we don’t get paid twice. You can either take a check or receive gratitude. Learn to prefer the check to gratitude.
- Always do a thankless job. The only thank-you note you can expect is the payment made on time.
- Be thankful if you get some work to do, and you get regular payments. These are the only two objective measures that you can use to assess yourself. The rest are delusions.
- The Shareholders control the company, the founders operate the company, the employees are the company, and a consultant is associated with the company. All data modelers know that association is the weakest relationship – it has no integrity constraint imposed on it. It is the easiest and most convenient relationship to terminate. No reason or rationale is required to terminate it – because such a termination does not break down the underlying system.
- The success of a consultant is directly proportional to how fast he/she makes himself/herself redundant to the client.
- Solve that problem as quickly as you can and get out.
- If there is no problem to be solved, get out.
- If you cannot solve the problem – get out.
- Luxuries that a freelancer cannot afford:
- Anger – never walk out of the contract in anger. Don’t leave especially after you have had a fight with your client over some issue.
- Resignation – an employee can resign. A freelancer cannot. There are only two ways to exit: either complete the assignment, or get fired. Sometimes getting fired is not as bad as we imagine it to be.
- Procrastination – you sell your time, so it does not belong to you.
The Weinberg touch to Rule No: 11:
I had some difficulty and even used to get angry with the last rule. I felt that there was something wrong with it – it did not sound right. But, I kept it – because, it served me very well in my initial years as a freelancer. My discomfort also meant that there is something deeper lurking inside that rule. I was confident that I will eventually discover what it is.
Recently I posted these rules to Gerald Weinberg’s Shape Forum. Jerry disagreed with this rule and suggested a very subtle shift to this rule. The nudge he provided suddenly threw light in the dark corners of the rule and the full beauty of the rule suddenly became evident to me. So, I dedicated the Rule Eleven to Gerald Weinberg.
I call it the Weinberg Touch:
11a. never leave anything behind – especially your anger.
11b. An employee has an option to resign – a freelancer has an option to renegotiate.
11c. Your non billing time is more valuable than you’re billing time.
11d. Getting fired is never as bad we imagine it to be and it is more often than not a blessing.
Effective application of these rules however requires some tact. Here are two Nasruddin stories that capture the essence of these rules and how to practice them:
******
Whose Servant Am I?
Mulla Nasruddin had become a favorite at Court. He used his position to show up the methods of courtiers.
One day the King was exceptionally hungry. Some aubergines had been so deliciously cooked that he told the palace chief to serve them everyday.
‘Are they not the best vegetables in the world, Mulla?’ he asked Nasruddin.
‘The very best, Majesty’
Five days later, when the aubergines had been served for the tenth meal in succession, the King roared: ‘Take these things away. I HATE them!’
‘They are the worst vegetables in the world, Majesty’ agreed Nasruddin
‘But, Mulla, less than a week ago you said that they are were the very best’
‘I did. But, I am a servant of the King, not of the vegetable’
The Gold, the cloak and the horse
‘I cannot get a job’ said the Mulla, ‘because I am already in the service of the All-Highest’
‘In that case’, said his wife ‘ask for your wages, because every employer must pay’
Quite right, thought Nasruddin
‘I have not been paid simply because I have never asked’, he said aloud
‘Then you had better go and ask’
Nasruddin went into the garden, knelt and cried out: ‘O Allah, send me a hundred pieces of gold, for all my past services are worth at least that much in back pay’
His neighbor, a moneylender, thought he would play a joke on Nasruddin. Taking a bag of hundred gold pieces he threw it down from a window.
Nasruddin stood up with dignity and took the money to his wife. ‘I am one of the saints,’ he told her. ‘Here are my arrears’
She was very impressed.
Presently, made suspicious of by the succession of delivery men carrying food, clothing and furniture into Nasruddin’s house, the neighbor went to get his money back.
‘You heard me calling for it, and now you are pretending it is yours’ said Nasruddin. ‘You shall never have it’.
The neighbor said that he would take Nasruddin to the court of summary jurisdiction.
‘I cannot go like this’, said Nasruddin. ‘I have no suitable clothes, not have I a horse. If we appear together the judge will be prejudiced in your favor by my mean appearance.’
The neighbor took off his own cloak and gave it to Nasruddin, then he mounted him on his own horse, and they went before the judge.
The plaintiff was heard first.
‘What is your defense?’ the magistrate asked Nasruddin.
‘That my neighbor is insane’.
‘What evidence have you, Mulla?’
‘What better than from his own mouth? He thinks that everything belongs to him. If you ask him about my horse, or even my cloak, he will claim them, let alone my gold’
‘But they are mine!’ roared the neighbor.
Case dismissed.
Subscribe to:
Posts (Atom)
