"Persuasive" Secrets!

Recently, I found a new area of Interest and that was behavioral science. There was a reason behind it. A few days ago I got a call from a Real Estate Agency as I was looking for a house in Mumbai. The real estate agency was registered on the same housing website as I was so it was easy for them to track me and my requirements (Analytics in play :P) .  The agent was so fluent and concurring that I could not say no to his request. So, after the phone call I thought was this offer to my benefit? or he had the perfect skills to make me say yes to his request. So I thought of digging deep into this topic.

I found out that researchers have been studying the factors to say yes to the request of others and there can be no doubt that there is a science in how we are persuaded. The science is surprising. In making the decision we may feel that people consider all the options and information they have. But, the reality is very often different.

In the increasingly overloaded life we live, more than ever, we need shortcuts or rules of thumb to guide our decision making. We humans can't possibly think through every situation from scratch, so we also have "triggers" that set off automatic, unthinking action. Our brain is built to help us simplify life by identifying patterns moment-by-moment, and automatically setting in motion set sequences of standard, well-rehearsed behavior.

Someone says "Good morning" to you, and without thinking, you say it back. You see a red sign at a signal, and you automatically stop your car (or at least slow down). A group of people around you laugh, and you laugh right along even if you didn't hear the joke.

Human triggers and patterns of response are more varied and complex, and certainly less predictable. And people are more aware than ever of the manipulative nature of selling and advertising. However, if you can discover the most powerful human behavior sequences and find the thing that triggers them, you can lead people to say "yes" to your requests more often.

So after some observations and initial studies and references I want you to take you through just 6 of these shortcuts that guide our behavior.

They are.

1. Reciprocity

2. Scarcity

3. Authority

4. Consistency

5. Liking

6. Social Proof

By implying these shortcuts in ethical manner there is a high chance that someone may get persuaded by your request. Let us understand these 6 factors with a few examples.

1. Reciprocity : Simply put , People are obliged to give back to others the form of behavior, gifts or service that they have received first. If a friend invites you to their party then there is an  obligation for you to invite them to the future party you are hosting. If a colleague does you a favor then you owe that colleague a favor. In the context of the social obligation people are likely to say yes to those to they owe. One of the best demonstration I can tell you is that of the restaurant. The last time I had visited a restaurant for a birthday party of my friend the waiter gave him the small birthday gift (A box of 3 cookies) on behalf of the restaurant. They did this when they brought the bill. The tip that he gave was twice than what he could have given earlier. So here is the question does the giving of that gift had any influence of him giving that tip ? Many people will say NO! But let me tell you that this small gift can make a surprising difference in your decision making. Either you will give a high tip or the probability of you visiting that restaurant again is very high. It is because you owe them an another visit. You not influenced by what was given but by how It was given. So, the key to use principle of reciprocation is be, the first to give and give something unexpected and personalized.

2. Scarcity : People want more of those things that they can have less of. When Indigo announced that they will have just one flight from Kolkata to Pune instead of two daily, the sales increased!

The airfare was constant, time required to travel was same, the service did not change so what made this increase in sales ? It was because it had become a scarcity. When there was a news that onions in Maharashtra were scarce that was the time when demand suddenly increased. When you look for a career and you find out that "XYZ" professionals / resources are scarce in the industry right now, you choose to prefer that career path. When a thing is scarce people want it more.

So, when it comes to effectively persuading others with the principle of scarcity , the science is clear. It is not enough to tell people the benefits of your products or services. You will need to point out what is unique in your offering and what they stand to loose.

3. Authority : This principle is based on the idea that people follow the credible knowledgeable experts. Physiotherapists can persuade patients with their recommended exercises if they display their medical degree on the walls of their clinic. People will easily give parking charge to the one with uniform instead of casual clothing although both of them are complete strangers. What it tells us is it is important to  signal to others what makes you a credible and knowledgeable authority before you make your influence attempts. Of course, this could pose problems if you go on telling your customer that how brilliant you are. But, you can arrange someone to do it for you. That is the reason for arranging the reception staff. When the receptionist tells you that let me connect you to our subject matter expert who has the experience in this field for 16 years. You suddenly feel that he is going to make your decision easy. So to persuade, you must let know the credentials of the man who is going to assist or help the other person making decision.

4. Consistency : Consistency is a valuable adaptive behavior. Socially and personally, consistency is beneficial, a practical survival skill in a complex world. Doing things the same way time and time again, or maintaining the same ideas about things, gives us helpful shortcuts.

Consistency makes thinking easy, because there's little thinking needed. Make up your mind about something once, and you never have to think about it ever again. Faced with new situations, you simply recall how you handled it before and the problem is solved. When the Rule of Consistency is triggered, decisions are made almost automatically.

In addition to simplifying our thinking, Consistency also helps us avoid unpleasant emotions. It helps us stick to what we know and avoid the chance of disappointment, embarrassment, failure, and loss.

Consistency can be activated by making small initial commitments or asking for small initial commitments. people are not only driven to "be" consistent, but are also driven to "appear to be" consistent. That's because, socially, consistency is a desirable personal trait. It is seen as rational, trustworthy, stable, and decisive. Inconsistency, on the other hand, is usually frowned upon. It is seen as irrational, deceptive, unstable, and indecisive.

Think about the people you most admire. Aren't most of them remarkably "consistent" in their actions and beliefs? Don't they seem incredibly decisive and sure?

5. Liking :  No matter how reasonable we may think ourselves to be, we are always more likely to say "yes" to those we know and like. We readily comply with requests from those who are similar to us and for whom we have good feelings. It's what makes refusing to buy Tupperware from a friend or relative next to impossible. There are 3 important factors while we like some one:

a. We like people similar to us

b. We like people who pay us compliments

c. We like people who co operate with us towards mutual goals

I still remember as an exercise during my post graduation course we were given an assignment of winning a contract. There were two different groups and naturally there were two different methodologies. First group thought time is money and so they got down to the business directly. The second group, however, started with a personal touch and then came to the point of offering. The contract was won by the second team.

6. Social Proof : Especially when confused, people will look at the action of the others before their own. recently I had chance to visit my friend in an hotel room. there was a small card in bathroom in attempt to persuade their clients to reuse their towels and napkins. They had also placed a board telling that 75 percent of their clients reused their towels and napkins. Automatically me and my friend reused the towel. Most of us are imitators in most of what we do. We look to others for guidance, especially when we are uncertain about something. We ask, "What do others think about this? What do others feel? What do others do?" Then we act accordingly. The more people who do something, the more likely it is correct — or at least relatively safe. Not always, but for most things it works with a high degree of reliability.

And when you understand the rule, you can trigger it at will to generate a "yes" response to your offers and requests. That is why Organisations display the list of satisfied customers or clients.

As I wrote this post there were many occurrences came to my mind. Occurrences when I said YES . I got to know the reason behind me saying yes and the people who were successful in persuading me. I hope you will recollect such similar incidences as well. So, next time when you say "Yes"  you will think of all these 6 factors.

Caution : Do not make your life complicated by over-thinking, it already complicated enough !!

Choice Overload :A Paradox

An eCommerce website sells sells 9,763 kinds of of mobile phones and in India we have options of buying from at-least 10 of them. I know this because I recently spent an evening trying to choose one of them. Nearly an hour later, after having read countless contradictory reviews and pondering far too many choices, I felt grumpy and tired and simply gave up. The next day, i limited my search to what I want and from where I want and was able to come to a decision in 10 minutes.

Too many choices exhaust us, make us unhappy and lead us to sometimes abscond from making a decision all together.
This "Choice overload" can lead us to avoid making important decisions.As the number of options increases, the costs, in time and effort, of gathering the information needed to make a good choice also increase.The level of certainty people have about their choice decreases. And the anticipation that they will regret their choice increases.
Understanding how and why we make decisions can perhaps help us make better choices down the line.


We make poorer decisions when we are tired. It's caused by decision fatigue. The mind can only sort through so many options and make so many choices before it starts to run out of steam. That's why impulse buys like candy bars and magazines at the checkout aisle in the grocery store are hard to resist. We've exhausted all our good decision-making skills.

The same goes for our workday. Making lots of decisions not only exhausts us, it can put us in a fowl mood. A study out of Columbia University found that judges were more likely to give prisoners a favorable ruling in the beginning of the day and after a food break, than at the end of the day.

That's why it's important to make your most important decisions in the morning rather than at the end of an exhausting day when your energy has been depleted. The "sleep on it" idiom really does have clout when it comes to making big decisions.

Dark side of too many options

When we're tired, we tend to conserve our energy by making choices based on a single factor like price, for example, rather than considering all the other determinants that go into making the best decision. When we are given a lot of features to choose from when buying a car or suit, after a while, we start asking for the default option rather than carefully weighing each decision.

This can also happen when faced with a decision in your creative work. Given the endless options of which route to take, we can sometimes end up going with the more conventional path simply because it's the easier way to go.


Limit your options.

Imposing your own constraints when trying to make a choice in your professional and creative work can help you make a better thought-out decision.If we restrict the choice of creative inputs it may actually enhance our creativity.

In other words, letting yourself have less options to choose from can help you arrive at a more creative answer.

A "key" question of your life

Someone recently asked me, “What do you want out of life?” After a long thought I said, “I want to be happy and have a great family and a job I like”. But, that answer didn't satisfy me. My thoughts were so ubiquitous that they didn't even mean anything.

Then came an interesting question, a question that perhaps I had never considered before, "what pain do I want in your life?" "What am I willing to struggle for?" Because that seems to be a greater determinant of how our lives turn out.

Everybody wants to have an amazing job and financial independence—but not everyone wants to suffer through 60-hour work weeks, long commutes, obnoxious paperwork, and navigate arbitrary corporate hierarchies. People want to be rich without the risk, sacrifice, and delayed gratification necessary to accumulate wealth.

Everybody wants to have great relationship—but not everyone is willing to go through the tough conversations, the awkward silences, the hurt feelings and the emotional psychodrama to get there. And so they settle. They settle and wonder “What if?” for years and years and until the question morphs from “What if?” into “Was that it?”

Happiness requires struggle. The positive is the side effect of handling the negative. You can only avoid negative experiences for so long before they come roaring back to life.

At the core of all human behavior, our needs are more or less similar. Positive experience is easy to handle. It’s negative experience that we all, by definition, struggle with. Therefore, what we get out of life is not determined by the good feelings we desire but by what bad feelings we’re willing and able to sustain to get us to those good feelings.

People want an amazing physique. But you don’t end up with one unless you legitimately appreciate the pain and physical stress that comes with living inside a gym for hour upon hour, unless you love calculating and calibrating the food you eat, planning your life out in tiny plate-sized portions.

People want to start their own business or become financially independent. But you don’t end up a successful entrepreneur unless you find a way to appreciate the risk, the uncertainty, the repeated failures, and working insane hours on something you have no idea whether will be successful or not.

People want a partner, a spouse. But you don’t end up attracting someone amazing without appreciating the emotional turbulence that comes with weathering rejections, building the sexual tension that never gets released, and staring blankly at a phone that never rings. It’s part of the game of love. You can’t win if you don’t play.

What determines your success isn’t “What do you want to enjoy?” The question is, “What pain do you want to sustain?” The quality of your life is not determined by the quality of your positive experiences but the quality of your negative experiences. And to get good at dealing with negative experiences is to get good at dealing with life.

There’s a lot of crappy advice out there that says, “You’ve just got to want it enough!” Everybody wants something. And everybody wants something enough. They just aren’t aware of what it is they want, or rather, what they want “enough.” Because if you want the benefits of something in life, you have to also want the costs. If you want the beach body, you have to want the sweat, the soreness, the early mornings, and the hunger pangs. If you want the yacht, you have to also want the late nights, the risky business moves, and the possibility of pissing off a person or ten thousand.

If you find yourself wanting something month after month, year after year, yet nothing happens and you never come any closer to it, then maybe what you actually want is a fantasy, an idealization, an image and a false promise. Maybe what you want isn’t what you want, you just enjoy wanting. Maybe you don’t actually want it at all.

So now I ask myself, “How do I choose to suffer?”  I ask because that tells me far more about me than m desires and fantasies. Because I have to choose something. You can’t have a pain-free life. It can’t all be roses and unicorns. And ultimately that’s the hard question that matters. Pleasure is an easy question. And pretty much all of us have similar answers. The more interesting question is the pain. What is the pain that you want to sustain?

That answer will actually get you somewhere. It’s the question that can change your life. It’s what makes me "me" and you "you". It’s what defines us and separates us and ultimately brings us together.

For most of my teenage, I fantasized about being a musician — a tabla player, in particular. Any concert I heard, I would always close my eyes and envision myself up on stage and people absolutely losing their minds to my mesmerizing performance. This fantasy could keep me occupied for hours on end. The fantasizing continued up through college, even after I graduated and started working and stopped playing seriously. I was biding my time before I could invest the proper amount of time and effort into getting out there and making it work. First, I needed to finish my college. Then, I needed to make money. Then, I needed to find the time. Then … and then nothing.

Despite fantasizing about this for over 12 years of my life, the reality never came and I was nowhere even close to it. And it took me a long time and a lot of negative experiences to finally figure out why: I didn’t actually want it.

I was in love with the result—the image of me on stage, people cheering, me rocking out, pouring my heart into what I’m playing—but I wasn’t in love with the process. And because of that, I failed at it. Repeatedly. Hell, I didn’t even try hard enough to fail at it. I hardly tried at all.

It was a mountain of a dream and a mile-high climb to the top. And what it took me a long time to discover is that I didn’t like to climb much. I just liked to imagine the top.

Our culture would tell me that I’ve somehow failed myself, that I’m a quitter or a loser. Someone would say that I either wasn’t courageous enough, determined enough or I didn’t believe in myself enough. The entrepreneurial/start-up crowd would tell me that I chickened out on my dream and gave in to my conventional social conditioning. I’d be told to do affirmations or join a mastermind group or manifest or something.

But the truth is far less interesting than that: I thought I wanted something, but it turns out I didn’t. End of story.

I wanted the reward and not the struggle. I wanted the result and not the process. I was in love not with the fight but only the victory. And life doesn’t work that way. Who you are is defined by the values you are willing to struggle for. People who enjoy the struggles of a gym are the ones who get in good shape. People who enjoy long workweeks and the politics of the corporate ladder are the ones who move up it. People who enjoy the stresses and uncertainty of the starving artist lifestyle are ultimately the ones who live it and make it.

This is not a call for willpower or “grit.” This is not another warning of “no pain, no gain.”

This is the most simple and basic component of life: our struggles determine our successes. So let us choose our struggles wisely, my friend or the struggles will keep on choosing us.

I hope you find some inspiration from this post as this post is also taken and inspired by something, something very deep.
Note: The thoughts in this blog are not completely mine but 90 percent relate to a post read and published somewhere else.

Top 10 interview Puzzles with solutions

This blog contains the my collection of top 10 puzzles asked in data science/analyst interviews. I thought of making a catalog so that they can be useful to one who is preparing for the interview.

So lets start this "Puzzling" journey.

1. Ant and Triangle Problem

Three ants are sitting at the three corners of an equilateral triangle. Each ant starts randomly picks a direction and starts to move along the edge of the triangle. What is the probability that none of the ants collide?

Solution

Let’s start by reversing the way the riddle is asked and find the probability that there willnot be a collision instead. When will this happen?

Well, what if all of the ants are walking in the same direction? Then there will never be a collision between any of the ants because they are all walking in the same direction. And, the only time they will be walking in the same direction is if they are all walking either counter-clockwise around the triangle, or clockwise around the triangle. So, there are only two scenarios in which a collision will not happen between the ants.

Now that we know that there are only two scenarios where the ants will not collide, we have to ask ourselves how many different ways are there for the ants to move on the sides of the triangle? Well, each ant can move in 2 different directions. Because there are 3 ants, this means that there are 23 (which equals eight) possible ways that the ants can move. And since we already know that there are only 2 ways in which the ants can avoid collision entirely, this means that there are 6 scenarios where the ants will collide. And 6 out of 8 possible scenarios, means that the probability of collision is 6/8, which equals 3/4 or .75. Thus, the probability of the ants colliding is .75.

2. Crossing the Bridge Puzzle

Four people need to cross a rickety bridge at night. Unfortunately, they have only one torch and the bridge is too dangerous to cross without one. The bridge is only strong enough to support two people at a time. Not all people take the same time to cross the bridge. Times for each person: 1 min, 2 mins, 7 mins and 10 mins. What is the shortest time needed for all four of them to cross the bridge?

Solution : 

 The solution is to have W(10 mins) carry the torch and begin to cross with any of the others lets say X(1). Each continues at their own pace so in 1 minute X has reached the other side and one of the others can begin to cross say Y(2), when he reaches the other side Z(7) can begin to cross. W and Z will cross the bridge at the same time i.e. at the end of 10th minute.

This solution satisfies the criteria that there are never more than 2 people on the bridge and the torch is always on the bridge whilst someone is crossing. So in that sense it is a neat solution.

3. Burning Rope Problem

A man has two ropes of varying thickness (Those two ropes are not identical, they aren’t the same density nor the same length nor the same width). Each rope burns in 60 minutes. He actually wants to measure 45 mins. How can he measure 45 mins using only these two ropes.

He can’t cut the one rope in half because the ropes are non-homogeneous and he can’t be sure how long it will burn.

Solution :

Take one rope and burn it at both ends.At the same time, burn one end of the other rope. When the first rope is burnt (i.e. 30 minutes), light the other end of the other rope (half of the

remaining 30 minutes gives you 15 minutes) When it burns out, that`s 45 minutes.

4. Heaven’s Gate Problem

You are standing before two doors. One of the path leads to heaven and the other one leads to hell. There are two guardians, one by each door. You know one of them always tells the truth and the other always lies, but you don’t know who is the honest one and who is the liar. You can only ask one question to one of them in order to find the way to heaven. What is the question?

Solution : 

Ask one of the guard:" what would the other guard say if i ask him which way is the hell?" 

And whatever answer he gives that is the way to heaven. 

If you end up asking the question to the truthful one, he will tell the true and he knows the other guard is going to lie so he will so the way to heaven. 

If you end up asking the question to the liar he will lie what the answer will be and he lies and instead shows the way to heaven.

5. 10 Coins Puzzle

You are blindfolded and 10 coins are place in front of you on table. You are allowed to touch the coins, but can’t tell which way up they are by feel. You are told that there are 5 coins head up, and 5 coins tails up but not which ones are which. How do you make two piles of coins each with the same number of heads up? You can flip the coins any number of times.

Solution :

Make 2 piles with equal number of coins. Now, flip all the coins in one of the pile.

How this will work? lets take an example.

So initially there are 5 heads, so suppose you divide it in 2 piles.

Case:

P1 : H H T T T

P2 : H H H T T

Now when P1 will be flipped

P1 : T T H H H

P1(Heads) = P2(Heads)

Another case:

P1 : H T T T T

P2 : H H H H T

Now when P1 will be flipped

P1 : H H H H T

P1(Heads) = P2(Heads)

6. King and Wind Bottles Puzzle

A bad king has a cellar of 1000 bottles of delightful and very expensive wine. A neighboring queen plots to kill the bad king and sends a servant to poison the wine. Fortunately (or say unfortunately) the bad king’s guards catch the servant after he has only poisoned one bottle. Alas, the guards don’t know which bottle but know that the poison is so strong that even if diluted 100,000 times it would still kill the king. Furthermore, it takes one month to have an effect. The bad king decides he will get some of the prisoners in his vast dungeons to drink the wine. Being a clever bad king he knows he needs to murder no more than 10 prisoners – believing he can fob off such a low death rate – and will still be able to drink the rest of the wine (999 bottles) at his anniversary party in 5 weeks time. Explain what is in mind of the king, how will he be able to do so ?

Solution :

Think in terms of binary numbers. (now don’t read the solution, give a try).

Number the bottles 1 to 1000 and write the number in binary format.

bottle 1 = 0000000001 (10 digit binary)

bottle 2 = 0000000010

bottle 500 = 0111110100

bottle 1000 = 1111101000

Now take 10 prisoners and number them 1 to 10, now let prisoner 1 take a sip from every bottle that has a 1 in its least significant bit. Let prisoner 10 take a sip from every bottle with a 1 in its most significant bit. etc.

prisoner = 10 9 8 7 6 5 4 3 2 1

bottle 924 = 1 1 1 0 0 1 1 1 0 0

For instance, bottle no. 924 would be sipped by 10,9,8,5,4 and 3. That way if bottle no. 924 was the poisoned one, only those prisoners would die.

After four weeks, line the prisoners up in their bit order and read each living prisoner as a 0 bit and each dead prisoner as a 1 bit. The number that you get is the bottle of wine that was poisoned.

1000 is less than 1024 (2^10). If there were 1024 or more bottles of wine it would take more than 10 prisoners

7. Mislabeled Jar Puzzle

You have 3 jars that are all mislabeled. One jar contains Apple, another contains Oranges and the third jar contains a mixture of both Apple and Oranges.

You are allowed to pick as many fruits as you want from each jar to fix the labels on the jars. What is the minimum number of fruits that you have to pick and from which jars to correctly label them?

Solution

Let’s take a scenario. Suppose you pick from jar labelled as Apple and Oranges and you got Apple from it. That means that jar should be Apple as it is incorrectly labelled. So it has to be Apple jar.

Now the jar labelled Oranges has to be Mixed as it cannot be the Oranges jar as they are wrongly labelled and the jar labelled Apple has to be Oranges.

Similar scenario applies if it’s a Oranges taken out from the jar labelled as Apple and Oranges. So you need to pick just one fruit from the jar labelled as Apple and Oranges to correctly label the jars.

8. Red and Blue marbles Puzzle

You have two jars, 50 red marbles and 50 blue marbles. You need to place all the marbles into the jars such that when you blindly pick one marble out of one jar, you maximize the chances that it will be red. When picking, you’ll first randomly pick a jar, and then randomly pick a marble out of that jar. You can arrange the marbles however you like, but each marble must be in a jar.

Solution :

Seems tricky at first right? Given that the number of red and blue marbles are the same, you would tend to think that the odds are 50-50. You would try different combinations, such as 25 of each colored marble in a jar or putting all red marbles in one jar and all the blue in the other. You would still end up with a chance of 50%.

So lets think of a better way to distribute the marbles. What if you put a single red marble in one jar and the rest of the marbles in the other jar? This way, you are guaranteed at least a 50% chance of getting a red marble (since one marble picked at random, doesn’t leave any room for choice).  Now that you have 49 red marbles left in the other jar, you have a nearly even chance of picking a red marble (49 out of 99).

So let’s calculate the total probability.

P( red marble ) = P( Jar 1 ) * P( red marble in Jar 1 ) + P( Jar 2 ) * P( red marble in Jar 2 )

P( red marble ) = 0.5 * 1 + 0.5 * 49/99

P( red marble ) = 0.7474

Thus, we end up with ~75% chance of picking a red marble.

9. Gold Bar Problem

You’ve got someone working for you for seven days and a gold bar to pay him. The gold bar is segmented into seven connected pieces. You must give them a piece of gold at the end of every day. What and where are the fewest number of cuts to the bar of gold that will allow you to pay him 1/7th each day?

Solution :

The trick is not to try and how to cut in such a way to make 7 equal pieces but rather to make transactions with the worker. Make two cuts on the gold bar such that you have the following sizes of bars.

1/7, 2/7 and 4/7. For convenience sake, I would just refer to the bars as 1, 2 and 4.

At the end of Day 1: Give Bar 1 (You- 2 and 4, Worker- 1)

At the end of Day 2: Give Bar 2, Take back Bar 1 (You- 1 and 4, Worker- 2)

At the end of Day 3: Give Bar 1 (You- 4, Worker- 1 and 2)

At the end of Day 4: Give Bar 4, Take back Bar 1 and Bar 2 (You- 1 and 2, Worker- 4)

At the end of Day 5: Give Bar 1 (You- 2, Worker- 1 and 4)

At the end of Day 6: Give Bar 2, Take back Bar 1 (You- 1, Worker- 2 and 4)

At the end of Day 7: Give Bar 1 (You- Empty, Worker- 1, 2 and 4)

10. 100 Doors Puzzle

You have 100 doors in a row that are all initially closed. you make 100 passes by the doors starting with the first door every time. the first time through you visit every door and toggle the door (if the door is closed, you open it, if its open, you close it). the second time you only visit every 2nd door (door #2, #4, #6). the third time, every 3rd door (door #3, #6, #9), ec, until you only visit the 100th door. What state are the doors in after the last pass? Which are open which are closed?

Solution :

For example, after the first pass every door is open. on the second pass you only visit the even doors (2,4,6,8…) so now the even doors are closed and the odd ones are opened. the third time through you will close door 3 (opened from the first pass), open door 6 (closed from the second pass), etc..

what state are the doors in after the last pass? which are open which are closed?

You can figure out that for any given door, say door #42, you will visit it for every divisor it has. so 42 has 1 & 42, 2 & 21, 3 & 14, 6 & 7. so on pass 1 i will open the door, pass 2 i will close it, pass 3 open, pass 6 close, pass 7 open, pass 14 close, pass 21 open, pass 42 close. for every pair of divisors the door will just end up back in its initial state. so you might think that every door will end up closed? well what about door #9. 9 has the divisors 1 & 9, 3 & 3. but 3 is repeated because 9 is a perfect square, so you will only visit door #9, on pass 1, 3, and 9… leaving it open at the end. only perfect square doors will be open at the end.

Puzzles are good way to exercise your brain. I hope this was a good start.

Data Warehouse vs Data Lake

In my last post we talked about Data Lake and its need and advantages. In this post we will compare Data Lake with Data Warehouse.

The emergence of the data lake has periodically led to a somewhat regrettable dialogue that compares data warehouses unfavorably with data lakes. These kinds of discussions are inevitable as a new, hot concept elbows the previous decade’s paradigm out of the way. But, it’s a misleading thought process. While data lakes can accomplish a number of things that are very difficult for a data warehouse to do, the opposite is also true. The two technologies are inherently different and serve different enterprise needs. One is not necessarily better than the other. In some use cases, the data lake can be quite deficient when contrasted with a data warehouse.

A data warehouse is a carefully designed data store that contains data from pre-selected sources, such as enterprise resource planning (ERP) and logistics systems. Data warehouses tend to emphasize transactional / structured data over semi-structured and unstructured data. The data in a warehouse is usually organized using multi-dimensional, star or snowflake schemas in order to streamline execution of queries, reports, dashboards or the running of advanced analytical models.In Data Warehouse (DW), schema is defined before data is stored. This is called “Schema on WRITE” or required data is identified and modeled in advance. The quality of data that exists in a traditional Data Warehouse is cleansed whereas typical data that exist in Data Lake is Raw.

A data lake (typically implemented using Hadoop) is a mix of structured, semi-structured or unstructured data. For example, transactions, spreadsheets, Documents, sensor data, images, social media, etc. may all be stored in the data lake). The data lake may be fed using traditional-style batch jobs or by connecting the data lake to real-time data feeds.In Data Lake the schema is defined after the data is stored. This is called “Schema on READ”. So the data must be captured in code for each program accessing the data. Data Lake provides cheaper storage of large volumes of data and has potential to reduce the processing cost by bringing analytics near to data.The data lake gives business users immediate access to all data. They don’t have to wait for the data warehousing (DW) team to model the data or give them access. Rather, they shape the data however they want to meet local requirements. The data lake speeds delivery which is required in a dynamic market economy.Data Lakes offers unparalleled flexibility since nobody or nothing stands between business users and the data.

Pertinence in Big Data world: 

Traditional approach of manually curated data warehouses, provides limited window view of data and are designed to answer only specific questions identified at the design time. This may not be adequate for data discovery in today’s big data world. Moreover, data lake can contain any type of data – clickstream, machine-generated, social media, and external data, and even audio, video, and text. Traditional data warehouses are limited to structured data. The data lake can hold any type of data.

The question should not be, “Which one is better?” Rather, you should ask, “Which one is better for my scenario, given my unique needs?” The suitability of one over the other will depend on many factors that are particular to an organization.

A Data Lake

Recently, I came across a new term which is a product of evolving data science. The term is "Data Lake". With data multiplying on on a daily basis this term adds a different dimension to data science and has already created a lot of excitement. The data lake concept has quickly gained traction in the world of big data.

So, in this blog I wish to demystify the concept.

The term “data lake” uses a metaphor to simplify an approach to storing data for analysis that is abstract and cumbersome to describe in plain, factual terms. James Dixon, “Chief Geek” at Pentaho, is credited with coining the phrase. Dixon posted that each specialized data mart in a data warehouse could be likened to a bottle of water. The data was ready for use in a small, identifiable container. In contrast, a data “lake” is a massive, intermingled repository of all data in its raw form.

Need of Data Lake :

The reason that data lake concept came into picture was to adopt to the freewheeling, deep analysis, ask-anything ethos of big data. It’s harder to retool a data warehouse to do the kind of wide-ranging data correlation that big data solutions make relatively easy. 

Moreover, It is estimated that a staggering 70% of the time spent on analytics projects is concerned with identifying, cleansing, and integrating data. Data is often difficult to locate because it is scattered among many business applications and business systems. The data needs re-engineering and reformatting frequently in order to make it easier to analyze.

The data must be refreshed regularly to keep it up-to-date.Acquiring data for analytics in an ad hoc manner creates a huge burden on the teams that own the systems supplying data. Often the same type of data is repeatedly requested and the original information owner finds it hard to keep track of who has copies of which data.

As a result, implementing a data lake can be a solution. A data lake is a set of one or more data repositories that have been created to support data discovery, analytics, ad hoc investigations, and reporting. The data lake contains data from many different sources. People in the organization are free to add data to the data lake and access any updates as necessary.

In my next blog I will talk about Data Lake vs Data Warehouse.

A tale of a nation that Imports Garbage

Global waste production has doubled over the past ten years. By 2025, the world will produce garbage at a rate of 2.5 billion tonnes per year. Poor waste management in many developing countries is a threat to human health and the environment. Many developing countries struggle towards a sustainable model for waste collection and disposal.Despite large expenditures many nations have failed to take care of this problem and India is one of them. 

As we say this,Something incredible has been taking place in Sweden over the past several years, somewhat of a “recycling revolution,” . Currently less than one percent of the garbage produced in Swedish homes ends up in the landfill today, with the other ninety-nine percent being recycled or composted.

Sweden has been known for years now for the amazing and resourceful waste management system that they have had in place for some time. They have 32 waste-to-energy (WTE) plants and this burned waste powers 20 percent of Sweden’s district heating as well as electricity for about 250,000 Swedish homes.

Here’s the problem: Swedes have become so good at recycling that there’s no longer enough garbage to meet the heating plants’ needs. Sweden now has to import the trash that most other countries are trying to dispose of — some 800,000 tons in 2014, up from 550,000 tons in 2010.

For Sweden, Waste today is a commodity in a different way than it has been. It’s not only waste, it’s business. For now, Sweden imports its trash mostly from Britain and Norway.

Waste to Energy process (In brief)

After recyclable content has been removed, the garbage is placed in incinerators that produce heat or energy, which is then transported to nearby homes. From the ashes, small pieces of metal, which do not burn, are separated and recycled, while those of porcelain and tile are sifted to extract gravel, which is used in road construction. The remaining one percent goes into landfills. And though garbage-infused smoke sounds highly poisonous, thanks to electric filters that give the particles a negative electric charge, in Sweden the smoke is almost entirely nontoxic carbon dioxide and water, which are cleaned again before release. 

The Environmental Effects

Of course there is some controversy over this method of waste management and energy production. There are fumes that are produced that are toxic, but many argue that this is still a much better alternative to the typical landfills that we see more often in our country. On average, more than forty percent of the world’s trash is burned and mostly in open air, this is much different from the regulated, low-emission process that has been adopted by Sweden. This makes Sweden’s method a lot more eco-friendly.

It is unfortunate that at this time the options are to either pollute the air, or pollute the Earth because we are producing so much garbage and it doesn’t just simply disappear without leaving its mark. It doesn’t look like manufacturers are going to stop making products that can’t be recycled anytime soon, so it is up to us to be more conscious with what we are consuming. We need to stop the problem at the source.

How Can We All Be A Bit More Like The Swedish? 

Sweden is an excellent example of a nation of citizens that care for their environment and their ecological footprint. If the Swedish are able to recycle 99% of their waste, why can’t we? Well we certainly can, however it does take a bit more effort(and that is where the problem lies with this country). I believe that it is truly worth the effort, soon it will become so natural to us that it won’t even feel effortful, we just have to start taking that step!

Dear "Import and Export Control Department of India", Can you please ask Sweden to come and get some trash from us as we've lots it and we haven't figured out what to do with it yet.

Although "Swachh Bharat Abhiyan" is an excellent initiative but the question is, Is it good enough?