Read my latest feature article for The Independent (front page story) and the “i” in full here or click here.

The friendly big red capital letters of the email welcoming me to the world of online gambling – “WELCOME ABOARD. GAME ON” – were in marked contrast to the Orwellian feel of the site’s age-verification policy that I had found just a click away.

It promised to use a third party to electronically verify that I was who I said I was by using my name, address, date of birth and phone number, and said the results might be retained for use by other businesses in the future. It meant that whatever I shared and whatever they had on me was going to remain forever out there in cyberspace, beyond my control.

And if I’d failed the test, I may even have had to verify my identity through a Skype interview.

According to John Aristotle Phillips, speaking at this year’s Silicon Valley Comes to Oxford event, this experience is likely to become more common as mainstream brands such as Tesco seek to follow the lead set by the “sin” industries of gambling, drinking and tobacco in applying one of the main lessons of Barack Obama’s victory last month – the effectiveness of big data.

“Big data” is a catch-all term for the collection, analysis and use of very large and complex sets of data. By asking people to volunteer their personal data, and then combining it with publicly available data, including what they post on social networks, Obama’s team was able to build a profile of individuals and their habits, made up of 50 or so individual data points, and target them more effectively.

So good were they at this nano-targeting that as one analyst put it, “the Obama campaign was able to identify voters that Romney’s camp didn’t even know existed”. “While some companies have been doing this effectively, many haven’t,” Phillips says.

Phillips is co-founder of the market-leading political technological consultancy Aristotle Inc, and has advised every US president since Reagan plus a few British prime ministers. His interest in the value of publicly available data began as an undergraduate at Princeton, when he used documents in the library to design an atomic bomb that the Pakistanis promptly tried to buy – earning himself the label “the A-bomb kid” in the process. Aristotle provides its integrity identity verification software to “seven out of 10 UK online gambling sites, and it’s soon to be eight”.

“I have always been surprised by how many businesses in the UK let visitors leave their sites without giving them much in the way of personal information,“ says Phillips. “There has been a misconception that if you ask people who they are they will run away from your site. Now though more companies are embracing identify verification.” Yet it is questionable how much people understand what this data could be used for.

For Phillips, what has made business “sit up and take notice” of the Obama campaign was the effective way they used a centralised database called Narwhal. It collected all the data the campaign had on an individual to build in effect a one-to-one relationship with the voter, enabling them to make decisions about the conduct of the campaign on a street-by-street basis.

Through its Facebook app, the Obama campaign asked voters to share their Facebook friends with the campaign so that potential Obama voters could be identified. According to recent figures from the campaign, 20 per cent of targeted friends responded in the “correct way”.

Charles Duhigg is a New York Times journalist and author of The Power of Habit. Like Phillips, he believes that “lots of people gave up their data during the election because all Obama was trying to do was to get his vote out. Whether they knew this meant they were going to get targeted – or would have cared if they did – is much harder to tell.”

Unlike Phillips, however, Duhigg believes that “this is going to be much harder to do for someone like Tesco, as people don’t tend to share with companies as they do with parties”. It is also going to be harder for any business to reproduce the effect in Europe, “as the truth is, in the US we just don’t get as worked up by privacy as the Europeans do”.

Daniel Kreiss, author of Taking Our Country Back and assistant professor at the School of Journalism and Mass Communication, University of North Carolina at Chapel Hill, is also sceptical as to whether people will hand over data in the same way to high-street brand names. For him, it is more a question of whether businesses in the UK will be able to use data to the same degree of effectiveness as Obama’s campaign did.

Phillips disagrees. “I think it’s the other way around, and I’m in the political business. I think most people tune in only during election season. Rest of the year, they want to be able to do the things that identity verification will facilitate in their day-to-day lives.”

However, for Jorn Lyseggen, founder and CEO of the global software company Meltwater Group, “what matters for business is not so much any particular technology or algorithm Obama’s team used, what matters is what they did worked”.

“Obama’s effective use of data mining has been a real proof point for the power of the concept of big data, as its accuracy astonished the world and baffled scientists, businessmen and analysts.”

“Now the penny has dropped and business executives are going to take the insights big data offers into their company seriously, as they can clearly see what a rather difficult concept actually means in concrete terms.”

Like Phillips, Lyseggen believes that up to now there has been a lot of uncertainty as people have been asking how it can create value for them. He thinks the success of the Obama campaign presents business with an “extraordinary opportunity” to be able to analyse in real time their customers’ attitudes to them and to their competitors.

For Duhigg and Kreiss it marks the death of intuitive leadership in business. No one is going to follow their gut and disagree with the data again.

Despite the feeling that this is something Orwell dreamt up, Phillips thinks the spread of identity verification comes down to Darwin. Business leaders in the sin industries and politicians in particular “adapt quickly to what works and compete very hard”.

“While they want what works in 2012, they are going to want to know what wins in 2014 or 2016.” And the rest of the world will follow. Lucky us?