Big data is still oil
Big data still gets a lot of attention. This is mostly because digital information is still growing at an increasing pace.
Businesses and organisations seek insights from this mountain of information. The term big data has been used to refer to the collection, storage, and analysis of these large and complex data sets. The availability of accessible technology for handling these tasks has been growing at a similar pace.
These tools provide insights that can help improve decision-making and drive business growth. However, despite the many potential benefits of big data, reports as of 2019 reported that 85% of data science projects fail. This is likely to have changed in recent times since companies are becoming more experienced. But I’d guess not by much.
Like oil did IRL, big data has transformed our digital landscape. But the rampant collection of big data comes with downsides.
The challenges of traditional big data
Some of the limitations to traditional big data approaches are:
Centralised control: data is typically collected and controlled by a few large corporations, which can lead to privacy and security issues. This makes most big data projects a black box to consumers.
Lack of ownership: Consumers often have little control over their data, which companies collect and use with or without their consent. This can end up in a loss of consumers’ trust.
Inaccuracy: Big data is often collected and analysed in aggregate. It also grows stale. Inaccurate or biased conclusions based on poor or generalised data can noticeably affect customers. For example, a stale or inaccurate credit score could leave would-be homeowners without any means to obtain a mortgage.
Management and analysis are difficult: As dataset volumes continue to grow, it becomes increasingly challenging for businesses to manage and make sense of the vast amount of information they collect. The vast number of tools could lead to analysis paralysis or just be ignored in favour of the less efficient choices.
Lack of interoperability: In the traditional model, data is often siloed and not easily shared or integrated with other data sources, which limits its utility and therefore, its value.
But there are alternatives.
The rise of self-sovereign data
As data privacy and security concerns have grown, self-sovereign data has also gained traction as a potential solution to our woes.
Instead of a few large companies collecting and controlling our data, self-sovereign data gives us control. We choose how our data is used and shared. This model is often associated with blockchain technology.
Blockchain
We won’t go into the details here, but blockchains allow self-sovereignty by design.
Doing so enables the creation of secure and transparent data marketplaces where individuals can share their data with businesses in exchange for value. This provides a new way for people to monetise their data and for businesses to access high-quality, relevant information.
Of course, in self-sovereignty, you must own your keys to have true custody of your data. It doesn’t work if you’ve given custody to a centralised entity like FTX Coinbase. To paraphrase Tezor, a hardware ledger provider that advocates self-custody:
Solid
Another technology worth mentioning is Solid.
The Solid project is another open-source platform that aims to give individuals control of their data with the ability to share it on their terms. Developed by Tim Berners-Lee, the inventor of the World Wide Web, Solid is built on the principles of self-sovereign data.
The platform allows users to store data in personal "pods" and grant access to various applications.
Social media
It’s worth mentioning that social media is a popular topic of conversation regarding decentralised data. Twitter alternatives are becoming popular discussion points (on Twitter, funnily enough).
Mastodon is a social network that runs on decentralised servers. This means that no single entity controls the running of the service. This gives some security that data will continue to exist for as long as at least one server hosts your social data. Worst case, you can always run your own server.
Another alternative is the nostr protocol. Even though nostr isn’t a social media platform itself, the technology can power a decentralised social network where every participant runs their own client. There are a few compatible clients available to try out.
It’s all decentralisation
Solid pods, Mastadon servers and nostr clients work very differently to blockchains, especially the proof-of-work ones. But all these projects aim to create a more decentralised and secure web where people control their information. This directly benefits individuals, but it also gives bussinesses new opportunities give value to their customers.
The benefits of self-sovereign data for businesses and consumers
Ways that self-sovereign data is a win for everyone:
Greater control and ownership: In a self-sovereign data model, people have complete control over their data and can share it with businesses on their terms. This can help build trust and improve relationships between consumers and businesses.
Improved accuracy and relevance: Businesses can access more accurate and relevant information by giving people control over their data, leading to better decision-making and more personalised services.
Increased security and privacy: Self-sovereign data models can help protect personal information from unauthorised access or misuse, which makes for better privacy and security.
Greater interoperability and flexibility: Self-sovereign data can be easily shared and combined with other data sources, increasing its value and utility.
Potential cost savings: By giving people control over their data, businesses can avoid the costs of collecting and storing a lot of data.
How will businesses make money?
Business models will no doubt have to change, but services will ultimately be more valuable for their customers.
There are a few ways that businesses can make money by pivoting to self-sovereign data models, including:
Data-driven services: By using insights from self-sovereign data, businesses can develop new products and services tailored to individual consumers' specific needs and preferences.
Data-driven advertising: Businesses can use self-sovereign data to target their advertising better, resulting in more effective and personalised marketing campaigns. People prefer relevant contextual ads anway.
Data brokerage: Businesses can act as intermediaries by enabling the movement of data between different parties and generating revenue from these transactions.
Data storage and management: Businesses can offer services for storing and managing self-sovereign data, generating revenue from these services. Managed services break my brain a little, but it’s a thing!
Conclusion
Some companies will find it painful or impractical to shift their business models from big data to self-sovereign data. A notable example is AI. Thanks to the adoption of transformer architectures, we’ve recently seen an explosion in the popularity of AI tools. Stable Diffusion uses LAION-2B, the 5.85 billion-pair dataset to train its model. Similarly, massive Internet datasets went into training ChatGPT.
But like the transition away from oil and into renewable energy, self-sovereign data might just be the next step in our cultural evolution.
Self-sovereign data is a new concept and it will take time for people to get on board. Consumer education will need to play a big role. So, both individuals and businesses need to change how they think about and use data.
To their benefit, businesses can access more accurate and relevant information, while individuals enjoy increased security and privacy. Companies that embrace these models will have an easier time complying with data protection laws like GDPR and CCPA. They’ll also potentially save a huge amount of money that would otherwise be spent on massive data collection and stale data sitting in underused siloes.
Despite the challenges, self-sovereign data is well worth it. It has the potential to improve our experiences, make products and services more efficient and drive innovation.
Ultimately, it makes for happier customers and more successful businesses. We should choose to embrace this change.