With an increased interest in big data, the amount of information that businesses collect has been steadily increasing over the years. In order to capture more data, companies are investing in technology and talent to leverage the value of this information.
Despite these efforts, it is estimated that 60-73 percent of all enterprise data goes unused for analytics. In some cases, the organization may not even be aware that useful data is being created. In fact, in the manufacturing industry, it is estimated that 90 percent of data generated by sensors and analog-to-digital conversions never get used.
If companies can better leverage their unused information and other forms of “dark data,” it may make a big difference in improving operations and making key business decisions.
What is Dark Data?
Dark data is information that is all but unusable due to certain circumstances, such as its location, quantity, or the resources needed to collect and/or analyze it. It can be broken down into three different types:
- Untapped Internal Data: Information that an organization already collects, processes, and stores during regular business activities, but generally remains unused for anything more than a single purpose. Many companies store all of the data that they generate in spite of not knowing what they are going to do with it.
- Nontraditional Unstructured Data: Data attached and related to audio, video, and image files. This media data could not be explored prior to the development of more advanced technologies, such as computer vision, advanced pattern recognition, and video and sound analytics.
- Deep Web Data: Data that is often hidden behind firewalls and requires specialized tools, vendors, or techniques to collect and analyze it.
Examples of Dark Data
Specific examples of dark data are wide-ranging and largely depend on our company and industry, but here are some that could fall into this category if they are outdated, unstructured, or unutilized:
- Log files (servers, systems, architecture, etc.)
- Previous employee data
- Financial statements
- Geolocation data
- Raw survey data
- Surveillance video footage
- Customer call records
- Email correspondences
- Notes, presentations, or old documents
How could these examples of dark data provide value to a company? Here’s a few examples. Server log files could provide clues to website visitor behavior. Footage from a surveillance video or recordings of customer calls could provide unstructured customer sentiment data. Geolocation data can offer new insight into your shipping and logistic operations. Whatever the view may be, the key is that it is new and untapped information. Dark data analysis encourages you to take another look at this data to see if there is any additional value or insight to gain.
The Benefits of Analyzing Dark Data
The International Digital Corporation predicts that organizations that can analyze all relevant data and deliver actionable information could achieve an extra $430 billion in productivity gains over their peers by 2020. They have the opportunity to take information that was previously hidden or unknown and turn it into powerful insights, leading to new opportunities, reduced risk, and increased return-on-investment (ROI).
On the flip side, not knowing the best way to apply dark data can actually cost businesses. As more companies begin to take advantage of their previously untapped data, those who are not may encounter lost revenue opportunities, lower efficiency, quality issues, and diminished productivity. Whether you’re motivated by maximizing gains or minimizing losses, it might be time to shed more light into the corners of your data and see what there is to be seen.
The Problems with Dark Data
Dark data needs to be brought into the light through dark data analytics in order to be useful. If you are not actively analyzing and using your dark data, it becomes useless clutter that is taking up valuable space. As you the space required for your data grows, the storage costs and security risk increase as well.
Dark data may contain some surprisingly sensitive information, and it is more susceptible to data breaches than the data that is being more closely monitored. You may not even notice that there was a data breach at first. Some dark data won’t have the same untapped potential as others. However, companies still need to protect, manage, and organize that information by implementing a process that gets rid of old unneeded data.
Read next: How to Spot and Stop Bad Data
How to Take Advantage of Dark Data
While it will be different for every company, based on their data architecture, here are some actionable tips to turn dark data into rich insights and opportunities:
- Ask questions: Determine what information will help you make better business decisions early on and let that guide your analysis and decisions.
- Audit your database: Take a look at your data sources and data collection tools and strategize what changes you need to make. It’s important to slow the build-up of new dark data by quickly identifying what is valuable and what isn’t. Don’t store data “just because,” make sure everything has a purpose.
- Expand your team: What good is dark data if you are unable to translate it? Build a solid analytics team that encompasses organizational, business, and technical knowledge.
- Invest in the right tools: Video and sound analytics, computer vision, machine learning, and advanced pattern recognition are tools and techniques that may help illuminate dark data.
- Keep goals in mind: Analytics is business-driven, so determine upfront what value must be delivered from your efforts. If some employees are reluctant to deal with dark data, it helps to tie your efforts back to at least one of the company’s goals or objectives.
- Think big: As new strategies for using dark analytics are developed, consider how you can apply them to your own efforts without biting off more than you can chew.
Bringing Visibility to Dark Data
Data visualization technologies can connect all of your data sources and present them in a single dashboard, providing real-time visibility to compiled data. This tool can be leveraged to help users sift through their dark data to uncover the information they need.
Instead of looking for a needle in a haystack, the most important information will stand-out if you use the right charts and graphs on your dashboard. Once organizations gain insight from their visualized dark data, they can use it to make quick, informed decisions.
