You can afford to store your surveillance data: Storage management basics

The Boston Marathon bombing turned the klieg lights on surveillance video. More people were taking photos and videos of the race finish line than of possibly any other location in the world at that moment, so the Boston Police Department (BPD) requested video and photos from businesses and spectators to help find the attackers. The BPD and Federal Bureau of Investigation searched thousands of videos and images for information; ultimately, video from a Lord & Taylor store surveillance camera proved instrumental in identifying the two brothers who carried out the attack.

Businesses have long relied on surveillance data, and recent events are prompting many municipalities, including Boston, to increase the number of surveillance cameras. This increase in the number of cameras produces an increased amount of video data and metadata—such as the data and time of the video or image capture—as well as the need to store it all. New generations of cameras, video management software, and storage solutions are available to facilitate the storage of video data and to provide access when needed. Understanding some of the surveillance market’s challenges, and relative advantages of today’s top storage technologies, including disk and tape, can help surveillance storage system managers minimize their solution’s cost and architectural complexity, while maximizing storage capacity and flexibility. 

A growing appetite for video storage

“I don’t necessarily agree with Moore’s Law,”—the observation that computing capacity, or the number of transistors on integrated circuits, doubles every 18 months—“but it’s very applicable to storage,” says Business Development Manager Vince Ricco at Axis Communications, which manufactures cameras and develops video management software systems. “We have a growing appetite for longer storage of video.” The question is how to store that video. Decisions include whether to rent storage (cloud) or own it (on-site disk and tape), the length of time to store the data, the resolution of the stored data, and a method for finding and readily accessing data after it’s stored.

Before these storage techniques can be implemented, however, the surveillance storage requirements must be reasonably well characterized. Retention requirements are a key consideration when designing the storage system. Chief spokesman Paul J. Browne of the New York Police Department (NYPD) says under its privacy policy, the department discards images after 30 days, unless the images are part of an active investigation.

More organizations are moving toward keeping large quantities of data, but may be using applications to reduce the frame rates of older video. As rate storage capacities expand and costs decline, the NYPD may already be able to retain older files for longer periods—and encrypt the older files to comply with their privacy requirements.

Video-aware information management

Today, the surveillance infrastructure marketplace offers far more dynamic, easily scalable, and efficient alternatives to this inflexible strategy than ever before. The practice of Information Lifecycle Management (ILM) is defined as the strategies based on risk, business rules, and events used to administer storage systems that manage business data and metadata from the moment of creation and initial storage to the time it is deleted. ILM principles can be applied to storage management, allowing organizations to retain surveillance video over its useful life, at appropriate file sizes and frame rates, for as long as the video data is needed. Now, “video-aware” software with Video Lifecycle Management (VLM) capabilities is available to address these challenges.

Managing video throughout its lifecycle

The available storage platforms for video data have typically been limited to disk storage. Everything has been designed for disk.

By expanding storage to include tape along with disk, however, storing video surveillance data becomes increasingly affordable. An effective VLM implementation is one in which newer video data and videos that may be accessed more frequently are stored on faster and more expensive storage media, while remaining video data is stored on a less expensive storage medium, moved offline, or deleted. This reduces the size and cost of storing images over time, and reduces risk without sacrificing retention time, discarding data, or creating video gaps.

With a good VLM software application and the use of some disk and tape, it’s possible and affordable to implement a storage strategy for video surveillance data, reducing the size and cost of storing images without sacrificing resolution, duration, or creating video gaps. Alternately, some choose cloud storage for their surveillance data.

Cloud storage, video surveillance as a service

Organizations are increasingly interested in cloud-based services and Video Surveillance as a Service (VSaaS), says Ricco, adding that it isn’t easy to identify any single technology in what he calls the “nebulous” idea of the cloud. IT managers are gaining confidence in security and data integrity as they move data to and from public or private clouds.

The cloud has drawbacks, however, including the bandwidth required to stream video data, and the costs involved in moving or accessing data that has been stored in a public cloud. For large amounts of video and other data, “the current monthly cost to rent storage [from cloud-based storage providers] is roughly equivalent to the amount that would be needed to purchase it outright,” says John Villasenor, Senior Fellow, Governance Studies, Center for Technology Innovation at Brookings, and professor of electrical engineering and public policy at University of California, Los Angeles, in a 2011 paper about digital storage growth.

In-house is possible: the plummeting costs of data storage

The ubiquity of the term “Big Data” is already desensitizing us to its staggering implications. Data production is projected to be 1.8 zettabytes this year. Not many people can envision a zettabyte, which is 1,000 exabytes, or 1,000,000 petabytes, or one billion terabytes, if that helps. The quantity of data generated in 2020 is projected to be 44 times greater than it was in 2009, and the growth curve is exponential. A highly scalable storage solution is the only way to prepare for the inevitable.

Once, organizations like the NYPD—and incidentally, many totalitarian governments, according to Villasenor—had to choose the data they wanted to retain because the cost of keeping all their files was too high. Villasenor says this is no longer true.

“Over the past three decades,” says Villasenor, “storage costs have declined by a factor of 10 approximately every four years, reducing the per-gigabyte cost [of storing data on disk] from approximately $85,000 (in 2011 dollars) in mid-1984 to about five cents today. In other words, storage costs have dropped by a factor of well over one million since 1984. Not surprisingly, that fundamentally changes the scale of what can be stored.” Villasenor adds. “Memory costs do not become a major obstacle to video surveillance unless the system is truly massive. Even then, the obstacle will only be temporary.”

Controlling costs with compression

The convergence on the H.264 compression algorithm standard supports data storage at user-specified levels of compression, which allows organizations to better manage storage costs by managing the sizes of stored files. H.264 has also enabled technology developers to support and integrate one another’s software and hardware innovations, allowing for more rapid advances across the video management systems industry. Compression is one of the keys to reducing video storage overhead using a VLM system.

In-house storage options: don’t forget tape

Ricco told me about a memorable, if humbling, meeting with a “very smart” senior IT manager at a global organization a few years ago. He had gone to that meeting armed with data about using solid-state disk (SSD) for the organization’s backup and archive needs, but “he schooled me on tape versus spindles,” Ricco said, adding that he learned a lot that day about the advantages of adding digital tape to the mix of storage technologies because of its high density, energy efficiency, and low total cost of ownership.

Many industries with large data storage requirements maintain that tape has never ceased being integral to maintaining costs in enterprise backup and archive operations. For these businesses, tape represents a mature medium, one well suited to low-cost, long-term storage of large quantities of data, and, not incidentally, one that in orders of magnitude may be more reliable than SATA and enterprise disk.

Tape offers tremendous advantages, including cost savings. The Clipper Group did a study showing the total cost of ownership of a disk system is 15 times higher than that for tape for storing the same amount of data in both systems over 12 years. The energy need to store data on disk during this time period is 238 times higher for a disk system than it is for tape. It’s easy to underestimate the energy costs of keeping disk spinning continuously and cooling the disk system.

ROI of surveillance video

Prior to recent innovations, conventional wisdom held that “Surveillance is a cost that can never be turned into a profit.” Only five years later, this statement is no longer true. Today’s data analytics tools—some of which are even built into the surveillance camera itself—can extract information about customer behavior, internal processes, and trends, to generate or increase ROI.

Yet cost remains the top factor in IT decision-making about data storage system purchases. Among the many factors that can affect the cost of storing video surveillance data over time are:

• The video management system and storage system architecture
• The size of the video files generated, which depends on how the camera has been programmed to record video, resolution, recording speed and the compression algorithm used in reducing backup and archive file sizes
• The storage medium
• How often the video data needs to be accessed in the future

Given the many tools on the market today designed to extract information from data, the value of stored video data only stands to increase.

Preserving potential value by archiving video

If your organization is like many, you store many files that you view initially and seldom—if ever—need to view again. To take advantage of data over time, organizations often archive data, rather than discarding it or backing it up. An archive differs from a backup in that it keeps your data accessible. An archive that relies on a mix of disk and tape can best serve to control costs and keep the files accessible as needed: disk arrays ensure rapid access to the most-used files, and tape storage affordably ensures the integrity of the data over the longer term.

While designing a video data storage architecture for your organization’s long-term needs may feel like leaping onto a moving train, choosing the appropriate mix of information management and storage technologies for your organization can allow you to protect your video assets over time with a low total cost of ownership, and without having to discard video because of cost or space limitations. An archive keeps most-used files accessible, protects data, and reduces storage system bandwidth. VLM and data analytics tools are making it possible to extract additional value from stored video data, even when the value of the data is not apparent at the time it is stored.