Files: Storage and transfer

Red directory over blue unicode data

By Carles Rams, Ebantic’s CEO

One of the trends implicit in the fourth industrial revolution is digitising the production and operating processes of any sector as much as possible. In the audiovisual industry, we started working with digitised data several years ago. We have seen one format after another, and IT systems have become increasingly more robust and functional.  Storage facilities also increased their capacities, and we are now in the midst of a CLOUD revolution. The past was never better, although, in this case, it has been an ordeal to get to where we are now. 

What we all agree on is that files have brought tremendous benefits to our sector. Advantages in the acquisition, management and delivery of content that, would otherwise, not be efficient for the emerging new business models. Think about Netflix for a moment. Netflix started in 1999 distributing content by subscription to the end consumer using a physical medium such as DVD. In 2009, Netflix had 10 million subscribers, and by 2011, the numbers rocketed to 23 million in the United States and 26 million in the rest of the world. By 2015, more than 60 million and in 2016, it reached the 89 million mark worldwide. Can anyone imagine this growth happening by sending DVDs out to members? It is clear that Netflix would continue with its local US market but would not have been able to manage it on a global scale and, above all, with such exponential growth.


The advantages compared to the analogue era

The main advantage lies in the unification of metadata and media in the same tool, allowing an immediate search and viewing of the content, all on the same device (desktop, tablet, mobile), anywhere, anytime. In the analogue era, to search and view content, of which we did not know the title or the identifier a priori, we had to speak to the documentation department first, which, in turn, searched in its database to find the content reference. With this reference, the documentary filmmaker would go to the video library to collect the tape and lend it to you (like in a library), after signing for it, almost with blood, promising that you would return it within the agreed term. Once we got the tape, we had to find a free viewing room, equipped with a very expensive tape recorder, which naturally entailed a limited number of places. Only then could you see the content and see whether it was indeed what you were looking for or if you had to start the process again.

If the content was correct, the next step was to make a copy on a working tape (which we obtained from the tape store after signing the delivery note) in anticipation of having an editing room to edit our programme. This copy also meant losing some quality. Do you remember how every time we copied images from one medium to the next they lost quality? It was a real nightmare for producers and programme directors! Another option was to make a copy on a cheap medium, such as VHS, and pre-edit it using drastically-cheaper equipment. This way, we already had our layout done before we entered the editing room and we were optimising the time we had available.

The arrival of files certainly democratised access to content. Everything explained above could involve days on end. The documentation staff was virtually always up to their ears in work and the waiting times for obtaining material tended to be lengthy. Besides, the viewing rooms-let alone editing rooms- usually had endless waiting lists. Today, one single user can do the entire process, on any device and from the office, from home or while sitting in a city park. We can perform simple or advanced searches, view the results, cut and mount a virtual EDL and then export it to our editing system, which can also run on an office workstation. This has led to an extraordinary reduction in resources and, above all, in the time needed for production. Bear in mind that a Betacam Digital VTR cost about €45,000 (+ maintenance) compared to €1,000 for a desktop PC or laptop. Do the math. What would it have taken to maintain a 100-user installation 15 years ago?

Stock image of data transfer and storage in the world

Another aspect to consider is storage. When the documentation person looked for the tape, he or she went to a room filled with large, heavy tape cabinets. Companies that could afford it had movable shelves that allowed for some degree of space efficiency, but the norm was to have large filing cabinets with several rows of tapes on the same unit. All well ordered because if it weren’t, it would have been impossible to find what you were looking for. The need for space was incredible compared to what an LTO tape robot or a hard drive NAS takes up. In 2017, the LTO Consortium launched its LTO8 version, with a storage capacity of 12 TB on a single tape. This capacity can store more than 850 hours in DV25 or 425 hours XDCAM 50, for example. Comparisons are odious, but 425 hours would take up a whopping 283 90-minute Digital Betacam tapes. Can you imagine the amount of physical space those almost 300 tapes would take up? About ten linear metres of shelving! Crazy, compared to the 21mm that an LTO tape takes up. Another comparison: up to 9 PB can be stored in LTO8 in a robot with two racks of size. Do the math. There are 49 years of video footage coming out in a linear metre!

The most critical aspect to consider when working with files, considering the requirements of the new business models that are appearing in our sector, is the ability to automate processes. We can move and deliver materials automatically without the need for trolleys or transport boxes. We can download content from the file and adapt the format to the recipient’s requirements by merely taking it through a transcoder farm. We can also check the quality of the material and send it in encrypted and accelerated form through the public network. Files also come with the peace of mind that the recipient has received it correctly to generate the invoice.  All this happens without having to get up from our chair, or hire a courier service, or book a satellite transponder. Also, we can get an overview of how our delivery is simply by consulting a dashboard that aggregates the data from the systems involved and displays it to the user in the form of reports. In the continuities, it was usual for the documentation personnel to visit you with the tapes that were to be broadcast the next day. Nowadays, it is the BPM (Business Process Management) system or workflow manager who orders the MAM to download the contents of the file and send them to the broadcast servers according to the play-list generated by the traffic system. It also notifies you of missing data through notifications.


And the drawbacks…

The dreaded drawbacks… As we can imagine, not everything is as good as it seems; working with files also has its complications. The main problem, from my point of view, is management. With the tapes, our copies were under control. Humans move better in physical environments, and tapes were physical objects that could be touched, seen and left on the table. Metadata could be easily added to it with a simple piece of paper attached to the inside of it and labels on the box and on the tape itself. This made the contents difficult to lose (although some were inevitably lost). At most, you had to look inside the drawers… But look on the bright side: you ended up finding that master tape you had been looking for! Managing versions was also quite easy. In a relatively tidy environment, you knew how many tapes you had of each programme. You could even reuse the tape and record over the previous version, and that’s it. Considering the cost of the tapes, you had to reuse them; there was no question about it!

With files, it’s not that easy, as you can’t touch them or see them with the naked eye. You have them in several systems and, as they can be duplicated without losing quality, we usually have several copies (sometimes without control) of the same content in several storage areas. In addition, the only identifying metadata is the file name, and if we use descriptions adapted to humans, we lose the ability to automate. We can, therefore, find directories full of files, whose unique identifier is an unintelligible code, all at the mercy of a human who can change the name, delete it, move it, copy it, without any control. And let’s not talk about security: it is straightforward to make a copy of a file and send it to a third party who may be our competitor or it may mean the beginning of indiscriminate copying of our content on the Internet. To avoid all this, we need management systems that organise and control our files. Just as companies use document systems to coordinate orders, delivery notes, validations and invoicing, we need applications that support the management of the files required for operations. This is where the convergence between IT software and our industry comes in. We have been working with files for many years and the initial problems, especially in processing capacity and storage capacity, have already been solved. This facilitates the use of much cheaper IT tools, as we have access to the entire universe of the open source community.


What about the future?

Read the whole article at:

Archiware announces
Clear-Com releases i