Duplication, recording and storage
By Carlos Medina, Audiovisual Technology Expert and Advisor
Throughout the history of audiovisual production, various solutions have been developed to log and record contents generated in the fields of cinema, television, video and advertising, among others. We all are well aware of the importance of the change from photo-sensitive recording to electromagnetic recording and then to optical media and/or to semiconductors; and also the move from analogue to digital.
At present innovations in the fields of engineering, electronics and IT are setting the direction and pace of changes in audiovisual technology as applied to the areas of production, post-production, distribution, storage and filing. Solutions spring up for each production process and a wide range of requirements from the audiovisual market, from the most professional environments to amateur contexts.
The first thing we found when we approached the subject of this article is the mumbo-jumbo of notions and terms used that have been gathered and transformed throughout the evolution of innovations that have been applied. And secondly, the extensive information and documentation existing on this topic, mostly due to the enormous range of products and solutions that both manufacturers and technology developers are nowadays capable of making available to us.
We will try to contribute our tiny bit by presenting here content that is well-organized, easy to understand, and appropriate for the purpose of dealing with the most usual needs of the audiovisual production sector at present. But the attractive feature of the times in which we are now is that each one of us can find –and we have a definite chance of finding- our own “technological recipe” to meet our professional goals within a given budget.
Recording, duplication and storage have all an element in common: information. Information concerning image, sound, and data. Information we work with and want to keep. It is what we call ‘signal’, regardless of nature (video, audio, synchronisms, metadata; be it analogue or digital) as well as origin thereof (a camera, a microphone, a recorder, a link…). Simply put, this is the so-called WHAT.
The following aspect to have in mind is format. This term is used by the audiovisual industry to make reference to a number of standards, parameters and encoding systems that have been laid down for efficient recording, duplication, storage and/or distribution of information signals. Format may be different (even if signal type is identical) depending on the level and stage within the production and edition processes used in order to obtain the audiovisual work. This is the solution implemented by engineering, IT and pure and applied science ‘wizards’ to materialize –give proper shape- the audiovisual signal for further use. It is the HOW.
There are specialist formats for capture/recording; others were designed for intermediation/post-production; some others are ready for mastering/filing; and, of course, for broadcast. In a digital environment we should discuss in more detail and length the various codecs and containers that progressively shape video and audio formats together with their related information on synchronism and metadata.
Before going into the third aspect as basic knowledge, we must make a distinction -within the video field- between the above-mentioned format notion and the image (in motion) format that we many times use and read, and which, in some instances, may lead to confusion.
When the term -image- format is used in the video field, we are actually making reference the image’s aspect ratio, that is, the ratio between width and height of images comprising what we know as video/film. What we know as 4:3, 16:9 in a TV/video environment; or 1.33:1, 1.77:1, 1.85:1 or 2:35:1 in cinema, among others.
And finally, the third issue: the physical or ‘virtual’ location (always some real location is involved) in which we place the result of the production and encoding processes (images, audios, data…). This what we all know as the medium. It is WHERE.
Signal, format and medium must be identified separately. Within the audiovisual sector an interaction between all three must take place in order to provide each professional environment with efficient solutions; and also based on the needs that are inherent to each production stage, which are quite different at recording, during post-production and when it comes to a context involving transfer and distribution of audiovisual content. We need to have a clear purpose to generating a signal, applying a format and using a medium. This is the WHY.
A good example of this, a classic in the history of audiovisual technology that many technicians either do not know or have never used: VHS.
VHS was a home recording system for composite video signals under JVC’s Video Home System format that used magnetic tape as medium. This is a recording, duplication and storage solution developed back in the 70s and launched in the market in 1976 by JVC themselves and their parent company Matsushita (Panasonic). This novel solution came out due to social and economic reasons: it was time for keen users to be able to view, record, play and store videos in a simple way and at an affordable cost. The important thing was to create the need for us to have video as part of our lives.
Either as professionals or amateurs, if we take a look through the history of the audiovisual sector or even around our own reality, we can see the coexistence of various types of signals, countless formats and a wide range of media, which are the result of multiple reasons such as quality (broadcast, semi-professional, home-user), quantity (bandwidth, recording/broadcasting time, bitrate…), use (cinema, television, documentaries, corporate video, concerts, advertising, among many others) and/or, in some instances, the result of patents and conflicting interests.
Once we have identified WHAT, HOW, WHERE and WHY, we are now in a position to become acquainted with current solutions, which translate into specific technical equipment for recording, duplication and storage from the various technology manufacturers.
We need a general, globalizing vision. This is, a comprehensive approach that includes all kinds of devices for visual capture (cameras), sound capture (microphones), recording and playback equipment or systems (known as recorders o by a typical acronym in the audiovisual field: VTR – Video Tape Recorder – PLAY or REC; or –a bit more up to date- DVR – Digital Video Recorder, or decks), as well as a myriad of ancillary equipment such as encoders, converters, amplifiers, modulators…
Leaving aside camera and microphone types -which is not within the scope of this article- we now focus on the field of what are regarded as recording/playback devices. Thus, we find those that are stationary, that is, made part of a fixed technical installation; and the so-called external or portable units, which can be used in different work situations (such as outdoors) because of their mobility, compact size and weight.
SONY and PANASONIC are ever present manufacturers of both stationary and portable solutions, but worth highlighting is the presence of new brands and manufacturers. Among stationary equipment and concerning video, the solutions provided by BLACKMAGIC through its Hyperdeck product line -in all variants- must be mentioned: Studio 12G, Extreme 8K, Studio Pro, Studio Mini. Also the AJA brand presents KiPro Rack, KiPro Go y KiPro Ultra and Ultra Plus.
As for external/portable devices –sometimes also called recording/monitoring extension units- mention must be made of manufacturer ATOMOS and its Samurai, Shogun (4K or Flame 4K HDR or Inferno 4K HDR 60p), Ninja (Blade, Assassin, Flame or V 4K), Shinobi and Sumo 4K HDR 60p models. Back to AJA, this manufacturer maintains the KiPro Quad and KiPro Mini line; in BLACKMAGIC we find Shuttle Hyperdeck and the 5″ and 7″ Video Assist Monitor; SHINING TECHNOLOGY has a wide range of CitiDISK HD solutions.
As for sound, in stationary equipment worth noting are manufacturers such as DENON and its DN and TASCAM product lines (CD and SS ranges). In portables, very high-level solutions are presented by MARANTZ and ZOOM NORTH AMERICA through the H6, H5, H4n Pro, H2n and H1n models (the latter being one of the manufacturers most firmly established in the market); in mini rack format we have implementations by the TASCAM brand (DR models) and from other manufacturers such as FOSTEX, ROLAND and SHURE.
With regards to duplication, being understood as duplicators equipment enabled for carrying as main task simultaneous copying of audiovisual material, we must once again mention manufacturer BLACKMAGIC and its Duplicator 4K device, which allows for duplication in 25 SD cards and long-lasting recording.
And now time to deal with storage. As it is easy to imagine, we are dealing here with an item involved throughout the whole audiovisual production chain and therefore a broad range of adapted solutions exist. However this is especially relevant when focused on something as specific as filing and preservation.
From a standpoint of what is more fashionable nowadays and leaving the evolution of technology to history books, it is now time to share somewhat more accurate information on the various media and systems (some of them more widely used in recording and others for exchange and storage):
– SD (Secure Digital) cards: This is a device adopting a memory card format developed by SanDisk, Panasonic and Toshiba and introduced in 1999 as an evolutionary improvement on MMC. The standard is maintained by the SD Association, an organization in which several manufactures take part, and was implemented in more than 400 product brands across dozens of categories. Includes four card versions: Standard SD, SDHC (High Capacity), SDXC (Extended Capacity) and SDUC (Ultra Capacity); and comes in three sizes: original standard SD, miniSD and/or microSD.
A constant innovation flow and a somewhat complex set of names have contributed to create some confusion among SD card users. Let us try to shed some light when dealing with the features of this kind of cards. First of all, with regards to storage capacity, values range from 2GB for SD cards and between 2TB and 128TB for SDUC cards.
In second place, we can reclassify SD card types by data speeds -the so-called class- and therefore those being more suitable for transferring video of varying quality, from SD (standard definition) up to 8K and 360º video:
– Regular speed mode: Class 2 (2MB/s), Class 4 (4MB/s) and Class 6 (6MB/s) cards
– High speed mode: Class 2, Class 4, Class 6, Class 10 (10MB/s), Video Class Speed V6 (6MB/s) and Video Class Speed V10 (10MB/s)
– UHS I (Ultra High Speed I) mode: Class 2, Class 4, Class 6, Class 10, U1 (10MB/s), U3 (30MB/s), V6, V10 and V30 (30MB/s)
– UHS II (Ultra High Speed II) mode: Class 4, Class 6, Class 10, U1, U3, V6, V10, V30, V60 (60MB/s) and V90 (90MB/s)
– UHS III (Ultra High Speed III) mode: Class 4, Class 6, Class 10, U1, U3, V6, V10, V30, V60 and V90
In summary, concerning the immediate present and short term, V60 and V90 SD cards are ready for FHD, 4K, 8K and 360º video recording.
– CF (CompactFlash) Cards: This was originally a data storage medium used in portable electronic devices. As a storage device it typically uses flash memory within a standard case and was specified and manufactured for the first time by SanDisk Corporation in 1994. There two main types of CF cards: Type I and Type II, the latter slightly thicker. Three card speeds exist: original CF, CF+/CFast 2.0 and CF 3.0.
As for storage capacity, we can find 32GB, 64GB, 128GB, 256GB and 512GB cards. Two pieces of information associated to these cards need clarification in order to better understand their performance. On the one hand, the UDMA (Ultra Direct Mode) acronym: this is a standard interface (in ATA mode) allowing higher transfer rates and associated to card version. From UDMA 0 (16MB/s) to UDMA 7 (167 MB/s) –corresponding to version 6.0-, it is now the state-of-the-art in maximum transfer speed for CF cards.
And, on the other hand, the second detail makes reference to the VPG (Video Performance Guarantee) specification, which ensures a minimum guaranteed writing speed for professional video capture (MB/s) as included from version CF 5.0 onwards. Thus, we have VPG 20 (equivalent to 20MB/s), VPG 65 (65MB/s), and so on.
Please note a CFast 2.0 card achieves reading speeds up to 525MB/s and writing speeds of 450MB/s, being more suitable for high-performance recordings such as those required with 4K and 8K. And for HD and FHD, a regular CF card reaching reading speed up to 160MB/s and writing speed up to 155MB/s will suffice.
– P2 (a trademark of “Plug-in Professional”) Cards: It is a professional digital solid-state recording medium. This storage format was introduced by Panasonic in 2004 and is especially tailored to electronic news gathering (ENG) in applications.
This kind of card is basically a RAID from Secure Digital – (SD) memory cards – packaged in a type-II PC card (formerly PCMCIA) die-cast, in such a way that the data transfer rate increases as storage capacity does. There are three P2 card classes: R series, F series (transfer rate up to 1.2GB/s). Both provide storage capacities of 16GB, 32GB and 64GB. And the latest version: The express series (thickness 50% higher than conventional P2 cards; writing speed up to 1.2GB/s; reading speed up to 2.4GB/s), which is used for Varicam recording and features 256GB capacity.
There is a microP2 series, with a size nearly identical to that of an ordinary SD card.
– SxS cards: This is a type of memory card based on CompactFlash technology and especially created as video medium by Sony in cooperation with SanDisk. It was introduced in 2007.
There are two versions of the Express card: Express Card\54, measuring 54 mm in width and a size similar to a PC Card; and the 34 mm Express Card\34.
Three models are available: SXS-1, the basic card; SxS PRO+, which works with RAID PSZ-RA6T and PSZ-RA4T units in order to provide users with a high-speed data transfer solution. 3.5GBps reading speed and 3.2GBps maximum writing speed. Storage capacities are 64GB, 128GB and 256GB.
And last, SXS PRO X cards, which make use of the advanced PCI Express 3.0 interface, featuring ultra quick transfer rates up to 10GBps and storage capacities of 64GB and 128GB.
– XQD cards: In November 2010, SanDisk, Sony and Nikon presented this medium that does not support CompactFlash or CFast under the PCI Express interface. In 2012, Sony marketed its first XQD card.
After the N, H and S series, SONY presented the G series with capacities of 32GB, 64GB, 128GB and 256GB, which allowed reading/writing speeds of 440/400MB/s. This meant the highest reliability and increase performance for 4K users.
– CFexpress Cards: Their introduction was announced in 2016. In 2019, specifications were laid down for this kind of memory cards, which come in three sizes: Type A, type B and type C.
CFexpress 2.0 type B is the new standard and uses the same form factor and interface but works under PCI Express Gen3 x2 for higher speeds, lower latencies and less energy consumption. Sony defines this as the natural evolution of the XQD and CFast standards, reaching reading transfer rates up to 1,700MB/s and writing transfer rates up to 1,480MB/s, which makes it a suitable solution for 4K. Three different storage capacities are available: 128GB, 256GB and 512GB.
– Removable hard disks: Metal disk units featuring rigid magnetic surfaces and mechanical reading/writing elements placed within a case and equipped with an appropriate connector that determines access speed, size and storage capacity. They can be internal or external (portable).
– SSD hard disks: SSD is the acronym for Solid State Disk. These disks, as their own name suggests, use memory comprising semiconductors, also known as solid-state memory. They can be internal or external (portable).
This kind of disks is the most usual one at present as they have a number of advantages that perfectly cater to the needs and situations of the audiovisual sector, especially those found at a recording or shooting: faster boot-up, greater writing speed and higher reading speed (up to ten times more than traditional hard disks), low reading and writing latency (hundreds of times faster than mechanical hard disks), lower power consumption and heat generation, no noise, improved security, resistant (endure drops, hits and vibrations without breaking as they have no mechanical elements), lower weight and size.
Manufacturers of SSD controlling this market are SAMSUNG, SEAGATE and WESTERN DIGITAL; or they already are proprietary solutions of a specific recorder or camera manufacturer such as RED ONE (RED DIGITAL CINMA MINI-MAG 480GB), ARRI (Codex Compact Drive 1TB) or AJA (KiStor 1TB); and SSD have been developed even for ATOMOS recorders: SONY AtomX (2TB), G-TECHNOLOGY Master Caddy 4K (1TB), ANGELBIRD (1TB).
– RAID systems: RAID is the acronym for “Redundant Array of Independent Disks”. This is a technology combining several rigid hard disks (HD) into a single logical unit in which data are stored in all disks (redundancy). It was born in the University of California at Berkeley (USA) in the late 80s.
There are 6 RAID levels that have nothing to do with JBOD (Just a Bunch of Disks) or SPAN (N-RAID) storage:
– RAID 0 – This level is also known as Striping (even splitting of data)
– RAID 1 – Also known as Mirroring
– RAID 2 – Fault detection in rigid disks
– RAID 3 – In this level, data are spread across the disks of the array save for one, which is used for storing parity information
– RAID 4 – An improvement on RAID 3 that offers higher speed and supports larger files
– RAID 5 and 6 – Parity in the array
Some relevant manufacturers in this field are STARDOM (SohoRaid 4 Hdd model), TERRAMASTER (D5THB3 5 Hdd model), LACIE (6Big or 12Big models), OWC (ThunderBay 4 RAID Dual model), among others.
– Shared Media Servers: These are solutions that make good use of the basic technologies implemented in the IT industry in general and are streamlined for stricter requirements allowing for real time, high availability and high capacity for TV broadcast and 4K production environments.
In order to know these servers in some more detail, storage methods must be mentioned:
– DAS or Direct-Attached Storage is the traditional storage method and entails connecting the storage device directly to the computer or server. Its drawback is that this storage is not usually shared with other devices.
– NAS or Network-Attached Storage is storage accessed through a network in which a computer acts as server and shares the drive with other equipment as required. The server acts as an intermediary and reads from and writes on the shared volume.
– Clustered NAS is an improved version of NAS and is based on the availability of several servers that share the same drives, thus enabling a better allocation of work loads and with the additional advantage of having more Ethernet communication interfaces.
– SAN or Storage Area Network is a storage system in which client equipment has the ability of reading from and writing directly on the shared volume as if were local storage. Through the iSCSI protocol, which is more economical although offering lower output; or, on the other hand, through Fibre Channel, which provides much lower latency and a better sustained bandwidth average.
– Cloud is a kind of storage accessed through an Internet or IP connection with remote servers located outside the local network.
Manufacturers offering the solutions providing better implementation are: AVID, PROMISE TECHNOLOGY, G-TECHNOLOGY, SM DATA, DELL EMC ISILON, QUANTUM, QNAP SYSTEMS, EDITSHARE.
– CPM tapes or cartridges: Based on the principles applied to magnetic tapes offering mass storage capacity, these devices have become an option for the challenges we face when filing 4K and shortly 8K materials, with very high transfer rates and at a low cost while providing excellent preservation times. Two are the most popular options:
– LTO (Linear Tape-Open): was originally developed in the late 90s by HEWLETT-PACKARD, IBM, and SEAGATE, companies that set up the LTO Consortium. Other manufacturers such as QUANTUM and SONY are also offering this solution.
Since 2000 several generations –LTO 0 to LTO 12- have appeared. LTO 11 and LTO 12 are pending forthcoming appearance with native capacities of 96TB and 192TB, respectively. Data for the last three ones used are LTO-8 (12TB native), LTO-9 (24TB native) and LTO-10 (48TB native).
– ENTERPRISE: These are high-end tapes that cover storage needs exceeding 5 PB, being two different proposals the most popular ones: Oracle’s and IBM’s.
IBM’s 3592 series gradually meets present generations: Gen1 (300 GB native) to Gen6 (20TB native, year 2018) and future ones Gen 8 (30TB native) y Gen 9 (40TB native).
As for Oracle, its proposal is marketed as StorageTek. We could establish year 1995 as a first date of reference and it had a native capacity of 10GB, with series T10000 reaching up to 8,500 GB native.
Tape-based technology excels on material preservation times, which directly depend on tape coating. Barium Ferrite technology enables preservation of data for much longer than the MP (Metal Particles) technology that was traditionally used for manufacturing in older generations of tapes such as LTO 4 and LTO 5. The first LTO tape manufactured with Barium Ferrite was LTO 6 and is 100% used in Enterprise tapes.
The result is that the Barium Ferrite tape is the only tape-based technology capable of preserving data for over 30 years, that is, seven times longer than a hard disk and 10 years more than through any tape-coating technology.
Other solutions such as SmartMedia, Memory Stick or XD-Picture Cards that exist at present have not become as popular as the above-mentioned solutions. Some of them because of their failure to increase performance and others due to a lack of implementation in the audiovisual market by developers and manufacturers using them. As for cassette tapes, a faithful partner in audiovisual productions for many of the most seasoned technicians, they are hardly ever used at present for recording and exchanging material.
Performance, quality, efficiency, capacity, speed, security and price are and will be the parameters considered by manufacturers and developers of resources for recording, duplication and storage at present and in the future.
What has been written in this article, although now the present, will become sooner than we think part of the history of audiovisual technology. That is why it is vital to identify what signal, format, and media are when it comes to implementing solutions catering to the needs and demands of the market as adapted to the times in which we are living.