JPEG XS, or should we say XL?

The JPEG XS standard, specifically the ISO/IEC 21122, is a codec developed by the Joint Photographic Experts Group (JPEG), as its name suggests, which promises to deliver interoperability, low latency without visual loss of quality and very low processing requirements for high-quality formats. Saying it this way looks really thrilling, but let’s delve in a little more detail what this means.

By Yeray Alfageme

 

The three pillars of JPEG XS

As we have mentioned, JPEG launched, back in 2018, the JPEG XS specification with three main features in mind:

• Zero visual quality loss: Content compressed in JPEG XS is visually indistinguishable from the original content.
• Low latency: Total latency in a complete chain of encoding, transport and decoding under JPEG XS is minimal, between 1 and 32 latency lines.
• Lightness: JPEG XS is designed to require very little computing power, thus allowing implementation both in software and hardware efficiently.

Each of these three essential pillars has more implications than those of a mere headline. With regards to zero visual quality loss, it allows replacing the original baseband or uncompressed video signal -which for HD environments means between 1.5 and 3 Gbps, unmanageable for many environments- by an equivalent JPEG XS signal which can still be regarded as an original signal. This entails spectacular bandwidth savings. Compared to the older sibling -JPEG 2000- which has a typical bandwidth of around 150 Mbps, JPEG XS offers 3-4 times better throughput, which would then lead to a decrease of the original 1.5 Gbps up to about 38 Mbps. Not bad, uh? And if we translate this to 12-Gbps UHD-HDR signals, it means that a signal to be carried in just 300 Mbps without loss, thus allowing it to be managed by means of 1-Gbps Ethernet links, something just unthinkable with other formats.

The fact that it is a low latency codec also makes it easy to use it anywhere throughout the transmission chain. If it only takes us a few lines to encode and decode this format, nothing will prevent us from using it to compress the output signal of our camera chains on the way to our production center located in a remote production environment, for example.

If the two previous are added to the fact that having enormous computing capacity is not a requirement -as it is sometimes the case with heavier or older codecs- this will allow us to have compression capacity anywhere. For example, a drone broadcasting high-quality 4k50pHDR images wirelessly and by means of JPEG XS without quality loss, with low latency and a controlled bandwidth, enables coverage and range to be reliably extended, making it possible to use it in environments that were not feasible up to now.

But let’s not just scratch the surface, let’s go a little more in detail about what this codec offers, as it is bound to replace JPEG 2000 and, one that many manufacturers have already implemented throughout their entire equipment ranges.

 

 

 

Is it really that good?

Video codecs are generally used for a wide range of applications, mainly aimed at distribution. For production, derivatives of these codecs with specific -although not streamlined- parameters are used.

The JPEG XS codec was specifically designed for use in production environments where high quality is the primary factor, far more important than compression ratio, although the latter must be also kept in mind. In production environments, more capacity is typically available for both streaming and storage. For transmission, private LANs or WANs with high bandwidths are typically used, and for storage, a SAN environment of several hundred Tbs is very typical in any production center. However, uncompressed encoding is difficult, so mezzanine compression really caters to these needs.

JPEG XS is used in instances where 422 or 444 subsampling with up to 12 bits per component and a compression ratio between 2:1 and 10:1 are needed, coupled with a low latency below 32 lines and multi-generation robustness. A typical example is streaming for 4k50p, which requires about 10 Gbit/s. So, reducing the bandwidth by a factor ranging between 3 and 4 enables secure transmission of multiple streams over a 10-Gbit Ethernet line.

And this has great implications on a creative level. As this codec needs very few resources, it can be implemented in FPGA chips for use in devices or environments that require low power consumption or higher compression speed and reliability, such as in software environments by means of a conventional CPU, and thus offer greater flexibility in editing workstations, for instance. A state-of-the-art workstation is not required for viewing in real time a post-production result in 4k50p on JPEG XS; a fairly modern graphics card can support it without problems by using Premiere Pro, for example.

 

But that’s not all

JPEG XS is not only a good, nice, cheap codec, it meets many technical requirements to be considered an industry standard such as JPEG 2000, MPEG2 or even MPEG4. However, it is difficult to compare it to other existing new-generation codecs as it is geared towards both production and broadcast environments. In order to get a better idea, let’s see some features in more detail:

• Truly constant bitrate: Although CBR (Constant Bit Rate) has always been associated with higher bandwidth consumption, a drawback offset by a lower encoding cost and lower latency -all the above being true- 10:1 compression ratios achieved by JPEG XS enable to set very precise, optimized bandwidths. Because JPEG XS allows us choosing the Mb, unlike ProRes, for example, in which we move between bitrate integer values. In this way, all our bandwidth can be used with more than one signal.
• Multiple passes without visual loss: JPEG XS allows up to 10 encoding cycles without much image degradation. We have to be skeptical about this, though. That said, this range is much higher than for other codecs, so we can rest assured and use it in our production environment where, typically, an image is encoded between 4 and 6 times.
• Mathematical Lossless Coding: Whenever visual quality loss is involved, a subjective factor comes into play, although from the EBU to the ITU many studies have been carried out to objectively determine what a visual quality loss really entails. However, as an engineer, bringing it into the mathematical realm always gives me confidence. And there are JPEG XS profiles that actually allow you to reconstruct, pixel by pixel, the same original image from one end to another end of the encoding chain. This is lossless indeed.
• Actual support for new formats: JPEG XS allows up to 16 bits of depth to be allotted to each pixel, which is above the current 12 bits used in HDR signals, for example. This means it can be used in image formats, such as UHD and higher, without the need of tightening codec specifications. In addition, we will be able to continue using it in environments with limited resources or low-latency needs, even with very high resolutions or BT.2020 color ranges and others that are yet to come.

There are other important features, such as bitrate control, but I believe that they do not have as important an impact as the previous ones in our work environments and they do not weigh heavily for JPEG XS to be implemented as a new standard worldwide.

 

 

 

Let’s discuss standards a bit

When it comes to new formats, making reference to standards is a must. Although we very well know that it is the most boring part, they are important, since knowing that a system complies with a certain standard gives us the assurance that the requirements we need are met and, in addition, this allows us to guarantee interoperability. They are boring but necessary.

As we have already mentioned, JPEG XS is the ISO/IEC 21122 standard and was released by the JPEG group in 2018. Back in 2019 is when the first editions of the standards came out, in separate parts, as it is customary.

Part 1, formally designated as ISO/IEC 21122-1, describes the core coding system of JPEG XS. This standard defines the syntax and, similarly to other JPEG and MPEG image codecs, the decompression process for reconstructing a digital image from its data stream. Part 1 provides some guidelines for the reverse process -compressing a digital image into a data stream, more simply called the encoding process- but leaves implementation-specific optimizations and options to implementers.

Part 2 (ISO/IEC 21122-2) builds on Part 1 to split different applications and uses of JPEG XS into subsets of encoding tools with more stringent restrictions. Defining profiles, levels and sublevels allows to reduce the complexity of the implementations in specific applications, while ensuring interoperability. Let’s remind that less complexity generally means less energy consumption, lower production costs, simpler constraints, etc. In addition, Part 2 also specifies a buffer model, consisting of a decoder model and a transmission channel model, in order to guarantee low-latency requirements at a fraction of a frame.

Part 3 (ISO/IEC 21122-3) specifies the transport and container formats for JPEG XS signals. Defines transport of important metadata, such as color spaces, display metadata mastering (MDM), and EXIF, for easy transport, editing, and viewing. In addition, this part defines XS-specific ISOBMFF tables, an Internet media-type registry, and additional syntax to allow XS to be embedded in formats such as MP4, MPEG-2 TS, or the HEIF image file format.

Part 4 (ISO/IEC 21122-4) is a JPEG XS support standard that provides compliance testing and buffer model verification. This standard is crucial for XS implementers and device compliance testing.

And finally, Part 5 (ISO/IEC 21122-5) represents a reference software implementation (written in ISO C11) of the JPEG XS Part 1 decoder, in compliance with Part 2 profiles, levels, and sublevels, as well as an exemplary encoder implementation.

All of these parts were released between May 2019 -Part 1- and October 2020 -Part 5- and all of them are under review. Part 1 was foreseen to be revised by January 2022 and is now in a final stage before release. After this, revisions of each of the subsequent parts will follow throughout the year, so we will have to be ready.

 

What are the profiles available in JPEG XS?

As we have mentioned before, one of the features of JPEG XS is that it allows us to choose the exact bandwidth for our encoded signal. This does not mean that we can encode a 12-bit 8k100p-HDR signal at 100 Mbps, as it would not be properly displayed anyway, so there are certain profiles that the standard offers us to make life a little easier (see Table 1).

 

Conclusion

I think JPEG has done a great job with JPEG XS. JPEG 2000 is already a codec that -the evidence is there- has set and will be a standard in many applications, but with JPEG XS they have managed to offer much more flexibility without overcomplicating its implementation.

It shows in the codec’s design and in the text for its standards, as these two elements are a sign of the fact that increaseingy fewer dedicated signal processing devices are being used in the industry, which makes JPEG XS a lightweight codec that can be implemented in hardware, software or even on mobile devices. In addition, it has been kept as separate as possible from current image formats such as UHD or the likes.

Although it may seem contradictory, this allows the codec to be durable and innovation-proof, because if a new VR format that we know nothing about today arrives, or a new definition or aspect ratio are created, JPEG XS will not care, it will not matter. the codec will apply its algorithm to achieve extremely high compression rates with “lossless” quality.

From 2019 to the present, more and more manufacturers have JPEG XS within their catalog of supported codecs and I believe that it will be a widely used production codec as soon as production formats, such as UHD require it for the most part.

I think we are currently in a time of transition from HD to new formats and all this will eventually stabilize and, almost certainly, JPEG XS will be one of those 2 or 4 chosen codecs as AVC-Intra, MPEG-2 or JPEG 2000 are nowadays.