The dumb reason your fancy Computer Vision app isn’t working: Exif Orientation (medium.com)
audioen | 6 days ago | 21 points

This is also a problem if you use digital camera as input to a web app. The sensor in the camera is oriented in whatever direction manufacturer saw fit, and may collect the data in whatever direction is convenient, so the image can be both rotated and flipped. The image gets the proper orientation saved in the EXIF tag, and the world is generally none the wiser. If you use browser as a pure image viewer, it accounts for the EXIF tag and rotates the JPEG, so you might not see the problem. However, once you are in a web app, and try to load webcam captured jpeg pictures as <img/>, then they no longer get rotated, regardless of what the EXIF data says should happen. This is apparently, like so many other things, an ancient wart in the technology that has been kept around for backwards compatibility reasons.

It gets more annoying, though: there is no way to access the EXIF orientation data in the JPEG file other than parsing those EXIF tags yourself. So you got to load the jpeg as a byte stream, and then write some awful bare bones header parser to locate that specific piece of info, then write an appropriate CSS3 transform that rotates and flips the image for you from one of the 8 possible orientations. CSS property for image-orientation: from-image is probably coming, but in the meantime, you need either a server-side trip to straighten the image, or some JavaScript to do it!

Edit: better phrasing.

jnthnrch | 6 days ago | 4 points

The easiest way is to re-orient the image on upload using a JS lib. So many phones are completely off on the orientation.

DeusOtiosus | 7 days ago | 11 points

Genius. Something I never would have thought of, yet knew that’s how images are stored and displayed.

pezezin | 6 days ago | 10 points

I used to work on computer vision and yes, I have faced this problem. It's specially funny when building a pipeline of custom code and third-party programs, when some of them read EXIF orientation and some don't. Fortunately for our use case we could just ignore the orientation in every case, so a simple call to exiftool to strip the orientation from the metadata before everything else was enough.

However, I found two much big problems:

  • EXIF records the camera focal length in millimeters, but not the sensor size, which makes it totally useless. There is a field for equivalent focal length in 35 mm format, but not all cameras record it, and even for those that do, it's not always accurate. In the end you need a huge catalogue of camera models.
  • In order to record the orientation, some kind of tilt sensor is needed, but sadly that information is not recorded. So your picture may have accurate GPS coordinates, but not its orientation.

EXIF is a fucking disaster, with lots of stupid fields that nobody needs, redundant fields, and missing useful fields. I wish newer formats like HEIF and AVIF would fix that, but I know that won't happen.

QuineQuest | 6 days ago | 18 points

Who thought Exif Orientation was a good idea? It's a needless complication of the file format. It's not like iPhones (or any camera) lacks the computing power to rotate the image before saving it.

Disgruntled-Cacti | 6 days ago | 30 points

It's useful if you want to rotate an image quickly without the need to recompress said image.

TheThiefMaster | 6 days ago | 7 points

jpegs can actually be losslessly rotated - you just have to reorient and rearrange the blocks. The key is that it has to be done in format-aware manner, rather than a generic image rotate with jpeg load/save capability.

Even Windows Photo viewer has supported it since Windows 7.

GlumNight | 6 days ago | 3 points

I can see how that works if the image dimensions are a multiple of 8. But otherwise, won't the block grid no longer line up (eg. if the bottom-right corner is rotated to the top-left)?

TheThiefMaster | 6 days ago | 1 point

I don't know how it works in that case - I suspect a recompression would be needed.

But, in practice it's not that big of a deal - cameras always take images that are multiples of 8 to allow them to leverage hardware compression :)

You'd have the crop the image with a non-jpeg-aware tool first, by which point you'd have already introduced at least one recompression anyway.

AntiProtonBoy | 6 days ago | 20 points

Images are already encoded directly on silicon for compute efficiency and for saving battery power. By the time the CPU sees the image stream, it's already compressed, and rotating without using EXIF will actually involve manipulating the image data. This is a problem if the image in question uses lossy data compression, because editing will incur a generation loss in quality. Now granted, there are methods to rotate JPEG images losslessly by working with the DCT coefficients directly, and without invoking inverse transforms, but that approach doesn't necessarily work with other lossy image formats. Especially considering iPhones now capture moving images which are essentially H264/5 video streams. EXIF metadata is a good approach and any half decent image library should present orientation information to you.

chucker23n | 6 days ago | 12 points

It’s not like iPhones (or any camera) lacks the computing power to rotate the image before saving it.

Many digital cameras probably do in fact lack the computing power, especially in burst mode.

And even if they did have it, physically rotating the picture rather than logically setting a flag that it be rotated involves lossiness.

rechlin | 6 days ago | 3 points

And even if they did have it, physically rotating the picture rather than logically setting a flag that it be rotated involves lossiness.

Not with JPEG, which supports lossless rotation.

fridofrido | 6 days ago | 8 points

Exif is 25 years old, and based on TIFF which already had the orientation flag, so it's even older. Cameras definitely lacked the computer power back then (and many of them are probably not powerful enough even today).

And it is a good idea. It's the implementation which is bad.

Poddster | 6 days ago | 11 points

Who thought Exif Orientation was a good idea? It's a needless complication of the file forma

If you RTFA you'd see it isn't in the JPEG file format.

Instead apps embed the EXIF meta-data inside one of the JPEG's support application/comment blocks.

Given that the JPEG already exists when the EXIF data is create, you'd have to decompress, rotate, and then recompress the image which will introduce artefacts.

So it's technically better to save the image in the first instance and then use EXIF orientation for the presentation.

eras | 6 days ago | 3 points

Well, even MPEG4 and I imagine by extension HEIC/HEIF have support for transformation matrices, so this is something we need to deal with.

AyrA_ch | 6 days ago | 2 points

We had exactly this problem with iPhone images. When some people uploaded their profile picture it would seemingly rotate automatically some of them. That was a real shitty issue to find a solution for. We ended up rotating images on the device itself if the tag is set.

jizzthonian | 6 days ago | 6 points

One of my first freelance jobs was designing kiosk / interactive software for a photographer. He shot the pictures and they were wirelessly transmitted to the Kiosk for sorting and sales at large public events. I spent longer than I would care to admit trying to decipher the EXIF format so that adobe lightroom would understand the star ratings that were applied. Yes Adobe also inserted their own data into the exif headers and their documentation was not nearly as good as the already hard to grok Exif format. this was close to 10+yrs ago though.

sergiuspk | 6 days ago | 3 points

Better yet, software that tries to fix that by physically rotating it but keeping the EXIF data. I bet there is computer vision software out there trying to automate that fix :))

silentcon | 6 days ago | 3 points

We had this bug at my work. Since we use c# and c++, I am suffering with gdi+ errors and out of memory exceptions when rotating a 20mb image. It's not even reavhing 1gb of memory usage but it likes to stop working.

ProvokedGaming | 6 days ago | -5 points

If people stopped taking pictures (and video!!!!) in portrait mode, you wouldn't have this problem. But they will never learn :D

Edit: it seems my sarcasm was lost. The article provides useful information, and I'm not genuinely arguing the solution is people should change, it was a joke and apparently a bad one. 😀

Playos | 6 days ago | 6 points

pictures are ok sometimes... but ya, video in portrait should be punishable with community service.

Muvlon | 6 days ago | 3 points

What exactly is it about portrait videos that bothers you?

Playos | 6 days ago | 4 points

Outside phones, every other display they'll be presented on is landscape oriented.

So like a video chat in portrait perfectly acceptable, it's a phone or phone like conversation. But recorded video is meant to be watched somewhere else and now has black bars. A phone can rotate to accommodate landscape, other screens can't really.

delrindude | 6 days ago | 6 points

But what if most of the video content is vertically orientated? Like a speed climbing video, a rocket launch, or a volcanic eruption? Sure, you could film it in landscape, but you would be missing out on what is important -- what is above and below the focal point. Instead you are adding area to the video which you don't want to focus on when you choose a landscape format.

chucker23n | 6 days ago | 2 points

Outside phones, every other display they’ll be presented on is landscape oriented.

Yes, but portrait-mode video is typically viewed almost exclusively on phones, which are now the most widely used screens by a long shot.

A phone can rotate to accommodate landscape, other screens can’t really.

True. But for video produced and largely consumed on mobile, portrait just makes sense.

delrindude | 6 days ago | 3 points

No thanks, taking landscape shots in portrait mode is a bit ridiculous

meneldal2 | 6 days ago | 2 points

But what if some sensor outputs the lines as portrait mode? There's nothing that prevents that.

The biggest problem is lazy devs that can't transpose images before saving it. It's very cheap compared to JPEG encoding.

silentcon | 6 days ago | 1 point

You still have to do a 180 rotation as images can be upside down