Jun 6, 2022
Lens Color Filtering Enhancement
The BackgroundLuxonis is customer obsessed. So when customers come to us with problems that are important to them, we listen. And often we can’t let go. So when customers came to us with advice on how to make their robots work outdoors, and questions about why their depth cameras don’t work outdoors, we listened. Even though the cameras being used were not made by us. In fact, these cameras were made by the incumbent in the market, RealSense, and the customer was already phasing them out. As noted here and shown below, the flagship of the RealSense depth camera line, the D455, was incapable of producing useful imagery outdoors.
This is a HUGE problem, as nearly every robotic system makes use of color data. And in agriculture and many industries, this is a TOTAL showstopper. So what did we do? Well, we’re customer obsessed, so we fixed it for the customer. We made a version of our OAK-D with the required OV9782 global shutter color, and we fixed the problem. How? We’ll get there. But below is a quick before/after:
The before ^. Look at that. Classic computer vision image sensor tuning problem. It’s called an improper “Lens Shading Correction” (LSC) that results in colors that aren’t in the scene, but a result of the image sensor, lens, and lens holder interacting and effectively “making up” colors that don’t exist. With proper LSC the image then shows more correct colors:
Before we get into the details of the how, let’s take a quick step back: All of our OAK-D S2 and OAK-D PoE S2 models come standard with the IMX378 as the center camera, but because we want to offer customers the greatest degree of flexibility we give them the option to replace this camera with the OV9782 to increase FOV from 120° to 150°. The one drawback with making this switch–until now–is that discoloration could occur when capturing video outside. Ultimately, this issue relates to an oversaturation problem caused by exposure of the OV9782 to bright light. In these conditions, pink or violet hues can creep into the image.
Intel RealSense D455 - Pink Color IssueWe’re not the only ones dealing with this problem, we’re just the only ones who solved it (more on that in a second). Intel is the closest thing we have to a competitor on the market, and their D455 camera–which uses the same OV9782 that our OAK Series does–struggles with outdoor coloration creep too. And Intel still hasn’t found a solution. A few minutes of Googling reveals the depth of their customers’ concerns, both in the form of direct support forums and industry news. Here are a few examples:
- The IntelRealSense/librealsense GitHub issue
- From DotProduct, Known Issue: Pink/Purple Off-Colors from RealSense D455 & L515
- And from their very own D455 RGB Color Problem community page, someone had this to say in the comments: “This is a production issue and Intel must recall and replace all of the affected units like we had to. Intel; You can't solve this issue with firmware update. I'm returning my D455 and I'll advise my customers NOT to buy them.”
Moving Forward - the Fix & the FutureTime to get down to it. How did we fix a problem that a beast like Intel couldn't (or wouldn’t)? To put it simply, we’re a team that doesn’t quit. We knew that proper color representation is essential for our customers, so worked the problem until discovering that a specialized implementation of Lens Shading Correction (LSC) made a huge improvement. Think of it like putting a pair of sunglasses on the camera lens that allows it to maintain a more consistent picture. We’ve seen the before/after images, but now let’s take a look at a video of a test in action! Even with very short shutter rates in high light environments, we can see how the presence of invasive coloring is significantly reduced. Want to know what’s even better? All of our wide FOV cameras–current and future models–now offer this enhanced functionality, while still allowing the option to choose between the IMX378 (120° DFOV 12MP Rolling Shutter Color) and OV9782 (150° DFOV 1MP Global Shutter Color) for the center camera. Be sure to take a look at our full range of Wide FOV camera options to find the one that’s right for you: OAK-D-W - Wide FOV, USB powered OAK-D-Pro-W - Wide FOV with night-vision and active stereo vision, USB powered OAK-D-W-PoE - Wide FOV, powered over ethernet OAK-D-Pro-W-PoE - Wide FOV with night-vision and active stereo vision, powered over ethernet
Accurate computer vision requires accurate inputs, and accurate computer vision means optimized machine learning. And in a dynamic world full of changing weather patterns and all kinds of outside human interference, having a camera that sees clearly regardless of conditions makes all the difference. And having a company that has your back makes it even better. Want to learn more about what we talked about here…or anything else? Ask us! Join our Discord!
Stuart MooreCommunications Director