Encyclopedia
Please note this is a comparison between Version 2 by Camila Xu and Version 1 by Vladimir Tadic.
RealSense technology comprises a microprocessor for image processing, a module for creating depth images, an IR emitter, a segment for tracking movements, and depth sensors.
- Intel RealSense depth sensors
- IR camera
- RGB camera
- depth image
- image processing
1. RealSense Depth Sensors
RealSense technology comprises a microprocessor for image processing, a module for creating depth images, an IR emitter, a segment for tracking movements, and depth sensors. These depth sensors are built on deep scanning technology, which allows devices to see shapes and objects in the same manner as humans. The complete hardware is also supported by appropriate open-source Software Development Kit (SDK) software called librealsense [7][1]. This software platform provides simple software support for all RealSense cameras. The software platform supports C/C++, ROS (Robot Operating System), Python, MATLAB, etc., systems and programming languages for the development of appropriate and various applications. Intel also provides two applications for the setup and use of the cameras [8][2].
Intel introduced the D400 series RealSense cameras in 2018 with the D415 and D435 models. By presenting these sensors, Intel became an important manufacturer on the market in terms of a balance of quality and price.
These RealSense sensors principally differ in the field of view (FOV) measured in angles and the type of shutter that tunes the exposure.
The D435 camera has a wider FOV (H × V × D—Horizontal × Vertical × Diagonal): 91° × 65° × 100° for RGB camera, which minimizes blind, black spots in the depth map, after which the acquired depth image is pleasing to the eye. The FOV of the corresponding depth sensor is (H × V × D): 85° × 58° × 90°. As a result, this depth sensor is suitable for applications where no great accuracy and precision are required, but where a global visual experience is more significant. Accuracy is the percentage of error with respect to the measured depth, while precision is the capability of the sensors to replicate the same measure in the same conditions [16][3]. These are important parameters of depth sensors, and they can be easily observed by observing the measured depth maps of a scene, or object [7][1]. Usually, visual assessment is used in practice for the characterization of resulted depth images [20,21][4][5]. Therefore, this sensor is often used in automotive applications and in drones. Furthermore, this sensor has a global shutter that ensures a better performance in situations where lighting is unsatisfactory, while capturing fast movements in a scene, and reduces the effect of blurring in images [7][1]. The D435 depth camera also yields better depth measurement results when the targeted objects are a few meters away from the camera itself.
The D415 depth camera has a narrower FOV (H × V × D): 69° × 42° × 77° for RGB camera, and this property results in a higher density of pixels, thus increasing the resolution of the depth map. Here, the FOV of the corresponding depth sensor is (H × V × D): 65° × 40° × 70°. Hence, if accuracy and precision are the main requirements in an application, e.g., avoiding obstacles in robotics application and in object detection, the D415 depth sensor gives much better results. The RealSense D415 sensor has a rolling shutter. This attribute improves the performance of this depth sensor when there are no unexpected fast movements during image capturing, but the image is static [7,11][1][6]. It should be noted that the D415 depth camera yields better depth measurement results when the targeted object is close to the sensor, i.e., 1 m or less. Both the D435 and D415 cameras yield the best quality depth maps at about 60–70 cm from the scene, according to the literature [7,11][1][6]. Figure 1 shows the RealSense depth cameras [7][1].
Figure 1.
(
a
) Intel RealSense D435 and (
Thanks to the RealSense SDK software kit, the user interface of the built-in applications provides the highest control levels, which was unthinkable until recently [11][6].
2. Working Principle
The RealSense depth sensors have three camera lenses: an IR camera, a RGB camera, and an IR laser projector. Hence, these depth sensors are called active devices since they contain a ranging IR laser projector in order to improve the depth measurement. All three lenses in conjunction enable it to assess the depth information by detecting the IR light that is reflected from the object in front of the lenses. The resulting visual information, combined with the SDK software, generates a depth estimate, i.e., produces a depth map. After further post-processing, a depth image yielded in this way can be used, for example, for tracking movements or detecting objects, by creating a user interface that gives the impression of touch, which reacts to movements of the head, the leg, the hand, or any other body part. Naturally, since the RealSense depth sensor also possess an RGB lens and an IR lens, it is therefore possible to capture images in color and in conditions of poor lighting [7,11][1][6].
RealSense sensors use stereovision to calculate depth [7][1]. The realization of stereovision consists of a right-side and a left-side sensor and of an IR projector. The IR projector projects invisible IR rays that improve the accuracy and precision of the depth data in scenes with poor textures. The right-side and left-side sensors capture the scene and send information about the real image to the microprocessor. Based on the received data, the image processor determines via stereometry calculation the depth values for each pixel of the recorded image, thus correlating the values obtained with the right-side camera to the image derived with the left-side camera. The depth data of each pixel processed in this manner result in the depth image. By linking up successive depth images, a depth video stream is generated.
As seen in Figure 2, the value of the depth pixel representing the depth (Z) of an obstacle/object is determined in relation to a parallel plane of the depth sensor doing the capturing and not in relation to the actual range (R) of the obstacle/object from the depth sensor.
A crucial role in the operation of the depth sensor is also played by the RealSense D4 digital signal processor (DSP) for image processing [7][1]. This DSP processor can process 36 million depth values in a second. Thanks to this high performance, these depth sensors are built into a multitude of electronic devices that necessitate high-speed data processing [7,11][1][6].
According to Intel [7][1], the main features of RealSense depth devices are summarized and compared in Table 1.
| Features | D415 | D435 |
|---|
| Depth resolution | 16 bit | 16 bit |
| Max. RGB resolution | 1920 × 1080 pixels | 1920 × 1080 pixels |
| Range | 0.2 m–10 m | 0.2 m–10 m |
| Diagonal field of view | 70° | 90° |
| Shutter | rolling shutter | global shutter |
References
- Tadic, V. Intel RealSense D400 Series Product Family Datasheet; Document Number: 337029-005; New Technologies Group, Intel Corporation: Satan Clara, CA, USA, 2019.
- Grunnet-Jepsen, A.; Tong, D. Depth Post-Processing for Intel® RealSense™ D400 Depth Cameras; Revision 1.0.2; New Technologies Group, Intel Corporation: Satan Clara, CA, USA, 2018.
- ZED Product Portfolio. Stereolabs Product Portfolio and Specifications; Revision 1; Stereolabs: Orsay, France, 2022.
- Tadic, V.; Odry, A.; Burkus, E.; Kecskes, I.; Kiraly, Z.; Vizvari, Z.; Toth, A.; Odry, P. Application of the ZED Depth Sensor for Painting Robot Vision System Development. IEEE Access 2021, 9, 117845–117859.
- Ortiz, L.E.; Cabrera, V.E.; Goncalves, L. Depth Data Error Modeling of the ZED 3D Vision Sensor from Stereolabs. ELCVIA Electron. Lett. Comput. Vis. Image Anal. 2018, 17, 1–15.
- Grunnet-Jepsen, A.; Sweetser, J.N.; Woodfill, J. Best-Known-Methods for Tuning Intel® RealSense™ D400 Depth Cameras for Best Performance; Revision 1.9; New Technologies Group, Intel Corporation: Satan Clara, CA, USA, 2018.
More
