Virtually and Depth Sensor Generated Moire Pictures in Screening and Treatment of Scoliosis
Akos Antal, Attila Katona, Peter Major, Jozsef Molnar, Norbert Szakaly, Peter Tamás, Klara Wenzel
Faculty of Mechanical Engineering, Budapest University of Technology and Economics, Budapest, Hungary
To cite this article:
Akos Antal, Attila Katona, Peter Major, Jozsef Molnar, Norbert Szakaly, Peter Tamas, Klara Wenzel. Virtually and Depth Sensor Generated Moiré Pictures in Screening and Treatment of Scoliosis. Optics. Special Issue: Academic Research for Multidisciplinary. Vol. 4, No. 3-1, 2015, pp. 24-28. doi: 10.11648/j.optics.s.2015040301.16
Abstract: In this paper, different moiré picture generation techniques are presented. This paper presents the basics of moiré effects, a virtual way of moiré generation, depth sensor based moiré picture generation and usage. One of the most common advantages of these methods is rapidity. The computer generated moiré pictures provide an opportunity of screening diagnosis. The depth sensor generated moiré fringes are precise and easy to process. A neural network based reconstruction of vertebral supports both the screening diagnosis and the treatment of scoliosis.
Keywords: Moiré, Depth Sensor, Screening of Scoliosis, 3D Model of Vertebral
Surface reconstruction is one of the most important topics in computer vision due to its wide field of application. Some examples of application are range measuring, industrial inspection of manufactured parts, object recognition and three-dimensional (3D) map rebuilding. There are several different techniques that can be used for optical 3D measurements on object surfaces, such as interferometry, stereovision, coded structured light and moiré methods. These techniques are based on both contact and non-contact procedures and present different sensitivities.
The moiré phenomenon can be observed when superimposing two periodic or quasi-periodic structures. When the two structures have the same or slightly different line spacing and their lines are set approximately parallel, a new coarse pattern appears. This pattern is called a moiré fringe pattern. The spacing and orientation of the moiré fringes depend on the spacing and orientation of the structures being overlapped whereas the visibility of fringes is related to the width of transparent or black lines with respect to the line spacing of the structures . Figure 1. shows a moiré pattern caused by two straight-line gratings with different frequencies tilted with respect to one another.
The moiré technique is a method commonly used in various medical applications. In dental clinics, F.R. Wouters et al. used moiré phenomenon to examine gum swellings caused by dental implants . Richard Smith et al. applied it in the improvement of the dimensional accuracy of tooth implants . In plastic surgery, Sungyeon Ahn et al. developed a special grating to quantify the elasticity of human skin based on moiré strain analysis . Tetsuo Kawara defined the topography of human cornea by forming the grating with a special fluid. He was able to reach a 5 µm resolution, and the method was useful in examining corneal deformities caused by cataract operations.
The most common usage of moiré technique in orthopedics is the quantification of spinal deformities. In this paper, we will present some methods of determining the 3D shape of examined human back and methods of screening and diagnosis. There are a lot of way of moiré picture generation. Computers offer a new way of moiré methods.
2. Virtual Moiré Method
Moiré topographical methods can be distinguished as: the basic grating-shadow, the grating-projection, the grating-TV and the synthetic grating methods. Shadow moiré provides a contour mapping technique that involves positioning a grating close to an object and observing its shadow on the object through the grating. Thus, the basic grating-shadow method offers the best accuracy and the simplest arrangement because the projected grating and the master grating are identical: they have the highest degree of binding. The shadow moiré technique has the disadvantage that the master grating has a similar size to the measured object. Projection moiré offers a contour mapping technique that involves projection of a grating onto an object to produce a shadow grating that is observed through another grating. The projection-type methods offer a lower degree of binding between the phenomenon and the observing grating, larger object size, and more flexibility in adjusting the sensitivity. However, there are very rigid demands for the performance of the projection and the master grating. The master grating is generated by an electronic time varying signal or by a computational process which offers the lowest degree of binding. This means complete independence of both gratings in amplitude and phase. The advantages are additional operations like detection, different types of superposition, and elevation detection. The disadvantage is the limited accuracy of all optoelectronic devices [6,7]. To cope with the above requirements, projection moiré has been chosen as the measuring method to be enhanced.
We have built an equipment for virtual moiré picture generation. The equipment includes a computer controlled projector and a camera. (Figure 2.)
A computer produces the moiré picture from camera images in real time. For 3D reconstruction, and for measuring model information (curves, angles and distances) picture processing is needed. Our picture processor is based on Fourier transformation. The objective is to create a depth map from the original virtual moiré picture. The original moiré picture is noisy by the basic grid, colors and other disturbance (left side on figure 3.) Picture processing is based on evaluation of closed level curves (right side on figure 3.) by the following steps:
1) Convert the picture into 8 bit gray scale due to computing capacity and filtering.
2) Gamma lightness and contrast optimizing based on experiences.
3) Fourier transformation.
4) Empirical frequency filtration by defined object distance and the frequency of virtual grating.
5) The inverse Fourier transformation of the filtered field (the higher grating frequency disappears).
6) Finding edges by Niblack method .
7) Linear filtering for edges and background.
8) Finding edges again.
Level curves are in the right side of figure 3 . It is a hard work to generate a clear moiré picture like on figure 3.
3. Moiré by Depth Sensors
There are four main parts in the Kinect sensor.
1) An infra-red projector to project a pattern of points . This optical copied set of pattern is based on the patent of PrimeSense named Light Coding . The infra-red interval of wavelength ensures that other effect of light does not disturb the situation very much. But there are some problems with infra-red component of sunshine, and with the inadequately reflecting surfaces (left lower corner of figure 4.).
2) The other important element is that the infra-red camera gives a grayscale picture. Due to the different optical axes of IR camera and projector, the distance can be calculated upon the pictures and the position of the IR camera and IR projector.
3) The integrated PrimeSense image processor calculates distances from the IR camera pixels and writes data into a depth puffer. The working distance of the equipment is from 0,6m-to 8m.
4) There is a color camera to show the real picture of the examined object. There is a zoom optic in the cameras to process different distances.
With the 3D model of the object, there is a possibility to select the optimal visualization method. It is possible to put the original texture on the model (first picture in figure 5.). Contour lines are defined by cutting of the model on different depth level. Coloring the surface by black and white between the contour line in turn it can be shown as it was a zebra (second picture in figure 5.).
The moiré picture or a depth model about the back of the patient can be diagnosed both at screening phase and in treatment. Cobb angle  is a widespread factor of vertebral diseases.
4. Determination of Cobb Angle
The vertebral from the back of the patient requires a complex processing software. In order to solve this problem, Kamal  had developed a method to calculate the angle of spinal curvature relatively easily. There are several methods to determine the degree (angle) of spinal curvature, i.e., angle calculation methods.
The Cobb angle is measured on plane radiographs by drawing a line through the superior endplate of the superior end vertebra of a scoliotic curve, and another line through the inferior endplate of the inferiormost vertebra of the same scoliotic curve, and then measuring the angle between these lines. Figure 6. shows the Cobb angles, schematically. Clinically, many Cobb measurements are still performed by hand using pencil and ruler on hardcopy X-ray films .
By using the Kamal’s method , we can calculate the θ angle of the spinal curvature from moiré images which highly correlates to the Cobb angle in case of single curve scoliosis. The following equations (1) can be obtained according to figure 7.
A computer system helps process moiré pictures, searching for the functionality between the moiré characteristics and Cobb angles.
5. Screening by Moiré Technique
There is an intelligent system which is able to store all data of examinations and it is expected to suggest a conclusion or a diagnosis to doctors based on the database stored cases.
The method of decision making is the familiar data mining method named n-nearest neighbor  and the interpolation technique . Data vectors of all examinations until now are stored in a database. The data vector of actual examination is compared with all of the stored exams data vectors. The k nearest (in space of data vectors) examinations are selected. The nearest examination is determined when the difference between the stored data vectors and the actual data vector is minimal. Conclusion can be defined on the subset with k element, where the known conclusions interpolated with a polynomial as the function of the familiar parameters. Actual parameters are substituted by the interpolated function values which give the suggested conclusion.
There is an intelligent frame system that can store and learn measured data and accepted decisions. The objective is to find the cases closest to the actual examination data set (green in the figure 8.). Based on this data the system evaluates the conclusion upon familiar cases (red in figure 8.).
The working diagram of the decision making system is shown in figure 9. Determined points of the back in the moiré images, and the medium severity of the case are presented in the yellow bar on the right side.
Information on moiré picture is enough at screening phase. Diagnosis and treatment needs more knowledge about vertebral.
6. Determination of Spine Geometry
There is a newly developed application defining the spine geometry. Application searches for the functionality between depth map of human back as the variables of the function and spine geometry data as the values of the function. In teaching phase a data base is filled by data of familiar depth maps and data of connected digital radiograms of spine. So data from previous examinations build the base of the learning process. In the concluding phase the depth map of Kinect is the input of the software and the neuron network concludes from the input depth map to the data of the spine geometry. The process is shown in the figure 10.
Figure 11. shows the idea of deduction. The 3D surface of the back is cut by horizontal planes. Coordinates of the cross section curves of the back are the input of the system as function variables. At teaching phase the vertebra points are measured on the radiogram (figure 12.)
At concluding phase the output of the system is the (ξ,ζ) positions of vertebra. Coordinates projected onto back surface result the 3D geometry of vertebral shown in figure 13.
7. Results and Discussion
In this paper different moiré like approaches were presented for computer aided measuring vertebral geometry. The presented virtual moiré method is quick. Moiré information can be stored in a database. On the measured data the presented n-nearest neighbor based decision support the system for adequate screening. Depth sensors (for example Kinect) can be used as precise moiré equipment too. A neural network based application integrated with Kinect data is able to reconstruct even the 3D geometry of vertebral.
In the field of orthopedics, different moiré techniques can be used in the field of screening scoliosis as well as planning the treatment. Creating moiré pictures has no side-effects. Presented screening method needs acceptance in health community. In case of diagnosis and treatment a serious validation process is needed. The Kinect based system could replace some X-ray tests.
National Office for Research and Technology (NKTH) of Hungarian Government for their support since this study has been carried out commonly as part of the project GERINCO2 TECH_08 – A1 / 2 – 2008 – 0121.