paella/doc
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

Initial commit

Browse Source
This commit is contained in:
Thomas FORGIONE
2015-04-21 16:47:12 +02:00
parent 4458dddae1
commit 8b08f66bff
141 changed files with 2612 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

93
report/subsections/animation.tex Normal file
View File

@@ -0,0 +1,93 @@
\subsection{Animation}
The objective of this part is to animate the mesh we previously generated
around the skeleton. We have as an input segments of the skeleton, and its
mesh. The idea is to deform the segments of the skeleton, and to
automatically have the impact of this deformation on the vertices of the
mesh.
\subsubsection{Trees}
When we will animate our mesh, we want to keep it as one piece. For
example, when you move your arm, your elbow and your forearm follows
the movement of your arm. This means that the transformation that we
will apply after the shoulders must be kept on the arms and forearms.
To manage to do this, we structured everything in trees. The tree for a
person will be the following for example (to simplify, we will take the
hypothesis that this human has no knees).
\begin{figure}[H]
\centering
\begin{tikzpicture}
% Head
\draw (0,0) circle [radius=1];
\draw (0,0) node{Head};
% Left son of head
\draw (0,-1) -- (-3,-2);
\draw (-3,-3) circle [radius=1];
\draw (-3,-3) node{Left arm};
% Left son of left arm
\draw (-3,-4) -- (-6,-5);
\draw (-6,-6) circle [radius=1];
\draw (-6,-6) node{Left forearm};
% Middle son
\draw ( 0,-1) -- ( 0,-2);
\draw ( 0,-3) circle [radius=1];
\draw ( 0,-3) node{Body};
% Left leg
\draw (0, -4) -- (-2, -5);
\draw ( -2,-6) circle [radius=1];
\draw ( -2,-6) node{Left leg};
% Right leg
\draw (0, -4) -- (2, -5);
\draw ( 2,-6) circle [radius=1];
\draw ( 2,-6) node{Right leg};
% Right son
\draw (0,-1) -- (3,-2);
\draw (3,-3) circle [radius=1];
\draw (3,-3) node{Right arm};
% Left son of left arm
\draw (3,-4) -- (6,-5);
\draw (6,-6) circle [radius=1];
\draw (6,-6) node{Right forearm};
\end{tikzpicture}
\caption{Tree for a human being skeleton}
\end{figure}
\paragraph{}
At each node of this, there will be a rotation (with a center, and
angles), and we will draw the animated mesh by traversing the tree.
At each node, we will apply the transformation of the current node, and
then draw the subtree.
Of course, we also need to know what faces are in each nodes, so we
will have a tree of rotations, and a tree of faces.
\subsubsection{Junctions processing}
All we said before is valid only if the three vertices of a face are
mapped to the same segment. But some faces have vertices that are
mapped to different segments. For these faces, we need to apply a
different transformation for each vertex.
For this purpose, we created a hashtable between vertex numbers and
paths in trees so that we are able to find the transformation for these
vertices.
\subsubsection{Graphical User Interface}
In order to be able to manipulate the animated mesh easily, two graphical interfaces were made :
\begin{enumerate}
\item the first one allows the user to add rotation points to the
skeleton (by default, only the junction points are considered
as rotation points, and there are no knees and elbows for
example).
\item the second one has the OpenGL rendering and a menu controlled
with the keyboard, allowing the user to select rotation points
and to change the value of the angles.
\end{enumerate}

39
report/subsections/branchesMatching.tex Normal file
View File

@@ -0,0 +1,39 @@
\subsection{Branches Extraction and Matching}
The branches extraction part was allocated to both groups. The skeleton we have extracted are composed of points. A skeleton point can be of different types:
\begin{enumerate}
\item a extremity point which only have one neighbor, i.e. this point is involved in only one edge of the skeleton.
\item a regular point which have two neighbors, i.e. this point is involved in only two edges of the skeleton.
\item a junction point which have at least three neighbors, i.e. this point is involved in three edges or more than three edges.
\end{enumerate}
In the cases of extremity points and junction points, we consider that the point is irregular. To extract the branches of the skeleton, we start from each irregular point and we follow the edges starting from this irregular point until we find another irregular point (junction or extremity).
Let us consider two irregular points $P_1$ and $P_5$ such as there is a path $P_1$, $P_2$, $P_3$, $P_4$, $P_5$ linking these two irregular points together. As a result, $P_1$, $P_2$, $P_3$, $P_4$, $P_5$ is a branch of the skeleton. However, the algorithm will also find $P_5$, $P_4$, $P_3$, $P_2$, $P_1$ as a branch of the skeleton. To avoid having each branch two times, we added a new branch to the branches set only if the contrary branch is not already in the set.
Once we extract all the branches on two correspoding skeletons, we would like to match these branches together thanks to the keypoints found on each picture. For this purpose, we would like to assign to each keypoint a branch of the skeleton. We will consider that both skeletons have the same number of branches. If not, the matching between branches would not be perfectly possible.
To determine the correspoding branch of a keypoint, we use the euclidian distance. Each keypoint is assigned to the nearest branch of the skeleton. As a branch is composed of segments, we will define the distance between a point and a branch as the shortest distance between this point and all the segments composing the branch. Thus, the nearest branch of a point is the branch that contains the nearest segment to the point.
The distance between a point and a segment is defined as following:
\begin{enumerate}
\item if the projection of the point is on the line that contains the segment, then the distance between the point and the segment is the distance between the point and its projection.
\item if the projection of the point is not on the line that contains the segment, then the distance between the point and the segment is the shortest distance between the distance between the point and each extremity of the segment.
\end{enumerate}
Once each keypoint has been associated to a branch, we use the matching between keypoints from the two pictures and, for each picture, the matching between keypoints and branches to deduce the matchings between the branches from the two skeletons.
$$\begin{matrix}
\text{Keypoints 1} & \longleftrightarrow & \text{Keypoints 2} \\
\updownarrow & & \updownarrow \\
\text{Branches 1} & & \text{Branches 2}
\end{matrix}$$
To compute the matches between branches, we define a matching a square matrix $M$ which size is the number of branches of each skeleton.
Let $KP_1$ and $KP_2$ be two matching keypoints, respectively from $Picture_1$ and $Picture_2$. Let $Skeleton_1$ and $Skeleton_2$ be the extracted skeletons of $Picture_1$ and $Picture_2$. Let $Branch_1$ be a branch of $Skeleton_1$, $Branch_2$ a branch of $Skeleton_2$. We assume that $KP_1$ is associated to $Branch_1$ and $KP_2$ is associated to $Branch_2$. Since $KP_1$ and $KP_2$ are matching keypoints, a vote is given to a match between $Branch_1$ and $Branch_2$.
Each time a vote is given to a match between $Branch_i$ and $Branch_j$, the element $m_{i,j}$ of $M$ is incremented. Once each pair of matching keypoints has voted, the branches are matching using the matching matrix.
Let $Branch_i$ be a branch of $Skeleton_1$ and $M$ be the matching matrix between branches of $Skeleton_1$ and the branches of $Skeleton_2$. The index $j$ of the matching branch $Branch_j$ with $Branch_i$ is defined such as the element $m_{i,j}$ of $M_{i,\cdot}$ is maximal.
In pratice, the skeletons we try to match have never the same number of branches: some differences may the algorithm unable to give good results. The test report details the differences between skeletons. We thought about solutions, such as merging short branches with others. We could also ask the user to click on matching branches. In this case, there is no need to detect keypoints anymore. Another solution is to ask the user to click on noising branches that often appears on extremities of the skeleton.

21
report/subsections/calibration.tex Normal file
View File

@@ -0,0 +1,21 @@
% Here we go
\subsection{Camera calibration}
The camera calibration is needed by the client during the reconstruction of the 3D skeleton. The calibration consists in two steps :
\begin{enumerate}
\item the \textbf{internal calibration} which is the estimation of the instrinsic parameters of the camera (such as the focal length, or the
principal point)
\item the \textbf{external calibration} which is the estimation of the
extrinsic parameters (the position of the camera from the chessboard)
\end{enumerate}
\subsubsection{Internal calibration}
The internal calibration is based on a program that we completed during a lesson last semester. By detecting chessboards from numerous positions of the camera, we are able to estimate the intrinsic parameters. This program takes as input a video of a chessboard where the camera position changes, and it captures some frames on which the chessboard is detected. With these pictures, we estimate the intrinsic parameters.
\subsubsection{External calibration}
For the external calibration, we wrote a program that uses an
estimation of the homography to compute the extrinsical parameters of
the camera. One should note that what we call the \emph{extrinsical
parameters} are the ones that convert points from the reference of the
chessboard to the reference of the camera.

42
report/subsections/circles.tex Normal file
View File

@@ -0,0 +1,42 @@
\subsection{Regular mesh}
At the beginning of this part we have a set of 3D splines. We sample this splines and calculate the characteristic circle associated to each sampled point. Then we match the points between successive circles and use a triangular mesh.
\subsubsection{Characteristic circles computation}
We take a point sampled on the spline. It gives us information about the sphere located on this point : $\overrightarrow{C}(t)$ the sphere center coordinates and r(t) its radius. We also have their derivatives : $\overrightarrow{C'}(t)$ and r'(t). The idea is to find the intersection between this sphere and the characteristic plane, which would give us the characteristic circle.
If a point P is on the circle, then we can write the formula \ref{eq1} :
\begin{equation}
<\overrightarrow{C'}(t),\overrightarrow{PC}(t)> -r'(t)r(t) = 0
\label{eq1}
\end{equation}
and then we calculate the center of the characteristic circle with the formula \ref{eq2}:
\begin{equation}
\overrightarrow{Cp}(t) = \overrightarrow{C}(t) - \frac{r'(t)r(t)}{\| \overrightarrow{C'}(t)\| ^{2}} \times \overrightarrow{C'}(t)
\label{eq2}
\end{equation}
\begin{figure}[H]
\begin{center}
\includegraphics[scale=0.8]{img/characteristicCircle}
\caption{\label{initialSchedule} Characteristic circle calculation}
\end{center}
\end{figure}
\subsubsection{Meshing}
After obtaining characteristic circles we sample them.
Then we need to find matching points. We decided to make points sampling start in one direction of our mark.
When this is done we project the mark on the new circle and sample again. Then we link each point to the point of the same index on the next circle.
The final mesh can be seen Figure \ref{regularMesh}.
\begin{figure}[H]
\begin{center}
\includegraphics[scale=0.4]{img/regularMesh}
\caption{\label{regularMesh} The regular mesh}
\end{center}
\end{figure}
%TODO

42
report/subsections/extremities.tex Normal file
View File

@@ -0,0 +1,42 @@
\subsection{Extremities}
After computing the mesh around the splines we need to draw the extremities using subdivisions to obtain smoother extremities.
The first step is to calculate the projection of the extremity circle on the correspondent sphere. Then for a subdivision of depth 0 we link this projection with the mesh points along the extrem circle (see Figure \ref{projection}). For one subdivision we take the middle of each segments computed earlier and we project it on the sphere (see Figure \ref{sub}). Finally we made a triangular mesh the way you can see on Figure \ref{mesh}).
The rendering of our extremities on dino.skl for example can be seen on Figures \ref{extremity2} (2 subdivisions) and \ref{extremity10}
(10 subdivisions).
\begin{figure}[H]
\begin{center}
\includegraphics[scale=0.2]{img/projection}
\caption{\label{projection}Projection of the extrem circle center on the sphere}
\end{center}
\end{figure}
\begin{figure}[H]
\begin{center}
\includegraphics[scale=0.2]{img/Subdivision}
\caption{\label{sub}One subdivision}
\end{center}
\end{figure}
\begin{figure}[H]
\begin{center}
\includegraphics[scale=0.2]{img/meshExtremity}
\caption{\label{mesh}The final extremity mesh for one subdivision}
\end{center}
\end{figure}
\begin{figure}[H]
\begin{minipage}[b]{0.45\linewidth}
\centering
\includegraphics[scale=0.5]{img/2}
\caption{\label{extremity2}The final extremity mesh for 2 subdivisions}
\end{minipage}
\hspace{0.5cm}
\begin{minipage}[b]{0.45\linewidth}
\centering
\includegraphics[scale=0.5]{img/10}
\caption{\label{extremity10}The final extremity mesh for 10 subdivisions}
\end{minipage}
\end{figure}

93
report/subsections/junctions.tex Normal file
View File

@@ -0,0 +1,93 @@
\subsection{Junctions}
The last step to complete is to mesh the junctions.
A junction is a point of a skeleton where more than 2 branches join. The process for meshing this portion of the skeleton is complex. In fact it needs to be applicable on multiple cases, for instance 3 or 4 branches (see Figure \ref{junction}, and to take into account the fact that the perfect case will not always be there.
\begin{figure}[H]
\begin{center}
\includegraphics[scale=0.5]{img/Junctions9}
\caption{\label{junction}Three splines junction}
\end{center}
\end{figure}
In the figure \ref{junction1} you can see what we should theoretically obtain with a perfect skeleton. You can see the sphere shared between three splines joining in this junction, and the three characteristic circles associated. Those circles are tangent by pair in one point.
\begin{figure}[H]
\begin{center}
\includegraphics[scale=0.5]{img/JunctionTheory}
\caption{\label{junction1}Theoretical case with perfect characteristic circles}
\end{center}
\end{figure}
But in most of practical cases it is not like this. This is due to number's approximation on computers. The idea is then to look for the closest points of two consecutive circles and to join them. Then we join every point of the circles in one point upside and one downside.
We begin by identifying the last circles of each spline of the junction (see figure \ref{junction3}).
\begin{figure}[H]
\begin{center}
\includegraphics[scale=0.5]{img/Junctions3}
\caption{\label{junction3}3-splines junction}
\end{center}
\end{figure}
To be able to do it the first step is to cut the circles in two parts to find the upside and downside of each of them. This can be done by computing the best fitting plane for the set of circles' center points (figure \ref{junction2}).
The equation to find is easy when there is only 3 circles : first we compute the normal vector of the plane and then the constant element of the equation.
However when there is more than 3 circles to join, so more than three points to approximate, we need to use a PCA (Principal component analysis) decomposition to find the best fitting plane.
The first thing to do is to create X the matrix of the points coordinates.
\[
X =
\begin{pmatrix}
x_{1} & x_{2} & \cdots & x_{n} \\
y_{1} & y_{2} & \cdots & y_{n} \\
z_{1} & z_{2} & \cdots & z_{n}
\end{pmatrix}
\]
Then subtract it with the centroid matrix associated.
\[
X_{c} = X - X_{m}
\]
The normal vector $N$ of the best-fitting plane will be the cross product of the 2 eigenvectors $v_1,v_2$ corresponding to the two biggest singular values of the covariance matrix of $X_{c}$ :
\[
X_{c}X_{c}^T
\]
\[
N = v_1 \wedge v_2
\]
Then to compute the constant value of the equation we use the centroid.
\[
d = -N*X_m
\]
At the end we have the parameters of the plane in N and d.
\begin{figure}[H]
\begin{center}
\includegraphics[scale=0.35]{img/Junctions1}
\includegraphics[scale=0.35]{img/Junctions2}
\caption{\label{junction2}Best fitting plane for the set of circles' center points}
\end{center}
\end{figure}
Then when points are sorted we connect the up-points of each circle with the up-projection of the sphere's center on itself, and idem for the down-points with the down-projection of the sphere's center.
The result of this process is presented in figure \ref{junction4}.
\begin{figure}[H]
\begin{center}
\includegraphics[scale=0.5]{img/Junctions4}
\includegraphics[scale=0.5]{img/Junctions5}
\caption{\label{junction4}Edges added and mesh result}
\end{center}
\end{figure}
To easily see the result of the mesh on a junction, we have made some tests with only the extreme circles that we use. On real skeletons the junction are not easy to watch.
Here \ref{junction5} are two examples of a junction's mesh for 3 and 4-branches junctions.
\begin{figure}[H]
\begin{center}
\includegraphics[scale=0.5]{img/Junctions10}
\includegraphics[scale=0.4]{img/Junctions11}
\caption{\label{junction5}Mesh result on 3 and 4-branches junctions}
\end{center}
\end{figure}

62
report/subsections/pointsDetectionAndMatching.tex Normal file
View File

@@ -0,0 +1,62 @@
\subsection{Points detection and matching}
\label{detection}
\subsubsection{Points detection}
As a first step the detection of keypoints was made by using SURF (Speeded-Up Robust Features) algorithm (provided by
OpenCV) because of its good
speed but this algorithm is computed in nonfree module which is not free to use in commercial
application. That is why we decided to choose the BRISK (Binary Robust Invariant Scalable Keypoints)
algorithm (also provided by OpenCV) is pretty close to the SIFT (Scale-invariant feature transform) algorithm.
Therefore the tests were made using SURF and BRISK algorithm but the final version is with BRISK as SURF is patented.
As the aera of interest in the images is the object, we provide a binary mask (computed during the segmentation) in input of
SURF and BRISK detectors.
SURF is a speed version of SIFT (see Figure \ref{siftSurf}). Indeed instead of approximate Laplacian of Gaussian (LoG) with Difference
of Gaussian for finding scale-space. SURF goes a little further and approximates LoG with Box Filter.
\begin{figure}[H]
\centering
\includegraphics[scale=0.45]{img/LapinSiftSurf}
\caption{\label{siftSurf}Points detection for SIFT and SURF algorithm}
\end{figure}
For BRISK (see Figure \ref{brisk3415}) points of interest are identified across both the image and scale dimensions using a saliency criterion.
In order to increase the speed of computation, keypoints are detected in octave layers of the image pyramid as well as
in layers in-between. The location and the scale of each keypoint are obtained in the continuous domain via quadratic function fitting.
A sampling pattern consisting of points lying on appropriately scaled concentric circles is applied at the neighborhood of each keypoint to retrieve
gray values. Finally, the oriented BRISK sampling pattern is used to obtain pairwise brightness comparison results which are
assembled into the binary BRISK descriptor. The BRISK constructor take in input three parameters which modify the results :
the thresh, the octave and the patternScale.
A detailed explanation of these algorithms can be found with the references \cite{brisk} (BRISK), \cite{surf} (SURF) and
\cite{sift} (SIFT).
\begin{figure}[H]
\centering
\includegraphics[scale=0.45]{img/brisk3415}
\caption{\label{brisk3415}Detected points with BRISK algorithm}
\end{figure}
\subsubsection{Points matching}
We did the points matching, using a brute force matcher provided by OpenCV and then applied filters to get rid of inaccurate points.
For each descriptor in the first set, this matcher finds the closest descriptor in the second set by trying each one.
The filters used are :
\begin{itemize}
\item symmetric filter : the matches found when we take the image\_1 as base need to be found when we take the image\_2 as base also.
\item order constraint : the position of each point is compared to each other in image\_1 and image\_2, if there is too much error these points
are deleted.
\item threshold filter : filter on the distance between the descriptors of the matching points. This filter is not used with BRISK
detection because the results are quite good without it.
\item geometric filter : filter which use epipolar geometry, and the fundamental matrix to filter strange points.
\end{itemize}
\begin{figure}[H]
\centering
\includegraphics[scale=0.45]{img/LapinSymetricGeometric}
\caption{Points matching obtained after symmetric and geometric filtering}
\end{figure}

27
report/subsections/segmentation.tex Normal file
View File

@@ -0,0 +1,27 @@
\subsection{Segmentation}
This part was done by the other group. It consists in partitioning the orignal image \ref{img} into two regions : the background with the chessboard in one part and the plush in the other part. With the binary mask that the segmentation allows us to extract \ref{binarymask} we will be able to create the 2D skeleton with the client's supply \ref{skeleton} and to match the skeleton's branches (see \ref{detection}).
\begin{figure}[H]
\centering
\begin{subfigure}[b]{0.3\textwidth}
\includegraphics[width=\textwidth]{img/Poupee}
\caption{Original image}
\label{img}
\end{subfigure}%
~ %add desired spacing between images, e. g. ~, \quad, \qquad, \hfill etc.
%(or a blank line to force the subfigure onto a new line)
\begin{subfigure}[b]{0.3\textwidth}
\includegraphics[width=\textwidth]{img/Poupeebin}
\caption{Binary mask}
\label{binarymask}
\end{subfigure}
~ %add desired spacing between images, e. g. ~, \quad, \qquad, \hfill etc.
%(or a blank line to force the subfigure onto a new line)
\begin{subfigure}[b]{0.3\textwidth}
\includegraphics[width=\textwidth]{img/Poupeeskl}
\caption{Skeleton extract from (b)}
\label{skeleton}
\end{subfigure}
\caption{Pictures of animals}\label{segmentation}
\end{figure}

13
report/subsections/skeleton.tex Normal file
View File

@@ -0,0 +1,13 @@
\subsection{Skeleton}
Bastien, our client, is working on the extraction of skeleton from pictures. A skeleton of an object is the elemental structure of it. In this project 2D-skeletons extracted, like in figure \ref{skeletonex}, are defined by a set of vertices and edges. The other group compute with it 2D-skeleton in another format : a set of B-Splines, each of them representing a branch of the skeleton.
B-Splines are given by : the degree of the spline, their control points and the nodes vectors.
The 3D-skeletons are given with 3D B-Splines with the same structure except that the control point are in 3D.
You will find in appendix A \ref{AppendixA} some examples of 2D and 3D skeletons.
\begin{figure}[H]
\begin{center}
\includegraphics[scale=0.35]{img/SkeletonEx}
\caption{\label{skeletonex}Best fitting plane for the set of circles' center points}
\end{center}
\end{figure}

39
report/subsections/splines.tex Normal file
View File

@@ -0,0 +1,39 @@
The splines are a necessary element for everything that will follow. This is
why we tried to make it as generic as possible. For this, we used a C++11
feature called \emph{variadic templates}. A spline is composed of control
points (represented as C++ tuples), a nodes vector and degree. The class we
made looks like this
\begin{lstlisting}[language=c++]
template<typename... Types>
class Spline
{
public:
std::tuple<Types...> operator()(float t);
std::tuple<Types...> prime(float t);
private:
std::vector<std::tuple<Types...>> controlPoints;
std::vector<float> nodes;
int degree;
};
\end{lstlisting}
We redefined the \texttt{operator()} to be able to \emph{evaluate} a spline at
a certain time, and the result will be an object which is a barycenter of those
control points. We also made a member function \texttt{prime} to be able to
compute the derivative of a spline.
The \texttt{operator()} computes the barycent of the control points like this
$$C(t) = \sum_{i=0}^m P_i N_i^n(t)$$
where $(P_i)_{i\in [[0,m]]}$ are the $m+1$ control points, $n$ is the degree of
the spline and
$$N_i^0(t) = \left\{\begin{matrix}1&\text{if } t \in [t_i, t_{i+1}] \\ 0 & \text{otherwise} \end{matrix} \right.$$
$$ N_i^k(t) = \frac{t-t_i}{t_{i+k}-t_{i}} N_i^{k-1}(t) + \frac{t_{i+k+1} - t}{t_{i+k+1}-t_{i+1}} N_{i+1}^{k-1}(t) \quad\text{assuming} \quad \frac{0}{0} = 0$$
The \texttt{prime} member function computes the derivative like this
$$C'(t) = \sum_{i=0}^m N_i^{\prime n} (t) P_i $$
with
$$ N_i^{\prime n}(t) = \frac{n}{t_{i+n}-t_{i}}N_i^{n-1}(t) - \frac{n}{t_{i+n+1}-t_{i+1}}N_{i+1}^{n-1}(t)$$
These formulas have been taken from the reference \cite{spline}.