3D reconstruction from a 2D image

Hello,

I have some difficulties to reconstruct a 3D scene from a 2D image. I would like to create a top view of scene removing the perspective, in other words, realize an inverse perspective mapping. Let's assume we know the camera position, orientation and its parameters. Moreover we consider all the captured points lie on the same plane XY. Then, it is easy to prove that a pixel at a (u,v) location in the image will move to the coordinate (X,Y,0) in the 3D space with:

X=-((u*P(3,4)-P(1,4))*(v*P(3,1)-P(2,1)) + (v*P(3,4)-P(2,4))*(P(1,1)-u*P(3,1)))/((u*P(3,2)-P(1,2))*(v*P(3,1)-P(2,1)) + (v*P(3,2)-P(2,2))*(P(1,1)-u*P(3,1)));
Y=(X*(u*P(3,2)-P(1,2)) + (u*P(3,4)-P(1,4)))/(P(1,1)-u*P(3,1));

P is the projection matrix such that: P=[KR KT] with K,R and T respectively the intrinsic, rotation and translation matrices.

Once all the 3D locations of each pixel are computed, I would like to display the XY plane with the color information of the original pixel as if it was a 2D image.

However, a pixel (u,v) can mapped in 3D space to a non integer location meaning that I get a non-regular scatter plot were each (X,Y) point contain a color information. I tried to divide the XY plane into small windows and then compute the average color of all points into each squares but it is very slow.

Please find my code enclosed. Some help would be appreciated!

Thank you in advance, Pm

[mostafa TAIBI]

Hello everyone, I'd like to know how you got this formula...

X=-((u*P(3,4)-P(1,4))*(v*P(3,1)-P(2,1)) + (v*P(3,4)-P(2,4))*(P(1,1)-u*P(3,1)))/((u*P(3,2)-P(1,2))*(v*P(3,1)-P(2,1)) + (v*P(3,2)-P(2,2))*(P(1,1)-u*P(3,1)));

Y=(X*(u*P(3,2)-P(1,2)) + (u*P(3,4)-P(1,4)))/(P(1,1)-u*P(3,1));

Pplease I want the expression of X,Y,Z Euclidean coordinates for a rotating camera as a function of u,v (pixel) and the projection matrix P

[Matt J]

I don't know if that formula is to be trusted, but I offered a simpler expression of the relationships in this comment,

https://www.mathworks.com/matlabcentral/answers/366084-3d-reconstruction-from-a-2d-image#comment_502971

[Damon Pierre-Marie]

Hello Mostafa,

First of all, you can not reconstruct 3D space since a 2D image. To that end you need to make assumptions. In my case, I considered Z=0 meaning all the 3D points are on the XY plane. Considering P as the projection matrix (known) and the previous assumption, you can rewrite the projection equation such that:

u=f(P(i,j),X,Y)

v=g(P(i,j),X,Y)

and with some mathematical gymnastics you can gte:

X=k(P(i,j),u,v)

Y=l(P(i,j),u,v)

I hope it would help you.

[Damon Pierre-Marie]

Here is a reference where I used the method for motorcycle riding aid systems:

http://www.pierremariedamon.com/Publications/IROS_2018_DAMON.pdf

In addition to Z=0, I did not consider the yaw motion for my problem.

[mostafa TAIBI]

For z different from zero the problem will be more difficult I can't ignore z because I'm looking for a 3d reconstruction.

[Matt J]

@mostafa TAIBI,

If the scene you are trying to reconstruct is not confined to a 3D plane, then this post is not relevant to you. It was all predicated on the assumption that the scene was planar.

If the scene you are interested is confined to a plane, then rotate your 3D coordinate system so that the plane becomes the xy-plane and the discussion below applies.

[mostafa TAIBI]

another question if you allow, for the 3d reconstruction is that you take all the pixels of the image after you use the formula to calculate X and Y?

[Matt J]

If you use imwarp as my answer below recommends,

Image2 = imwarp(Image1,projective2d(P(:,[1,2,4]).'))

then no transformation of coordinates need be done by you explicitly. That is all handled for you inside imwarp.

Answer 1

[Matt J]

Once all the 3D locations of each pixel are computed, I would like to display the XY plane with the color information of the original pixel as if it was a 2D image.

I don't think you need to be doing the work of mapping and interpolating the coordinates. I think it should be as simple as

Image2 = imwarp(Image1,projective2d(P(:,[1,2,4]).'))

[Pierre-Marie Damon]

Thank you for your reactivity. I tried but it does not work. I think it due to the fact that P(:,[1,2,4]) does not correspond to the homography matrix between the camera location and the virtual camera top view location.

Thanks, Pm

[Matt J]

You might need inv(P(:,[1,2,4])) instead. Regardless, though, are you saying the challenge is to derive the correct homography? If you've derived it, then just provide that.

[Pierre-Marie Damon]

Sorry, I think I was not clear in my last post. Let's remind that P=[KR KT] with KR a [3x3] matrix and KT a [3x1] vector. If you consider only P(:,[1,2,4]) it means you do not take into account the last column of KR which depend on the 3 rotations: roll, pitch and yaw.

To be honest, I am trying to understand the best way to work with the bird eye view concept.

I think there is two ways to proceed: 1- We know the position and the orientation of the current camera. Then, depending on our ROI we compute the height, the lateral and longitudinal positions of our virtual camera. Finally, we can compute the homography matrix H [3x3]. 2- We can indirectly convert the 2D image into 3D what I am trying to do. I am not sure it is necessary to remap the data even if the result is more visual on an image. It could be a good idea to work with the 3D map as for the previous code but I am afraid of the computation time.

Feel free to correct me if I am wrong.

Thanks, Pm

[Matt J]

Let's remind that P=[KR KT] with KR a [3x3] matrix and KT a [3x1] vector. If you consider only P(:,[1,2,4]) it means you do not take into account the last column of KR which depend on the 3 rotations: roll, pitch and yaw.

In your posted question, you make it sound like you are trying to map the image to the xy-plane in 3D space. In homogeneous coordinates, these points are of the form [X;Y;0;1], and so their projections are

   t*[u;v;1]=P*[X;Y;0;1] = P(:,[1,2,4])*[X;Y;1]

Therefore H=inv(P(:,[1,2,4])) is the homography mapping points in the original image (u,v) to points (X,Y) on the xy-plane.

But all of this assumed that the 3D plane of interest was the xy-plane in your given coordinate system. If it's really supposed to be a different plane, then you can do an analogous analysis for that plane.

[Pierre-Marie Damon]

I am sorry you are totally right, I was lost into my idea. I am wondering how the inwarp() function works. The problem is quite similar with my first question since (X,Y) could be non-integer location. Do you have an idea?

Many thanks, Pm

[Pierre-Marie Damon]

I am playing with the function which works pretty well but I got some mismatches. With the following transformation: projective2d( inv(P(1:3,[1,2,4]).') ), the computed image looks to be a down view with an inversion between X and Y. Do you have an idea about a proper way to fix the problem? Thanks, Pm

[Matt J]

You mean you want to interchange X and Y? Just add a coordinate flip to the homography

   projective2d( [0 1 0; 1 0 0; 0 0 1]*inv(P(1:3,[1,2,4]).') )

You can, in fact, add any other post-transformation that you like

[Pierre-Marie Damon]

I found another solution which is B=rot90(flip(B)). I think your way is proper.

Many thanks for your help Matt, Pm

[Matt J]

You're welcome, but please click "Accept" if we've reached a solution for you.

[Ayush singhal]

@Matt J

Hello, I have a similar doubt regarding this.

I have two 2D images (images are attached). In one image only stripe pattern on the plane and in the second image one can see the shifted stripe pattern on the object. I would like to measure the shift from these 2 images. There is a term relief images, these images are tell the location of pixels of objects. And with these pixel locations,I will reconstruct the object.

Can you guide me here how to start for this?

[Ayush singhal]

The surface plot will depict the shift of the pattern.

[Matt J]

@Ayush singhal

If it's a similar question, you should apply the posted solution. If you can't apply the posted solution, it probably means the question isn't similar enough, and you should probably post it in its own thread.

[Ayush singhal]

@Matt J

okah sure.