You state "I want to have fixed 3-D points as inputs to the camera. I want to calculate the derivatives of their images wrt camera position and orientation using automatic differentiation."
Step 1 is to identify the 2D image coordinates of the fixed 3D points, on each image. That is an image recognition problem. The solution depends on the nature of the 3D points you are tracking. If you need assistance with this step, provide more information about the points and how they can be recognized in each image. Include some sample images, please.
If you have solved step 1, then you have the image coordinates of each 3D point on each image frame. If you are tracking one 3D point, then let us call its image coordinates u(k), v(k), which are the row and column pixel indices of the 3D point in frame k.
I assume that you also know the position of the camera at each frame: x(k), y(k), z(k). I will assume the camera rotates about a known axis. Then the camera orientation is specified by a single angle, theta. I assume you know theta at each frame: theta(k).
Then you can compute the derivative of image position with respect to camera position:
. This quantity has units of pixels per meter, if x is in meters.You can compute derivatives 
the same way.
the same way.The same approach applies to derivatives with respect to camera orientation. If the rotation rate is constant, rr degrees (or radians) per frame, then the derivative of image position with respect to rotation is
and 
. These quantities have units of pixels per degree, if the rotation rate is expressed in degrees per frame.
