给定一个场景的多视角的图像，神经辐射场（NeRF）通过图像重建误差优化一个神经场景表征，优化后可以实现逼真的新视角合成效果。NeRF最先是应用在新视点合成方向，由于其超强的隐式表达三维信息的能力后续在三维重建方向迅速发展起来。
在详细解析NeRF代码之前，首要任务是成功运行NeRF代码【win10下参考教程】，后续学习才有意义。本博客讲解Im数据加载模块的代码，不涉及其他功能模块代码。
渲染和逆渲染是计算机图形学中两个重要的概念，分别涉及将三维场景转换成二维图像和从二维图像中还原出三维场景的过程。
通常使用一个中间3D场景表征作为中介来生成高质量的虚拟视角，即首先需要对中间3D场景进行表征，然后再对这个中间3D场景进行渲染，生成照片级的视角。
如何对这个中间3D场景进行表征，分为了“显示3D表征“和”隐式3D表征“
显式是离散的表达，不能精细化，导致重叠等伪影，耗费内存，限制了在高分辨率场景的应用。
隐式是连续的表达，能够适用于大分辨率的场景，而且不需要3D信号进行监督。
图像坐标系与像素坐标系的转换推导：
u
=
x
d
x
+
u
0
u = \frac{x}{{dx}} + {u_0}
u=dxx+u0
v
=
y
d
y
+
v
0
v = \frac{y}{{dy}} + {v_0}
v=dyy+v0
( u , v ) \left( {u,v} \right) (u,v)表示图像中像素的行数和列数， ( u 0 , v 0 ) \left( {u_0,v_0} \right) (u0,v0)表示图像坐标系下的原点在像素坐标系中的坐标， d x {{dx}} dx和 d y {{dy}} dy表示单个像素分别在 x {{x}} x轴和 y {{y}} y轴上的物理尺寸， x d x \frac{x}{{dx}} dxx与 y d y \frac{y}{{dy}} dyy的单位为像素。
矩阵表示形式：
[
u
v
]
=
[
1
d
x
0
0
1
d
y
]
[
x
y
]
+
[
u
0
v
0
]
\left[ {\begin{array}{cc} u\\ v \end{array}} \right] = \left[ {\begin{array}{cc} {\frac{1}{{dx}}}&0\\ 0&{\frac{1}{{dy}}} \end{array}} \right]\left[ {\begin{array}{cc} x\\ y \end{array}} \right] + \left[ {\begin{array}{cc} {{u_0}}\\ {{v_0}} \end{array}} \right]
[uv]=[dx100dy1][xy]+[u0v0]
齐次坐标表示形式：
[
u
v
1
]
=
[
1
d
x
0
u
0
0
1
d
y
v
0
0
0
1
]
[
x
y
1
]
\left[ {\begin{array}{ccc} u\\v\\1 \end{array}} \right] = \left[ {\begin{array}{ccc} {\frac{1}{{dx}}}&0&{{u_0}}\\ 0&{\frac{1}{{dy}}}&{{v_0}}\\ 0&0&1 \end{array}} \right]\left[ {\begin{array}{ccc} x\\ y\\ 1 \end{array}} \right]
uv1
=
dx1000dy10u0v01
xy1
相机坐标系到图像坐标系是透视关系
根据三角形相似原理可得：
f
Z
c
=
x
X
c
=
y
Y
c
\frac{f}{{Z{}_c}} = \frac{x}{{X{}_c}} = \frac{y}{{Y{}_c}}
Zcf=Xcx=Ycy
变换形式：
{
Z
c
⋅
x
=
f
⋅
X
c
Z
c
⋅
y
=
f
⋅
Y
c
\left\{ {\begin{array}{cc} {Z{}_c \cdot x = f \cdot X{}_c}\\ {Z{}_c \cdot y = f \cdot Y{}_c} \end{array}} \right.
{Zc⋅x=f⋅XcZc⋅y=f⋅Yc
齐次坐标表示形式：
Z
c
⋅
[
x
y
1
]
=
[
f
0
0
0
f
0
0
0
1
]
[
X
c
Y
c
Z
c
]
Z{}_c \cdot \left[ {\begin{array}{cc} x\\ y\\ 1 \end{array}} \right] = \left[ {\begin{array}{cc} {\begin{array}{cc} f&0&0 \end{array}}\\ {\begin{array}{cc} 0&f&0 \end{array}}\\ {\begin{array}{cc} 0&0&1 \end{array}} \end{array}} \right]\left[ {\begin{array}{cc} {X{}_c}\\ {Y{}_c}\\ {Z{}_c} \end{array}} \right]
Zc⋅
xy1
=
f000f0001
XcYcZc
带入像素坐标系与图像坐标系之间的转换公式:
Z
c
⋅
[
u
v
1
]
=
[
1
d
x
0
u
0
0
1
d
y
v
0
0
0
1
]
[
f
0
0
0
f
0
0
0
1
]
[
X
c
Y
c
Z
c
]
Z{}_c \cdot \left[ {\begin{array}{cc} u\\ v\\ 1 \end{array}} \right] = \left[ {\begin{array}{cc} {\begin{array}{cc} {\frac{1}{{dx}}}&0&{{u_0}} \end{array}}\\ {\begin{array}{cc} 0&{\frac{1}{{dy}}}&{{v_0}} \end{array}}\\ {\begin{array}{cc} 0&0&1 \end{array}} \end{array}} \right]\left[ {\begin{array}{cc} {\begin{array}{cc} f&0&0 \end{array}}\\ {\begin{array}{cc} 0&f&0 \end{array}}\\ {\begin{array}{cc} 0&0&1 \end{array}} \end{array}} \right]\left[ {\begin{array}{cc} {X{}_c}\\ {Y{}_c}\\ {Z{}_c} \end{array}} \right]
Zc⋅
uv1
=
dx10u00dy1v0001
f000f0001
XcYcZc
整理可得：
Z
c
⋅
[
u
v
1
]
=
[
f
x
0
u
0
0
f
y
v
0
0
0
1
]
[
X
c
Y
c
Z
c
]
Z{}_c \cdot \left[ {\begin{array}{cc} u\\ v\\ 1 \end{array}} \right] = \left[ {\begin{array}{cc} {\begin{array}{cc} {{f_x}}&0&{{u_0}} \end{array}}\\ {\begin{array}{cc} 0&{{f_y}}&{{v_0}} \end{array}}\\ {\begin{array}{cc} 0&0&1 \end{array}} \end{array}} \right]\left[ {\begin{array}{cc} {X{}_c}\\ {Y{}_c}\\ {Z{}_c} \end{array}} \right]
Zc⋅
uv1
=
fx0u00fyv0001
XcYcZc
f x {{f_x}} fx和 f y {{f_y}} fy分别表示相机在 x {{x}} x轴和 y {{y}} y轴方向上的焦距
相机内参(Camera Intrinsics)
K
{{K}}
K为：
K
=
[
f
x
0
u
0
0
f
y
v
0
0
0
1
]
K = \left[ {\begin{array}{cc} {\begin{array}{cc} {{f_x}}&0&{{u_0}} \end{array}}\\ {\begin{array}{cc} 0&{{f_y}}&{{v_0}} \end{array}}\\ {\begin{array}{cc} 0&0&1 \end{array}} \end{array}} \right]
K=
fx0u00fyv0001
当一个点
P
W
(
x
w
,
y
w
,
c
w
)
{P_W}\left( {{x_w},{y_w},{c_w}} \right)
PW(xw,yw,cw)绕
z
{{z}}
z轴旋转
θ
\theta
θ角得到点
P
C
(
x
c
,
y
c
,
c
c
)
{P_C}\left({{x_c},{y_c},{c_c}} \right)
PC(xc,yc,cc)，
z
{{z}}
z坐标保持不变，
x
{{x}}
x和
y
{{y}}
y组成的
x
o
y
{{xoy}}
xoy平面(
o
{{o}}
o是坐标原点)上进行的其实可以看作一个二维旋转。
{
X
c
=
X
w
cos
θ
+
Y
w
sin
θ
Y
c
=
−
X
w
sin
θ
+
Y
w
cos
θ
Z
c
=
Z
w
\left\{ {\begin{array}{cc} {{X_c} = {X_w}\cos \theta + {Y_w}\sin \theta }\\ {{Y_c} = - {X_w}\sin \theta + {Y_w}\cos \theta }\\ {{Z_c} = {Z_w}} \end{array}} \right.
⎩
⎨
⎧Xc=Xwcosθ+YwsinθYc=−Xwsinθ+YwcosθZc=Zw
矩阵形式如下:
[
X
c
Y
c
Z
c
]
=
[
cos
θ
sin
θ
0
−
sin
θ
cos
θ
0
0
0
1
]
[
X
w
Y
w
Z
w
]
=
R
z
[
X
w
Y
w
Z
w
]
\left[ {\begin{array}{cc} {{X_c}}\\ {{Y_c}}\\ {{Z_c}} \end{array}} \right] = \left[ {\begin{array}{cc} {\cos \theta }&{\sin \theta }&0\\ { - \sin \theta }&{\cos \theta }&0\\ 0&0&1 \end{array}} \right]\left[ {\begin{array}{cc} {{X_w}}\\ {{Y_w}}\\ {{Z_w}} \end{array}} \right] = {R_z}\left[ {\begin{array}{cc} {{X_w}}\\ {{Y_w}}\\ {{Z_w}} \end{array}} \right]
XcYcZc
=
cosθ−sinθ0sinθcosθ0001
XwYwZw
=Rz
XwYwZw
当一个点
P
W
(
x
w
,
y
w
,
c
w
)
{P_W}\left( {{x_w},{y_w},{c_w}} \right)
PW(xw,yw,cw)绕
x
{{x}}
x轴旋转
α
\alpha
α角得到点
P
C
(
x
c
,
y
c
,
c
c
)
{P_C}\left({{x_c},{y_c},{c_c}} \right)
PC(xc,yc,cc)，
x
{{x}}
x坐标保持不变，
y
{{y}}
y和
z
{{z}}
z组成的
y
o
z
{{yoz}}
yoz平面(
o
{{o}}
o是坐标原点)上进行的其实可以看作一个二维旋转。
y {{y}} y轴类似于二维坐标系中的 x {{x}} x轴， z {{z}} z轴类似于 y {{y}} y轴
{
X
c
=
X
w
Y
c
=
Y
w
cos
α
+
Z
w
sin
α
Z
c
=
−
Y
w
sin
α
+
Z
w
cos
α
\left\{ {\begin{array}{cc} {{X_c} = {X_w}}\\ {{Y_c} = {Y_w}\cos \alpha + {Z_w}\sin \alpha }\\ {{Z_c} = - {Y_w}\sin \alpha + {Z_w}\cos \alpha } \end{array}} \right.
⎩
⎨
⎧Xc=XwYc=Ywcosα+ZwsinαZc=−Ywsinα+Zwcosα
矩阵形式如下:
[
X
c
Y
c
Z
c
]
=
[
1
0
0
0
cos
α
sin
α
0
−
sin
α
cos
α
]
[
X
w
Y
w
Z
w
]
=
R
x
[
X
w
Y
w
Z
w
]
\left[ {\begin{array}{cc} {{X_c}}\\ {{Y_c}}\\ {{Z_c}} \end{array}} \right] = \left[ {\begin{array}{cc} 1&0&0\\ 0&{\cos \alpha }&{\sin \alpha }\\ 0&{ - \sin \alpha }&{\cos \alpha } \end{array}} \right]\left[ {\begin{array}{cc} {{X_w}}\\ {{Y_w}}\\ {{Z_w}} \end{array}} \right] = {R_x}\left[ {\begin{array}{cc} {{X_w}}\\ {{Y_w}}\\ {{Z_w}} \end{array}} \right]
XcYcZc
=
1000cosα−sinα0sinαcosα
XwYwZw
=Rx
XwYwZw
当一个点
P
W
(
x
w
,
y
w
,
c
w
)
{P_W}\left( {{x_w},{y_w},{c_w}} \right)
PW(xw,yw,cw)绕
y
{{y}}
y轴旋转
β
\beta
β角得到点
P
C
(
x
c
,
y
c
,
c
c
)
{P_C}\left({{x_c},{y_c},{c_c}} \right)
PC(xc,yc,cc)，
y
{{y}}
y坐标保持不变，
z
{{z}}
z和
x
{{x}}
x组成的
z
o
x
{{zox}}
zox平面(
o
{{o}}
o是坐标原点)上进行的其实可以看作一个二维旋转。
z {{z}} z轴类似于二维坐标系中的 x {{x}} x轴， x {{x}} x轴类似于 y {{y}} y轴
{
X
c
=
−
Z
w
sin
β
+
X
w
cos
β
Y
c
=
Y
w
Z
c
=
Z
w
cos
β
+
X
w
sin
β
\left\{ {\begin{array}{cc} {{X_c} = - {Z_w}\sin \beta + {X_w}\cos \beta }\\ {{Y_c} = {Y_w}}\\ {{Z_c} = {Z_w}\cos \beta + {X_w}\sin \beta } \end{array}} \right.
⎩
⎨
⎧Xc=−Zwsinβ+XwcosβYc=YwZc=Zwcosβ+Xwsinβ
矩阵形式如下:
[
X
c
Y
c
Z
c
]
=
[
cos
β
0
−
sin
β
0
1
0
sin
β
0
cos
β
]
[
X
w
Y
w
Z
w
]
=
R
y
[
X
w
Y
w
Z
w
]
\left[ {\begin{array}{cc} {{X_c}}\\ {{Y_c}}\\ {{Z_c}} \end{array}} \right] = \left[ {\begin{array}{cc} {\cos \beta }&0&{ - \sin \beta }\\ 0&1&0\\ {\sin \beta }&0&{\cos \beta } \end{array}} \right]\left[ {\begin{array}{cc} {{X_w}}\\ {{Y_w}}\\ {{Z_w}} \end{array}} \right] = {R_y}\left[ {\begin{array}{cc} {{X_w}}\\ {{Y_w}}\\ {{Z_w}} \end{array}} \right]
XcYcZc
=
cosβ0sinβ010−sinβ0cosβ
XwYwZw
=Ry
XwYwZw
旋转矩阵
R
=
R
x
R
y
R
z
R = {R_x}{R_y}{R_z}
R=RxRyRz：
R
=
[
c
o
s
β
c
o
s
θ
c
o
s
β
sin
θ
−
sin
β
sin
α
sin
β
c
o
s
θ
−
cos
α
sin
θ
sin
α
sin
β
sin
θ
+
cos
α
c
o
s
θ
0
c
o
s
∂
sin
β
c
o
s
θ
+
sin
α
sin
θ
c
o
s
∂
sin
β
sin
θ
−
sin
α
c
o
s
θ
c
o
s
∂
c
o
s
β
]
R = \left[ {\begin{array}{cc} {{\rm{cos}}\beta {\rm{cos}}\theta }&{{\rm{cos}}\beta \sin \theta }&{ - \sin \beta }\\ {\sin \alpha \sin \beta {\rm{cos}}\theta - \cos \alpha \sin \theta }&{\sin \alpha \sin \beta \sin \theta + \cos \alpha {\rm{cos}}\theta }&0\\ {{\rm{cos}}\partial \sin \beta {\rm{cos}}\theta + \sin \alpha \sin \theta }&{{\rm{cos}}\partial \sin \beta \sin \theta - \sin \alpha {\rm{cos}}\theta }&{{\rm{cos}}\partial {\rm{cos}}\beta } \end{array}} \right]
R=
cosβcosθsinαsinβcosθ−cosαsinθcos∂sinβcosθ+sinαsinθcosβsinθsinαsinβsinθ+cosαcosθcos∂sinβsinθ−sinαcosθ−sinβ0cos∂cosβ
将
R
R
R代入则完成世界坐标系到相机坐标系的转化：
[
X
c
Y
c
Z
c
]
=
R
[
X
w
Y
w
Z
w
]
+
t
\left[ {\begin{array}{cc} {{X_c}}\\ {{Y_c}}\\ {{Z_c}} \end{array}} \right] = R\left[ {\begin{array}{cc} {{X_w}}\\ {{Y_w}}\\ {{Z_w}} \end{array}} \right] + t
XcYcZc
=R
XwYwZw
+t
齐次坐标系：
[
X
c
Y
c
Z
c
1
]
=
[
R
t
0
1
×
3
1
]
[
X
w
Y
w
Z
w
1
]
\left[ {\begin{array}{cc} {{X_c}}\\ {{Y_c}}\\ {{Z_c}}\\ 1 \end{array}} \right] = \left[ {\begin{array}{cc} R&t\\ {{0_{1 \times 3}}}&1 \end{array}} \right]\left[ {\begin{array}{cc} {{X_w}}\\ {{Y_w}}\\ {{Z_w}}\\ 1 \end{array}} \right]
XcYcZc1
=[R01×3t1]
XwYwZw1
相机外参
R
t
{{Rt}}
Rt(Camera Extrinsics)：
[
R
t
0
1
×
3
1
]
\left[ {\begin{array}{cc} R&t\\ {{0_{1 \times 3}}}&1 \end{array}} \right]
[R01×3t1]
在相机外参基础上加入相机内参进行计算：
Z
w
⋅
[
u
v
1
]
=
[
f
x
0
u
0
0
f
y
v
0
0
0
1
]
[
R
t
0
1
×
3
1
]
[
X
w
Y
w
Z
w
1
]
Z{}_w \cdot \left[ {\begin{array}{cc} u\\ v\\ 1 \end{array}} \right] = \left[ {\begin{array}{cc} {\begin{array}{cc} {{f_x}}&0&{{u_0}} \end{array}}\\ {\begin{array}{cc} 0&{{f_y}}&{{v_0}} \end{array}}\\ {\begin{array}{cc} 0&0&1 \end{array}} \end{array}} \right]\left[ {\begin{array}{cc} R&t\\ {{0_{1 \times 3}}}&1 \end{array}} \right]\left[ {\begin{array}{cc} {{X_w}}\\ {{Y_w}}\\ {{Z_w}}\\ 1 \end{array}} \right]
Zw⋅
uv1
=
fx0u00fyv0001
[R01×3t1]
XwYwZw1
完成世界坐标系到像素坐标系的转换。
尽可能简单、详细的介绍关于NeRF_Pytorch的所需的预备基础知识(随时补充新的)，后续会根据自己学到的知识结合个人理解讲解NeRF的原理和代码。
