未验证 提交 a8ae6b7a 编写于 作者: W wangna11BD 提交者: GitHub

add doc for PPMSVSRPredictor and BasicVSRPredictor (#522)

上级 7c33750b
...@@ -48,6 +48,7 @@ GAN--生成对抗网络,被“卷积网络之父”**Yann LeCun(杨立昆) ...@@ -48,6 +48,7 @@ GAN--生成对抗网络,被“卷积网络之父”**Yann LeCun(杨立昆)
- 💛 **最新发布** 💛 - 💛 **最新发布** 💛
👀 **视频超分SOTA算法[PP-MSVSR](./docs/zh_CN/tutorials/video_super_resolution.md):一行命令从"马赛克"到"高清影像"** 👀 👀 **视频超分SOTA算法[PP-MSVSR](./docs/zh_CN/tutorials/video_super_resolution.md):一行命令从"马赛克"到"高清影像"** 👀
- **[完整在线教程](https://aistudio.baidu.com/aistudio/projectdetail/3205183)**
<div align='center'> <div align='center'>
<img src='https://user-images.githubusercontent.com/48054808/144848981-00c6ad21-0702-4381-9544-becb227ed9f0.gif' width='600'/> <img src='https://user-images.githubusercontent.com/48054808/144848981-00c6ad21-0702-4381-9544-becb227ed9f0.gif' width='600'/>
</div> </div>
......
...@@ -7,7 +7,12 @@ PaddleGAN(ppgan.apps)provides prediction APIs covering multiple applications ...@@ -7,7 +7,12 @@ PaddleGAN(ppgan.apps)provides prediction APIs covering multiple applications
* [DeepRemaster](#ppgan.apps.DeepRemasterPredictor) * [DeepRemaster](#ppgan.apps.DeepRemasterPredictor)
* Super Resolution: * Super Resolution:
* [RealSR](#ppgan.apps.RealSRPredictor) * [RealSR](#ppgan.apps.RealSRPredictor)
* [PPMSVSR](#ppgan.apps.PPMSVSRPredictor)
* [PPMSVSRLarge](#ppgan.apps.PPMSVSRLargePredictor)
* [EDVR](#ppgan.apps.EDVRPredictor) * [EDVR](#ppgan.apps.EDVRPredictor)
* [BasicVSR](#ppgan.apps.BasicVSRPredictor)
* [IconVSR](#ppgan.apps.IconVSRPredictor)
* [BasiVSRPlusPlus](#ppgan.apps.BasiVSRPlusPlusPredictor)
* Video Frame Interpolation: * Video Frame Interpolation:
* [DAIN](#ppgan.apps.DAINPredictor) * [DAIN](#ppgan.apps.DAINPredictor)
* Motion Driving: * Motion Driving:
...@@ -205,21 +210,83 @@ run_video(video) ...@@ -205,21 +210,83 @@ run_video(video)
> > - tuple(frame_path(str), out_path(str)): frame_path is the save path of each frame of the video after super resolution, and out_path is the save path of the video after super resolution. > > - tuple(frame_path(str), out_path(str)): frame_path is the save path of each frame of the video after super resolution, and out_path is the save path of the video after super resolution.
## ppgan.apps.EDVRPredictor
## ppgan.apps.PPMSVSRPredictor
```python ```python
ppgan.apps.EDVRPredictor(output='output', weight_path=None) ppgan.apps.PPMSVSRPredictor(output='output', weight_path=None, num_frames=10)
``` ```
> Build the instance of RealSR. EDVR is a model designed for video super resolution. For more details, see the paper, EDVR: Video Restoration with Enhanced Deformable Convolutional Networks (https://arxiv.org/abs/1905.02716). The interface imposes 2x super resolution on the input video. The recommended video format is mp4. > Build the instance of PPMSVSR. PPMSVSR is a multi-stage VSR deep architecture. For more details, see the paper, PP-MSVSR: Multi-Stage Video Super-Resolution (https://arxiv.org/pdf/2112.02828.pdf). The interface imposes 4x super resolution on the input video. The recommended video format is mp4.
> >
> *Note: The interface is only available in static graph, add the following codes to enable static graph before using it: > **Parameter**
> >
> ``` > ```
> import paddle > from ppgan.apps import PPMSVSRPredictor
> paddle.enable_static() #enable static graph > sr = PPMSVSRPredictor()
> paddle.disable_static() #disable static graph > # test a video file
> sr.run("docs/imgs/test.mp4")
> ``` > ```
> **参数**
>
> > - output (str): path of the output image, default: output. Note that the path should be set as output/EDVR.
> > - weight_path (str): path of the model, default: None,pre-trained integral model will then be automatically downloaded.
> > - num_frames (int): the number of input frames of the PPMSVSR model, the default value: 10. Note that the larger the num_frames, the better the effect of the video after super resolution.
```python
run(video_path)
```
> The execution interface after building the instance.
> **Parameter**
>
> > - video_path (str): path of the video files.
>
> **Return Value**
>
> > - tuple(str, str): the former is the save path of each frame of the video after super resolution, the latter is the save path of the video after super resolution.
## ppgan.apps.PPMSVSRLargePredictor
```python
ppgan.apps.PPMSVSRLargePredictor(output='output', weight_path=None, num_frames=10)
```
> Build the instance of PPMSVSRLarge. PPMSVSRLarge is a Large PPMSVSR model. For more details, see the paper, PP-MSVSR: Multi-Stage Video Super-Resolution (https://arxiv.org/pdf/2112.02828.pdf). The interface imposes 4x super resolution on the input video. The recommended video format is mp4.
>
> **Parameter**
>
> ```
> from ppgan.apps import PPMSVSRLargePredictor
> sr = PPMSVSRLargePredictor()
> # test a video file
> sr.run("docs/imgs/test.mp4")
> ```
> **参数**
>
> > - output (str): path of the output image, default: output. Note that the path should be set as output/EDVR.
> > - weight_path (str): path of the model, default: None,pre-trained integral model will then be automatically downloaded.
> > - num_frames (int): the number of input frames of the PPMSVSR model, the default value: 10. Note that the larger the num_frames, the better the effect of the video after super resolution.
```python
run(video_path)
```
> The execution interface after building the instance.
> **Parameter**
>
> > - video_path (str): path of the video files.
>
> **Return Value**
>
> > - tuple(str, str): the former is the save path of each frame of the video after super resolution, the latter is the save path of the video after super resolution.
## ppgan.apps.EDVRPredictor
```python
ppgan.apps.EDVRPredictor(output='output', weight_path=None)
```
> Build the instance of EDVR. EDVR is a model designed for video super resolution. For more details, see the paper, EDVR: Video Restoration with Enhanced Deformable Convolutional Networks (https://arxiv.org/abs/1905.02716). The interface imposes 4x super resolution on the input video. The recommended video format is mp4.
> >
> **Parameter** > **Parameter**
> >
...@@ -247,6 +314,111 @@ run(video_path) ...@@ -247,6 +314,111 @@ run(video_path)
> > - tuple(str, str): the former is the save path of each frame of the video after super resolution, the latter is the save path of the video after super resolution. > > - tuple(str, str): the former is the save path of each frame of the video after super resolution, the latter is the save path of the video after super resolution.
## ppgan.apps.BasicVSRPredictor
```python
ppgan.apps.BasicVSRPredictor(output='output', weight_path=None, num_frames=10)
```
> Build the instance of BasicVSR. BasicVSR is a model designed for video super resolution. For more details, see the paper, BasicVSR: The Search for Essential Components in Video Super-Resolution and Beyond (https://arxiv.org/pdf/2012.02181.pdf). The interface imposes 4x super resolution on the input video. The recommended video format is mp4.
>
> **Parameter**
>
> ```
> from ppgan.apps import BasicVSRPredictor
> sr = BasicVSRPredictor()
> # test a video file
> sr.run("docs/imgs/test.mp4")
> ```
> **参数**
>
> > - output (str): path of the output image, default: output. Note that the path should be set as output/EDVR.
> > - weight_path (str): path of the model, default: None,pre-trained integral model will then be automatically downloaded.
> > - num_frames (int): the number of input frames of the PPMSVSR model, the default value: 10. Note that the larger the num_frames, the better the effect of the video after super resolution.
```python
run(video_path)
```
> The execution interface after building the instance.
> **Parameter**
>
> > - video_path (str): path of the video files.
>
> **Return Value**
>
> > - tuple(str, str): the former is the save path of each frame of the video after super resolution, the latter is the save path of the video after super resolution.
## ppgan.apps.IconVSRPredictor
```python
ppgan.apps.IconVSRPredictor(output='output', weight_path=None, num_frames=10)
```
> Build the instance of IconVSR. IconVSR is a VSR model expanded by BasicVSR. For more details, see the paper, BasicVSR: The Search for Essential Components in Video Super-Resolution and Beyond (https://arxiv.org/pdf/2012.02181.pdf). The interface imposes 4x super resolution on the input video. The recommended video format is mp4.
>
> **Parameter**
>
> ```
> from ppgan.apps import IconVSRPredictor
> sr = IconVSRPredictor()
> # test a video file
> sr.run("docs/imgs/test.mp4")
> ```
> **参数**
>
> > - output (str): path of the output image, default: output. Note that the path should be set as output/EDVR.
> > - weight_path (str): path of the model, default: None,pre-trained integral model will then be automatically downloaded.
> > - num_frames (int): the number of input frames of the PPMSVSR model, the default value: 10. Note that the larger the num_frames, the better the effect of the video after super resolution.
```python
run(video_path)
```
> The execution interface after building the instance.
> **Parameter**
>
> > - video_path (str): path of the video files.
>
> **Return Value**
>
> > - tuple(str, str): the former is the save path of each frame of the video after super resolution, the latter is the save path of the video after super resolution.
## ppgan.apps.BasiVSRPlusPlusPredictor
```python
ppgan.apps.BasiVSRPlusPlusPredictor(output='output', weight_path=None, num_frames=10)
```
> Build the instance of BasiVSRPlusPlus. BasiVSRPlusPlus is a model designed for video super resolution. For more details, see the paper, BasicVSR++: Improving Video Super-Resolution with Enhanced Propagation and Alignment (https://arxiv.org/pdf/2104.13371v1.pdf). The interface imposes 4x super resolution on the input video. The recommended video format is mp4.
>
> **Parameter**
>
> ```
> from ppgan.apps import BasiVSRPlusPlusPredictor
> sr = BasiVSRPlusPlusPredictor()
> # test a video file
> sr.run("docs/imgs/test.mp4")
> ```
> **参数**
>
> > - output (str): path of the output image, default: output. Note that the path should be set as output/EDVR.
> > - weight_path (str): path of the model, default: None,pre-trained integral model will then be automatically downloaded.
> > - num_frames (int): the number of input frames of the PPMSVSR model, the default value: 10. Note that the larger the num_frames, the better the effect of the video after super resolution.
```python
run(video_path)
```
> The execution interface after building the instance.
> **Parameter**
>
> > - video_path (str): path of the video files.
>
> **Return Value**
>
> > - tuple(str, str): the former is the save path of each frame of the video after super resolution, the latter is the save path of the video after super resolution.
## ppgan.apps.DAINPredictor ## ppgan.apps.DAINPredictor
```python ```python
......
...@@ -7,9 +7,9 @@ ...@@ -7,9 +7,9 @@
### 🔥 PP-MSVSR 🔥 ### 🔥 PP-MSVSR 🔥
[PP-MSVSR](https://arxiv.org/pdf/2112.02828.pdf) is a multi-stage VSR deep architecture, with local fusion module, auxiliary loss and refined align module to refine the enhanced result progressively. Specifically, in order to strengthen the fusion of features across frames in feature propagation, a local fusion module is designed in stage-1 to perform local feature fusion before feature propagation. Moreover, an auxiliary loss in stage-2 is introduced to make the features obtained by the propagation module reserve more correlated information connected to the HR space, and introduced a refined align module in stage-3 to make full use of the feature information of the previous stage. Extensive experiments substantiate that PP-MSVSR achieves a promising performance of Vid4 datasets, which PSNR metric can achieve 28.13 with only 1.45M parameters. [PP-MSVSR](https://arxiv.org/pdf/2112.02828.pdf) is a multi-stage VSR deep architecture, with local fusion module, auxiliary loss and refined align module to refine the enhanced result progressively. Specifically, in order to strengthen the fusion of features across frames in feature propagation, a local fusion module is designed in stage-1 to perform local feature fusion before feature propagation. Moreover, an auxiliary loss in stage-2 is introduced to make the features obtained by the propagation module reserve more correlated information connected to the HR space, and introduced a refined align module in stage-3 to make full use of the feature information of the previous stage. Extensive experiments substantiate that PP-MSVSR achieves a promising performance of Vid4 datasets, which PSNR metric can achieve 28.13 with only 1.45M parameters.
Additionally, [PP-MSVSR](https://arxiv.org/pdf/2112.02828.pdf) provides two different models with 1.45M and 7.4M parameters in order to satisfy different requirements. Additionally, [PP-MSVSR](https://arxiv.org/pdf/2112.02828.pdf) provides two different models with 1.45M and 7.4M parameters in order to satisfy different requirements.
### EDVR ### EDVR
[EDVR](https://arxiv.org/pdf/1905.02716.pdf) wins the champions and outperforms the second place by a large margin in all four tracks in the NTIRE19 video restoration and enhancement challenges. The main difficulties of video super-resolution from two aspects: (1) how to align multiple frames given large motions, and (2) how to effectively fuse different frames with diverse motion and blur. First, to handle large motions, EDVR devise a Pyramid, Cascading and Deformable (PCD) alignment module, in which frame alignment is done at the feature level using deformable convolutions in a coarse-to-fine manner. Second, EDVR propose a Temporal and Spatial Attention (TSA) fusion module, in which attention is applied both temporally and spatially, so as to emphasize important features for subsequent restoration. [EDVR](https://arxiv.org/pdf/1905.02716.pdf) wins the champions and outperforms the second place by a large margin in all four tracks in the NTIRE19 video restoration and enhancement challenges. The main difficulties of video super-resolution from two aspects: (1) how to align multiple frames given large motions, and (2) how to effectively fuse different frames with diverse motion and blur. First, to handle large motions, EDVR devise a Pyramid, Cascading and Deformable (PCD) alignment module, in which frame alignment is done at the feature level using deformable convolutions in a coarse-to-fine manner. Second, EDVR propose a Temporal and Spatial Attention (TSA) fusion module, in which attention is applied both temporally and spatially, so as to emphasize important features for subsequent restoration.
...@@ -203,7 +203,7 @@ VSR quantitative comparis on the Vimeo90K, Vid4, UDM10 ...@@ -203,7 +203,7 @@ VSR quantitative comparis on the Vimeo90K, Vid4, UDM10
} }
``` ```
- 4. [PP-MSVSR: Multi-Stage Video Super-Resolution]() - 4. [PP-MSVSR: Multi-Stage Video Super-Resolution](https://arxiv.org/pdf/2112.02828.pdf)
``` ```
@article{jiang2021PP-MSVSR, @article{jiang2021PP-MSVSR,
......
...@@ -8,14 +8,14 @@ PaddleGAN提供一系列影像修复能力,包括 **[图片上色](./photo_col ...@@ -8,14 +8,14 @@ PaddleGAN提供一系列影像修复能力,包括 **[图片上色](./photo_col
``` ```
cd applications cd applications
python tools/video-enhance.py --input you_video_path.mp4 --process_order DAIN DeOldify EDVR --output output_dir python tools/video-enhance.py --input you_video_path.mp4 --process_order DAIN DeOldify PPMSVSR --output output_dir
``` ```
### **参数** ### **参数**
- `--input (str)`: 输入的视频路径。 - `--input (str)`: 输入的视频路径。
- `--output (str)`: 输出的视频路径。 - `--output (str)`: 输出的视频路径。
- `--process_order`: 调用的模型名字和顺序,比如输入为 `DAIN DeOldify EDVR`,则会顺序调用 `DAINPredictor` `DeOldifyPredictor` `EDVRPredictor` - `--process_order`: 调用的模型名字和顺序,比如输入为 `DAIN DeOldify PPMSVSR`,则会顺序调用 `DAINPredictor` `DeOldifyPredictor` `PPMSVSRPredictor`
<div align='center'> <div align='center'>
<img src='https://user-images.githubusercontent.com/48054808/117925494-e9a70400-b329-11eb-9f38-a48ef946a3a4.gif' width='600'> <img src='https://user-images.githubusercontent.com/48054808/117925494-e9a70400-b329-11eb-9f38-a48ef946a3a4.gif' width='600'>
......
# 视频分辨率提升 # 视频分辨率提升
针对视频超分,PaddleGAN提供了两种模型,[RealSR](#RealSR)[EDVR](#EDVR) 针对视频超分,PaddleGAN提供了七种模型,[RealSR](#RealSR)[PPMSVSR](#PPMSVSR)[PPMSVSRLarge](#PPMSVSRLarge)[EDVR](#EDVR)[BasicVSR](#BasicVSR)[IconVSR](#IconVSR)[BasiVSRPlusPlus](#BasiVSRPlusPlus)
## RealSR ## RealSR
[完整模型教程](https://github.com/PaddlePaddle/PaddleGAN/blob/develop/docs/zh_CN/tutorials/single_image_super_resolution.md) [完整模型教程](https://github.com/PaddlePaddle/PaddleGAN/blob/develop/docs/zh_CN/tutorials/single_image_super_resolution.md)
[RealSR](https://github.com/PaddlePaddle/PaddleGAN/blob/develop/docs/en_US/apis/apps.md#ppganappsrealsrpredictor)模型通过估计各种模糊内核以及实际噪声分布,为现实世界的图像设计一种新颖的真实图片降采样框架。基于该降采样框架,可以获取与真实世界图像共享同一域的低分辨率图像。并且提出了一个旨在提高感知度的真实世界超分辨率模型。对合成噪声数据和真实世界图像进行的大量实验表明,该模型能够有效降低了噪声并提高了视觉质量。 [RealSR](https://github.com/PaddlePaddle/PaddleGAN/blob/develop/docs/en_US/apis/apps.md#ppganappsrealsrpredictor)是图像超分模式,其通过估计各种模糊内核以及实际噪声分布,为现实世界的图像设计一种新颖的真实图片降采样框架。基于该降采样框架,可以获取与真实世界图像共享同一域的低分辨率图像。并且提出了一个旨在提高感知度的真实世界超分辨率模型。对合成噪声数据和真实世界图像进行的大量实验表明,该模型能够有效降低了噪声并提高了视觉质量。
<div align='center'> <div align='center'>
<img src='https://user-images.githubusercontent.com/48054808/117925551-02afb500-b32a-11eb-9a11-14e484daa953.png'> <img src='https://user-images.githubusercontent.com/48054808/117925551-02afb500-b32a-11eb-9a11-14e484daa953.png'>
...@@ -40,6 +40,86 @@ deep_remaster.run("/home/aistudio/Peking_input360p_clip6_5s.mp4") #原视频所 ...@@ -40,6 +40,86 @@ deep_remaster.run("/home/aistudio/Peking_input360p_clip6_5s.mp4") #原视频所
``` ```
## PPMSVSR
[完整模型教程](https://github.com/PaddlePaddle/PaddleGAN/blob/develop/docs/zh_CN/tutorials/video_super_resolution.md)
[PPMSVSR](https://github.com/PaddlePaddle/PaddleGAN/blob/develop/docs/en_US/apis/apps.md#ppganappsppmsvsrpredictor)为PaddleGAN自研的轻量视频超分模型,是一种多阶段视频超分深度架构,具有局部融合模块、辅助损失和细化对齐模块,以逐步细化增强结果。具体来说,在第一阶段设计了局部融合模块,在特征传播之前进行局部特征融合, 以加强特征传播中跨帧特征的融合。在第二阶段中引入了一个辅助损失,使传播模块获得的特征保留了更多与HR空间相关的信息。在第三阶段中引入了一个细化的对齐模块,以充分利用前一阶段传播模块的特征信息。大量实验证实,PP-MSVSR在Vid4数据集性能优异,仅使用 1.45M 参数PSNR指标即可达到28.13dB。
[PPMSVSR](https://github.com/PaddlePaddle/PaddleGAN/blob/develop/docs/en_US/apis/apps.md#ppganappsppmsvsrpredictor)模型是一个轻量视频超分模型,在当前轻量视频超分模型(模型参数量小于6M)中,PPMSVSR以最小的参数量在4个常用视频超分测试数据集Vimeo90K、Vid4、UDM10和REDS4上达到最优超分效果。
<div align='center'>
<img src='https://user-images.githubusercontent.com/79366697/145384020-a98c74df-a3b4-4477-a071-23605739ce80.png'>
</div>
```
ppgan.apps.PPMSVSRPredictor(output='output', weight_path=None, num_frames=10)
```
### 参数
- `output (str,可选的)`: 输出的文件夹路径,默认值:`output`.
- `weight_path (None,可选的)`: 载入的权重路径,如果没有设置,则从云端下载默认的权重到本地。默认值:`None`
- `num_frames (int,可选的)`: 模型输入帧数,默认值:`10`。模型输入帧数设置的越大,模型超分效果越好.
### 使用方式
**1. API预测**
```
from ppgan.apps import PPMSVSRPredictor
sr = PPMSVSRPredictor()
# 测试一个视频文件
sr.run("/home/aistudio/Peking_input360p_clip6_5s.mp4") #原视频所在路径
```
**2. 命令行预测**
```
!python applications/tools/video-enhance.py --input /home/aistudio/Peking_input360p_clip6_5s.mp4 \ #原视频路径
--process_order PPMSVSR \ #对原视频处理的顺序,此处注意“EDVR”四个字母都需大写
--output output_dir #成品视频所在的路径
```
## PPMSVSRLarge
[完整模型教程](https://github.com/PaddlePaddle/PaddleGAN/blob/develop/docs/zh_CN/tutorials/video_super_resolution.md)
[PPMSVSRLarge](https://github.com/PaddlePaddle/PaddleGAN/blob/develop/docs/en_US/apis/apps.md#ppganappsppmsvsrlargepredictor)为PaddleGAN自研的高精度超分模型,是一种多阶段视频超分深度架构,具有局部融合模块、辅助损失和细化对齐模块,以逐步细化增强结果。具体来说,在第一阶段设计了局部融合模块,在特征传播之前进行局部特征融合, 以加强特征传播中跨帧特征的融合。在第二阶段中引入了一个辅助损失,使传播模块获得的特征保留了更多与HR空间相关的信息。在第三阶段中引入了一个细化的对齐模块,以充分利用前一阶段传播模块的特征信息。
[PPMSVSRLarge](https://github.com/PaddlePaddle/PaddleGAN/blob/develop/docs/en_US/apis/apps.md#ppganappsppmsvsrlargepredictor)模型是为满足精度提升,对PPMSVSR通过增加基础快数量而构造的一个大模型。PPMSVSRLarge与当前精度最高的BasicVSR++模型相比,以相似的参数量达到了更高的精度。
```
ppgan.apps.PPMSVSRLargePredictor(output='output', weight_path=None, num_frames=10)
```
### 参数
- `output (str,可选的)`: 输出的文件夹路径,默认值:`output`.
- `weight_path (None,可选的)`: 载入的权重路径,如果没有设置,则从云端下载默认的权重到本地。默认值:`None`
- `num_frames (int,可选的)`: 模型输入帧数,默认值:`10`。模型输入帧数设置的越大,模型超分效果越好.
### 使用方式
**1. API预测**
```
from ppgan.apps import PPMSVSRLargePredictor
sr = PPMSVSRLargePredictor()
# 测试一个视频文件
sr.run("/home/aistudio/Peking_input360p_clip6_5s.mp4") #原视频所在路径
```
**2. 命令行预测**
```
!python applications/tools/video-enhance.py --input /home/aistudio/Peking_input360p_clip6_5s.mp4 \ #原视频路径
--process_order PPMSVSRLarge \ #对原视频处理的顺序,此处注意“EDVR”四个字母都需大写
--output output_dir #成品视频所在的路径
```
## EDVR ## EDVR
[完整模型教程](https://github.com/PaddlePaddle/PaddleGAN/blob/develop/docs/zh_CN/tutorials/video_super_resolution.md) [完整模型教程](https://github.com/PaddlePaddle/PaddleGAN/blob/develop/docs/zh_CN/tutorials/video_super_resolution.md)
...@@ -65,18 +145,12 @@ ppgan.apps.EDVRPredictor(output='output', weight_path=None) ...@@ -65,18 +145,12 @@ ppgan.apps.EDVRPredictor(output='output', weight_path=None)
### 使用方式 ### 使用方式
**1. API预测** **1. API预测**
目前API预测方式只支持在静态图下运行,需加上启动静态图命令,后续会支持动态图,敬请期待~
``` ```
paddle.enable_static()
from ppgan.apps import EDVRPredictor from ppgan.apps import EDVRPredictor
sr = EDVRPredictor() sr = EDVRPredictor()
# 测试一个视频文件 # 测试一个视频文件
sr.run("/home/aistudio/Peking_input360p_clip6_5s.mp4") #原视频所在路径 sr.run("/home/aistudio/Peking_input360p_clip6_5s.mp4") #原视频所在路径
paddle.disable_static()
``` ```
**2. 命令行预测** **2. 命令行预测**
...@@ -87,7 +161,119 @@ paddle.disable_static() ...@@ -87,7 +161,119 @@ paddle.disable_static()
--output output_dir #成品视频所在的路径 --output output_dir #成品视频所在的路径
``` ```
## BasicVSR
[完整模型教程](https://github.com/PaddlePaddle/PaddleGAN/blob/develop/docs/zh_CN/tutorials/video_super_resolution.md)
[BasicVSR](https://github.com/PaddlePaddle/PaddleGAN/blob/develop/docs/en_US/apis/apps.md#ppganappsbasicvsrpredictor)在VSR的指导下重新考虑了四个基本模块(即传播、对齐、聚合和上采样)的一些最重要的组件。 通过添加一些小设计,重用一些现有组件,得到了简洁的 BasicVSR。与许多最先进的算法相比,BasicVSR在速度和恢复质量方面实现了有吸引力的改进。
```
ppgan.apps.BasicVSRPredictor(output='output', weight_path=None, num_frames=10)
```
### 参数
- `output (str,可选的)`: 输出的文件夹路径,默认值:`output`.
- `weight_path (None,可选的)`: 载入的权重路径,如果没有设置,则从云端下载默认的权重到本地。默认值:`None`
- `num_frames (int,可选的)`: 模型输入帧数,默认值:`10`。模型输入帧数设置的越大,模型超分效果越好.
### 使用方式
**1. API预测**
```
from ppgan.apps import BasicVSRPredictor
sr = BasicVSRPredictor()
# 测试一个视频文件
sr.run("/home/aistudio/Peking_input360p_clip6_5s.mp4") #原视频所在路径
```
**2. 命令行预测**
```
!python applications/tools/video-enhance.py --input /home/aistudio/Peking_input360p_clip6_5s.mp4 \ #原视频路径
--process_order BasicVSR \ #对原视频处理的顺序,此处注意“EDVR”四个字母都需大写
--output output_dir #成品视频所在的路径
```
## IconVSR
[完整模型教程](https://github.com/PaddlePaddle/PaddleGAN/blob/develop/docs/zh_CN/tutorials/video_super_resolution.md)
[IconVSR](https://github.com/PaddlePaddle/PaddleGAN/blob/develop/docs/en_US/apis/apps.md#ppganappsiconvsrpredictor)是由BasicVSR扩展而来,其是在BasicVSR基础之上,通过添加信息重新填充机制和耦合传播方案以促进信息聚合。与BasicVSR相比,IconVSR提升了一点精度。
```
ppgan.apps.IconVSRPredictor(output='output', weight_path=None, num_frames=10)
```
### 参数
- `output (str,可选的)`: 输出的文件夹路径,默认值:`output`.
- `weight_path (None,可选的)`: 载入的权重路径,如果没有设置,则从云端下载默认的权重到本地。默认值:`None`
- `num_frames (int,可选的)`: 模型输入帧数,默认值:`10`。模型输入帧数设置的越大,模型超分效果越好.
### 使用方式
**1. API预测**
```
from ppgan.apps import IconVSRPredictor
sr = IconVSRPredictor()
# 测试一个视频文件
sr.run("/home/aistudio/Peking_input360p_clip6_5s.mp4") #原视频所在路径
```
**2. 命令行预测**
```
!python applications/tools/video-enhance.py --input /home/aistudio/Peking_input360p_clip6_5s.mp4 \ #原视频路径
--process_order IconVSR \ #对原视频处理的顺序,此处注意“EDVR”四个字母都需大写
--output output_dir #成品视频所在的路径
```
## BasiVSRPlusPlus
[完整模型教程](https://github.com/PaddlePaddle/PaddleGAN/blob/develop/docs/zh_CN/tutorials/video_super_resolution.md)
[BasiVSRPlusPlus](https://github.com/PaddlePaddle/PaddleGAN/blob/develop/docs/en_US/apis/apps.md#ppganappsbasicvsrpluspluspredictor)通过提出二阶网格传播和导流可变形对齐来重新设计BasicVSR。通过增强传播和对齐来增强循环框架,BasicVSR++可以更有效地利用未对齐视频帧的时空信息。 在类似的计算约束下,新组件可提高性能。特别是,BasicVSR++ 以相似的参数数量在 PSNR 方面比 BasicVSR 高0.82dB。BasicVSR++ 在NTIRE2021的视频超分辨率和压缩视频增强挑战赛中获得三名冠军和一名亚军。
<div align='center'>
<img src='https://user-images.githubusercontent.com/79366697/145386802-5533f3df-a52b-4917-aa72-20e91833f53c.jpg'>
</div>
```
ppgan.apps.BasiVSRPlusPlusPredictor(output='output', weight_path=None, num_frames=10)
```
### 参数
- `output (str,可选的)`: 输出的文件夹路径,默认值:`output`.
- `weight_path (None,可选的)`: 载入的权重路径,如果没有设置,则从云端下载默认的权重到本地。默认值:`None`
- `num_frames (int,可选的)`: 模型输入帧数,默认值:`10`。模型输入帧数设置的越大,模型超分效果越好.
### 使用方式
**1. API预测**
```
from ppgan.apps import BasiVSRPlusPlusPredictor
sr = BasiVSRPlusPlusPredictor()
# 测试一个视频文件
sr.run("/home/aistudio/Peking_input360p_clip6_5s.mp4") #原视频所在路径
```
**2. 命令行预测**
```
!python applications/tools/video-enhance.py --input /home/aistudio/Peking_input360p_clip6_5s.mp4 \ #原视频路径
--process_order BasiVSRPlusPlus \ #对原视频处理的顺序,此处注意“EDVR”四个字母都需大写
--output output_dir #成品视频所在的路径
```
### 在线项目体验 ### 在线项目体验
**1. [老北京城影像修复](https://aistudio.baidu.com/aistudio/projectdetail/1161285)** **1. [PaddleGAN SOTA算法:视频超分模型PP-MSVSR详解及应用](https://aistudio.baidu.com/aistudio/projectdetail/3205183)**
**2. [老北京城影像修复](https://aistudio.baidu.com/aistudio/projectdetail/1161285)**
**2. [PaddleGAN ❤️ 520特辑](https://aistudio.baidu.com/aistudio/projectdetail/1956943?channelType=0&channel=0)** **3. [PaddleGAN ❤️ 520特辑](https://aistudio.baidu.com/aistudio/projectdetail/1956943?channelType=0&channel=0)**
...@@ -9,7 +9,7 @@ ...@@ -9,7 +9,7 @@
### ⭐ PP-MSVSR ⭐ ### ⭐ PP-MSVSR ⭐
百度自研的[PP-MSVSR](https://arxiv.org/pdf/2112.02828.pdf)是一种多阶段视频超分深度架构,具有局部融合模块、辅助损失和细化对齐模块,以逐步细化增强结果。具体来说,在第一阶段设计了局部融合模块,在特征传播之前进行局部特征融合, 以加强特征传播中跨帧特征的融合。在第二阶段中引入了一个辅助损失,使传播模块获得的特征保留了更多与HR空间相关的信息。在第三阶段中引入了一个细化的对齐模块,以充分利用前一阶段传播模块的特征信息。大量实验证实,PP-MSVSR在Vid4数据集性能优异,仅使用 1.45M 参数PSNR指标即可达到28.13dB。 百度自研的[PP-MSVSR](https://arxiv.org/pdf/2112.02828.pdf)是一种多阶段视频超分深度架构,具有局部融合模块、辅助损失和细化对齐模块,以逐步细化增强结果。具体来说,在第一阶段设计了局部融合模块,在特征传播之前进行局部特征融合, 以加强特征传播中跨帧特征的融合。在第二阶段中引入了一个辅助损失,使传播模块获得的特征保留了更多与HR空间相关的信息。在第三阶段中引入了一个细化的对齐模块,以充分利用前一阶段传播模块的特征信息。大量实验证实,PP-MSVSR在Vid4数据集性能优异,仅使用 1.45M 参数PSNR指标即可达到28.13dB。
[PP-MSVSR](https://arxiv.org/pdf/2112.02828.pdf)提供两种体积模型,开发者可根据实际场景灵活选择:[PP-MSVSR](https://arxiv.org/pdf/2112.02828.pdf)(参数量1.45M)与[PP-MSVSR-L](https://arxiv.org/pdf/2112.02828.pdf)(参数量7.42)。 [PP-MSVSR](https://arxiv.org/pdf/2112.02828.pdf)提供两种体积模型,开发者可根据实际场景灵活选择:[PP-MSVSR](https://arxiv.org/pdf/2112.02828.pdf)(参数量1.45M)与[PP-MSVSR-L](https://arxiv.org/pdf/2112.02828.pdf)(参数量7.42)。
### EDVR ### EDVR
...@@ -27,7 +27,7 @@ ...@@ -27,7 +27,7 @@
这里提供4个视频超分辨率常用数据集,REDS,Vimeo90K,Vid4,UDM10。其中REDS和vimeo90k数据集包括训练集和测试集,Vid4和UDM10为测试数据集。将需要的数据集下载解压后放到``PaddleGAN/data``文件夹下 。 这里提供4个视频超分辨率常用数据集,REDS,Vimeo90K,Vid4,UDM10。其中REDS和vimeo90k数据集包括训练集和测试集,Vid4和UDM10为测试数据集。将需要的数据集下载解压后放到``PaddleGAN/data``文件夹下 。
REDS([数据下载](https://seungjunnah.github.io/Datasets/reds.html))数据集是NTIRE19公司最新提出的高质量(720p)视频数据集,其由240个训练片段、30个验证片段和30个测试片段组成(每个片段有100个连续帧)。由于测试数据集不可用,这里在训练集选择了四个具有代表性的片段(分别为'000', '011', '015', '020',它们具有不同的场景和动作)作为测试集,用REDS4表示。剩下的训练和验证片段被重新分组为训练数据集(总共266个片段)。 REDS([数据下载](https://seungjunnah.github.io/Datasets/reds.html))数据集是NTIRE19比赛最新提出的高质量(720p)视频数据集,其由240个训练片段、30个验证片段和30个测试片段组成(每个片段有100个连续帧)。由于测试数据集不可用,这里在训练集选择了四个具有代表性的片段(分别为'000', '011', '015', '020',它们具有不同的场景和动作)作为测试集,用REDS4表示。剩下的训练和验证片段被重新分组为训练数据集(总共266个片段)。
处理后的数据集 REDS 的组成形式如下: 处理后的数据集 REDS 的组成形式如下:
``` ```
...@@ -198,7 +198,7 @@ Vimeo90K,Vid4,UDM10测试数据集上超分性能对比 ...@@ -198,7 +198,7 @@ Vimeo90K,Vid4,UDM10测试数据集上超分性能对比
} }
``` ```
- 4. [PP-MSVSR: Multi-Stage Video Super-Resolution]() - 4. [PP-MSVSR: Multi-Stage Video Super-Resolution](https://arxiv.org/pdf/2112.02828.pdf)
``` ```
@article{jiang2021PP-MSVSR, @article{jiang2021PP-MSVSR,
......
Markdown is supported
0% .
You are about to add 0 people to the discussion. Proceed with caution.
先完成此消息的编辑!
想要评论请 注册