The base model for x2 and x4 scale are based on EDSR and SRResNet architectures, respectively.

We count only matrix multiplications operations of binarized tensors as binary multiply–accumulate operations (MACs). You can binarize the matrix operation of the model in any way you want, but some requirements must be satisfied, to count this operation as the binary.

For simplicity, we provide the baseline code and the training pipeline. You can use it to train your models and create and check your submission.

Evaluation is performed on closed test dataset: ShadowSet.

The metric for leaderboard (final metric) is based on computation complexity metric and its quality - PSNR metric value for each model.

The complexity metric of the model depends on the amount of MACs and defined by the following formula (the same for x4 model):

$$\text{compexity}\big(\text{model}^{\text{bin}}_{x2}\big) = c^{\text{bin}}_{x2} = {1 \over \text{MAC}^{\text{fp}}\big(\text{model}^{\text{fp}}_{x2}\big)}\bigg(\text{MAC}^{\text{fp}} \big(\text{model}^{\text{bin}}_{x2}\big) + {1 \over 8} \text{MAC}^{\text{bin}} \big(\text{model}^{\text{in}}_{x2}\big)\bigg)$$

where the functions MAC$^\text{fp}$(⋅) and MAC$^{bin}$(⋅) denote the number of MAC operations for float point (fp) and binary operations (bin) respectively. The values MAC$^\text{fp}$(model$_{x2}^{\text{fp}}$) and MAC$^\text{fp}$(model$_{x4}^{\text{fp}}$) are calculated and strictly fixed, these values are corresponds to the checkpoints of the float point models model$_{x2}^{\text{fp}}$ and model$_{x4}^{\text{fp}}$ respectively.

Quality metrics psnr$_{x2}^{\text{fp}}$, psnr$_{x4}^{\text{fp}}$, psnr$_{x2}^{\text{bin}}$, psnr$_{x4}^{\text{bin}}$ for float point and binarized $\textbf{x2}$ and $\textbf{x4}$ models, respectively, are based on average PSNR metric value on test dataset.

The score is calculated for each model separately and based on complexity metric and the model quality (PSNR). The formula for the score is the following (the same for x4 model):

$$\text{score}\big(\text{model}_{x2}^{\text{bin}}\big) = \begin{cases} {1 - \epsilon_{x2}^{c} \over \epsilon_{x2}^{\text{psnr}}} \cdot \bigg( \text{psnr}_{x2}^{\text{bin}} + \epsilon_{x2}^{\text{psnr}} - \text{psnr}_{x2}^{\text{fp}} \bigg) + \bigg(1 - \epsilon_{x2}^{c} - c_{x2}^{\text{bin}} \bigg), \quad \text{psnr}_{x2}^{\text{bin}} + \epsilon_{x2}^{\text{psnr}} < \text{psnr}_{x2}^{\text{fp}} , \\ {\epsilon_{x2}^{c} \over \epsilon_{x2}^{\text{psnr}}} \cdot \bigg( \text{psnr}_{x2}^{\text{bin}} + \epsilon_{x2}^{\text{psnr}} - \text{psnr}_{x2}^{\text{fp}} \bigg) + \bigg(1 - \epsilon_{x2}^{c} - c_{x2}^{\text{bin}} \bigg), \quad \text{otherwise} \end{cases}$$

The $\epsilon_{x2}^{c}$, $\epsilon_{x4}^{c}$ are the complexity metric correction constants, the $\epsilon_{x2}^{\text{psnr}}$, $\epsilon_{x4}^{\text{psnr}}$ are the margin of error that is used to determine whether model$^{\text{bin}}_{x2}$ or model$^{\text{bin}}_{x4}$ is lossless or not. These constants and psnr$^{\text{fp}}_{x2}$, psnr$^{\text{fp}}_{x2}$ are predefined and strictly fixed. You can find its values in baseline solution repository.

To have better understanding of the scoring metric function for the model one can see on the following picture:

For the FP model the score is equal to the zero. In general, to obtain better score the complexity should be reduced and the quality should more than psnr$^{\text{fp}}_{x\cdot}$ $- \epsilon_{x\cdot}^{\text{psnr}}$. One can see the projection on complexity vs. PSNR surface, $\textbf{the greener, the better}$.

Notes:

1. If the score (model$^{\text{bin}}_{x\cdot}$) will be less than 0.0 it will be replaced with zero as score for model$^{\text{fp}}_{x\cdot}$

2.If the compexity (model$^{\text{bin}}_{x\cdot}$) will be more than 1.0 the model will be replaced with model$^{\text{fp}}_{x\cdot}$, so the score will be zero as for model$^{\text{fp}}_{x\cdot}$.

The score for leaderboard is the weighted of the two scores for model$^{\text{bin}}_{x2}$ or model$^{\text{bin}}_{x4}$: $$\text{FinalScore}(\text{model}^{\text{bin}}_{x2}, \text{model}^{\text{bin}}_{x4}) = 0.4\cdot \text{score}(\text{model}^{\text{bin}}_{x2}) + 0.6\cdot \text{score}(\text{model}^{\text{bin}}_{x4}).$$

If the model$^{\text{bin}}_{x2}$ or model$^{\text{bin}}_{x4}$ will not be provided the score for this model will be equal to zero as for model$^{\text{fp}}_{x2}$ or model$^{\text{fp}}_{x4}$