enables automatic content creation (Ramesh et al., 2021) and manipulation (Yu et al., 2018) by

selecting a part of an image to be cleared out and getting an ML algorithm to fill it in. For example,

you can see in Figure 1 (a) the recent Bolsonaro supporter’s attack on the Brazilian Congress.

The same image on the right (b) shows one of these supports removed by inpainting. The inpainter

here does a very good job with very little effort. We know this because it took seconds to do it

ourselves mainly spent on creating a decent mask. This short anecdote about recreating a mask will

be important later in the report. For now, we know that it is hard for an untrained eye to tell that

this image was edited and even harder to where this edition occurred (we circled it in red for clarity).

expand on in the theoretical background. We will then explain the neural network (NN) architecture

implementations, divided into three main steps. First, we will use a NN to inpaint images, which will

constitute our attack. Then, we will fit a different NN able to, by implementing some perturbations,

we will make the watermark harder to remove, constituting our defense. Later, we will use a third

NN to classify the image as one with or without a watermark. Finally, we will analyze how

different parameters affect both the attack and the defense. Most interestingly, we will show how

the unification of these three NN constructs a Generative Adversarial Network (GAN) and what

this implies.

Network (Deep CNN). This class of networks has become particularly popular for image genera-

tion and restoration. Yet, their excellent performance can always end in the wrong hands. In the

context of this project, it proved to be incredibly easy to remove a watermark with many online

available inpainter codes, with minor adaptions needed. We will take advantage of the structure

of a generator network, discussed in the Deep Image Prior paper, as sufficient to capture the prior

to any learning. In their paper, randomly-initialized neural networks are used as handcrafted prior

with excellent results when inpainting [4].

remove watermarks. The opposition is that ML inpainters can be manipulated using techniques

adapted from the field of adversarial machine learning. We will adapt the term ”markpainting” to

ness on a white background for the NN to be able to adapt to different contexts. Then, a specific

mask in a trained network will clearly do the job. Different inpainting NNs already can do this

quite well and even keep improving by generating self-tailored masks.

to answer. In a general sense, even if new defense techniques arise constantly, it is undeniable that

the attacks have the upper hand. Attacks don’t need to wait to find different ways to attack, while

there is an underlying delay in the defense’s need to adapt their response. More specifically in this

report, we discuss how watermarks can be more robust, which is clearly a challenge with NN being

so clever to reconstruct images. Yet, as with any perturbed image properly intervened, we can

find a specific configuration that doesn’t damage significantly the original image, but is still able

outputs a binary result based on the probability that an attack was successful or not.

4.1. Datasets. For combining and running the three NN implemented in this report, we searched

for super-resolution and inpainting examples. Both attack and defense algorithms were pre-trained

and had access to their saved model-trained epochs.

4.2. Repositories. In addition to the datasets and models mentioned above, three main reposito-

ries were used throughout this project: one for inpainting, one to generate adversarial images, and

a watermark classification algorithm. They all had to be slighted modified to be adapted to our

experiments, and are divided into three ipynb files, respectively.

4.3. Inpainting. The inpainting algorithm we used was developed by D. Ulyanov et al. [6].

They used a neural network that is initialized randomly to be effectively used as a pre-defined prior

in common inverse problems. The architecture of this model is shown in Figure 2.

4.4. Markpainting. For this part of the project, we needed an algorithm that took the image and

added some perturbation to it, so that the inpainting would fail to reconstruct the image without

a watermark. Let’s say we feed the attacked image X, which is classified by our model (M) as an

image without a watermark. We want to find adversarial image

L

adversarial image [8].

of a loss function, as that would be an implementation of Fast Gradient Descent Method. We

do this by implementing five different types of alterations to the image. Namely, we test five

different transformations: JPEG compression, low-pass filtering, Gaussian blurring, white noise,

and brightness adjustments. For JPEG compression, we explore its effects at different levels of ϵ.

this classifier has a very good performance, classifying images either as ”watermarked” (0) or ”clear”

(1) [9].

the inpainting should be applied on. We want to point out that generating an automatic mask is

possible and can be implemented. Our first experiment intends to highlight its importance, intro-

duced with the outcome of the successful attack shown in Figure 4.

Intuitively, tailored-fitted masks should perform better than generalized ones. Yet, in our results,

we could argue that this is not the full story. Inpainting of Vincent Van Gogh’s self-portrait requires

both local and global knowledge of the image. With just local knowledge, the watermark can be

1 (see Figure 5). On the other extreme, the image on the right features an exaggerated mask that

does not allow the algorithm to distinguish that there should be a beard. With this experiment

we want to illustrate how running the attacking algorithm with an appropriate mask should give

us successful watermark removals. Thus, inpainting techniques are clearly sensitive to the mask,

preventing an unsuited one from reconstructing an image optimally.

we want the inpainter algorithm to fail by adversarial examples. As expected, each manipulation

significantly increased defense performance, with a different impact on the inpainting. Figure 6

them, but how the level of alteration can play a role is investigated. Extreme cases show a big

impediment to deteriorating the image in its process, but we exaggerate these perturbations to

evaluate better our results and for them to be visually clearer (see Appendix B. As seen in

Figure 7, we have an original image with a generalized mask, followed by the attacked images

at different levels of random perturbations. Notice that the watermark of the image with zero

perturbation is completely removed. Meanwhile, the watermarks of the rest of the images are still

somewhat visible. Images with lower levels of perturbation do not preserve the watermark as well

as the higher ones do. Yet, as mentioned before, we want images to look as similar to the original

as possible, which is not the case at high perturbations. In the end, a perturbation between 0.05

We also found optimal values for epsilon, which are in the range of [0.05,0.1]. Specifically, in Table

1 we found that epsilon values of 0.10 or below are not sufficient to make the inpainting algorithm

fail. Contrarily, epsilon values of 0.05 or above have proved to be enough to make the inpainting

NN fail in image reconstruction since the classifier is able to recognize them as watermarked.