Contrast loss for Siamese networks with Keras and TensorFlow - PyImageSearch (2023)

In this tutorial, you will learn more about contrast loss and how it can be used to train more accurate Siamese neural networks. We will implement the contrast loss using Keras and TensorFlow.

Previously, I wrote a three-part series on the basics of Siamese neural networks:

1. Creating image pairs for Siamese networks with Python
2. Siamese Networks with Keras, TensorFlow and Deep Learning
3. Compare images for similarity with Siamese networks, Keras and TenorFlow

This series covered the basics of Siamese networks, including:

• create pairs of images
• Implement Siamese neural network architecture
• Using binary cross-input to train Siamese network

But while the binary cross entropy is certainly onevalidchoice of loss function, it's not thatOnlyChoice (or even thebetterSelection).

State of the art Siamese networks tend to use some form of contrast loss or triplet loss when training– These loss functions are better suited to Siamese networks and tend to improve accuracy.

By the end of this guide you will understand how to implement Siamese networks and train them with loss of contrast.

To learn how to train a Siamese neural network with loss of contrast,just keep reading.

Contrast loss for siamese networks with Keras and TensorFlow

In the first part of this tutorial, we'll discuss what contrast loss is and, more importantly, how it can be used to train Siamese neural networks more accurately and effectively.

Next, let's set up our development environment and review our project directory structure.

We need to implement several Python scripts today, including:

• A configuration file
• Utilities to generate image pairs, track training history, and implement custom levels
• Our implementation of contrast agent loss
• A training schedule
• A test/inference script

We'll look at each of these scripts; However, some of them have been coveredin my previous guides on Siamese Neural Networks, so I may refer you to my other tutorials for more details.

We'll also spend a lot of time discussing our implementation of contrast loss to make sure you understand what you're doing, how it works, and why we're using it.

By the end of this tutorial, you will have a fully working contrast loss implementation capable of training a Siamese neural network.

What is contrast loss? And how can the loss of contrast be used to train Siamese networks?

In our previous Siamese Neural Networks tutorial series, we learned how to train a Siamese network using the binary cross entropy loss function:

Binary cross-entropy was a valid choice here since we are essentially doing a 2-class classification:

1. Either the two images presented to the network belong to thesame class
2. Either the two pictures belong to itdifferent classes

So framed we have a classification problem. And since we only have two classes, the binary cross entropy makes sense.

However, there really is a loss functionmuch better suitednamed for Thai networksloss of contrast:

paraphraseHarshavardhan Gupta, we must remember that the goal of a Siamese network is notclassifya set of image pairs, but insteaddistinguish between them.Essentially, the loss of contrast evaluates how good the Siamese network is at distinguishing between pairs of images. The difference is subtle but incredibly important.

To decompose this equation:

• ÖValue is our label. It will beif the pairs of images belong to the same class, and it will beif the image pairs belong to another class.
• ÖVariable is the Euclidean distance between the exits of the sister network fusions.
• Ömaximalfunction takes the greatest value ofand the margin, minus the distance.

We will implement this loss function later in this tutorial using Keras and TensorFlow.

If you want more mathematically motivated details on contrast loss, be sure to read Hadsell et al.'s article,Dimensionality reduction by learning an invariant mapping.

Set up your development environment

This Siamese network tutorial series uses Keras and TensorFlow. If you intend to follow this tutorial in the previous two parts of this series, I suggest you take the time to set up your deep learning development environment now.

You can use either of these two guides to install TensorFlow and Keras on your system:

• How to install TensorFlow 2.0 on Ubuntu
• How to install TensorFlow 2.0 on macOS

Each of the tutorials will help you set up your system in a convenient Python virtual environment with all the software required for this blog post.

Having trouble setting up your development environment?

All in all you are:

• Shortly?
• Learning in the administratively closed system of your employer?
• Want to save yourself the hassle of the command line, package managers, and virtual environments?
• Ready to run the codeat the momenton your Windows, macOS or Linux system?

so join usPyImageSearch PlusToday!

Get access to Jupyter Notebooks for this tutorial and other PyImageSearch guidespreconfiguredin the Google Colab ecosystem right in your browser!No installation required.

Best of all, these Jupyter notebooks run on Windows, macOS and Linux!

Project structure

Today's tutorial on Contrast Loss in Siamese Networks builds on my three previous tutorials, which cover the basics of building image pairs, implementing and training Siamese networks, and using Siamese networks for inference:

1. Creating image pairs for Siamese networks with Python
2. Siamese Networks with Keras, TensorFlow and Deep Learning
3. Compare images for similarity using Siamese networks, Keras and TensorFlow

We're going to be building on the knowledge we've gained from those guides (including the directory structure of the project itself) today, so consider previous guidesrequired readingbefore continuing today.

Once you're up to speed, we can proceed to review our project directory structure:

$arvore. --dirsfirst.├── exemplos│ ├── bild_01.png│ ├── bild_02.png│ ├── bild_03.png...│ └── bild_13.png├── output│ ├_messia │ ├── ─ ─ assets│ │ ├── variables│ │ │ ├── variables.data-00000-of-00001│ │ │ └── variables.index│ │ └── save_model.pb│ └b─p contrast.p pyimagesearch│ ├── config.py│ ├──metrics.py│ ├── siamese_network.py│ └── utils.py├── test_contrastive_siamese_network.py└── train_contraiess_network, siamese_2

Withinimage searchModule you will find four Python files:

We then have two Python driver scripts:

Again, I can't stress the importance of checking mineprevious series of tutorials on siamese networks. doing this is aabsolute requirementbefore we continue here today.

Implementation of our configuration file

Our configuration file contains important variables used to train our contrast-losing Siamese network.

open thisconfig.pyfile in your project's directory structure and let's take a look inside:

# import the necessary packages import os# specify the shape of the inputs for our networkIMG_SHAPE = (28, 28, 1)# specify the batch size and the number of epochsBATCH_SIZE = 64EPOCHS = 100# set the path to the output directory baseBASE_OUTPUT = "output" # Use base output path to derive path to serialized model# along with training history, "contrastive_plot.png"])

line 5defines ourIMG_SHAPEDimensions. We will work with the MNIST digit data set, which has28×28Grayscale images (i.e. single channel).

Then we define oursCHARGENGRÖSSEand number ofSEASONStrain beforehand. These parameters were adjusted experimentally.

Lines 16-19Define the output file paths for our serialized model and training history.

See my tutorial for more details on the configuration fileSiamese Networks with Keras, TensorFlow and Deep Learning.

Creating our auxiliary service functions

To train our Siamese network model, we need three utilities:

Let's start with our imports:

# import the necessary packages import tensorflow.keras.backend as kimport matplotlib.pyplot as pltimport numpy as np

then we have oursmake_pairsFeature that I have discussed extensively in myCreating Siamese Network Image Pairs Using Python Tutorial(Be sure to read this guide before proceeding):

def make_pairs(images, labels):# initializes two empty lists to hold the pairs (image, image) and# labels to indicate whether a pair is positive or negative pairImages = []pairLabels = []# computes the total number of existing classes o dataset# and then create an index list for each class label that# gives the indices for all examples with a given label numClasses = len(np.unique(labels))idx = [np.where(labels) == i) [0 ] for i in range(0, numClasses)]# Loop through all images for idxA in range(len(images)):# get the current image and the label belonging to the current one# iterationcurrentImage = images [idxA]label = labels [ idxA] # randomly choose an image that belongs to the *same* class# labelidxB = np.random.choice(idx[label])posImage = images[idxB]# prepare a positive pair and update the images and labels # lists or pairImages .append ([currentImage, posImage])pairLabels.append([1])# receives the indices for each of the class labels ls *not* equal# current label and randomly choose an image and corresponding# label *not* equal to current labelnegIdx = np.where(labels != label)[0]negImage = images[np.random.choice(negIdx )] # prepare a negative image pair and update our list. np.array(pairLabels))

The next functionEuclidean distance, accepts a 2-tuple ofvectorsand then calculates the Euclidean distance between them using the Keras/TensorFlow functions, so the Euclidean distance can be calculatedwithinThe Siamese Neural Network:

def euclidean_distance(vectors):# unpack the vectors into separate lists (featsA, featsB) = vectors# calculate the sum of the squared distances between vectors ensumSquared = K.sum(K.square(featsA - featsB), axis=1,keepdims = True )# returns the Euclidean distance between vectors return K.sqrt(K.maximum(sumSquared, K.epsilon()))

Finally, we have a utilityplot_training, which accepts aPlotPfad, plots our training and validation contrast loss over training, then saves the plot to disk:

def plot_training(H, plotPath):# Create a plot that plots and saves or traininghistoryplt.style.use("ggplot")plt.figure()plt.plot(H.history["loss"], label="train_loss " )plt.plot(H.history["val_loss"], label="val_loss")plt.title("Training Loss")plt.xlabel("Epoch #")plt.ylabel("Loss")plt .legend( loc="short bottom margin")plt.savefig(plotPath)

Let's proceed to the implementation of the Siamese network architecture itself.

Implementation of our Siamese network architecture

Our Siamese neural network architecture is essentially a basic CNN:

# Import packages required by tensorflow.keras.models import Modelfrom tensorflow.keras.layers import Inputfrom tensorflow.keras.layers import Conv2Dfrom tensorflow.keras.layers import Densefrom tensorflow.keras.layers import Dropoutfrom tensorflow.keras.layers import GlobalAveragePooling2D from tensorflow. keras .layers import MaxPooling2Ddef build_siamese_model(inputShape, embeddingDim=48):# defined specifically as an input to extract network inputs = Input(inputShape)# or primarily connected to CONV => RELU => POOL => DROPOUT layerx = Conv2D(64 , ( 2, 2), padding="same", activation="relu")(inputs)x = MaxPooling2D(pool_size=(2, 2))(x)x = Dropout(0.3)(x)# segundo conjunto de CONV = > RELU => POOL => DROPOUT layerx = Conv2D(64, (2, 2), padding="same", activation="relu")(x)x = MaxPooling2D(pool_size=2)(x)x = Dropout( 0.3)(x)# Prepare as said finaispooledOutput = GlobalAveragePooling2D()(x)outputs = Dense(embeddingDim)(pooledOutput)# construa o modelomodel = Model(input s, exits)# retorne o modelo para o modelo de retorno da função de chamada

You can consult my tutorial onSiamese Networks with Keras, TensorFlow and Deep Learningfor more details on the architecture and implementation of the model.

Implementation of contrast loss with Keras and TensorFlow

With our helper utilities and our model architecture, we can define thatloss of contrastFeature in Keras/TensorFlow.

For reference, here is the contrast loss function equation that we will implement in the Keras/TensorFlow code:

The full implementation of the contrast loss is concise and only 18 lines, including comments:

# Import required packages, import tensorflow.keras.backend as KImport tensorflow as tfdef contrastive_loss(y, preds, margin=1):# Explicitly cast the true datatype of the class label to the predicted datatype of the class label # (otherwise we take the risk two # having separate datatypes, which causes bugs in TensorFlow)y = tf.cast(y, preds.dtype)# calculates the loss of contrast between the true labels and# the predicted labelssquaredPreds = K.square(preds)squaredMargin = K.square( K.maximum(margin - preds, 0))loss = K.mean(y * squaredPreds + (1 - y) * squaredMargin)# returns the calculated loss of contrast for the return loss of the calling function

line 5defines ourloss of contrastFunction that accepts three arguments, two of which are required and the third is optional:

line 9ensures our real labels from thesame data typelike ourPrint. Failure to do this explicit conversion can result in TensorFlow errors when attempting to perform math operationsjePrint.

Then we calculate the contrast loss by:

The calculated contrastLossThe value is then returned to the calling function.

I suggest you check them out"What is contrast loss? And how can the loss of contrast be used to train Siamese networks?”section above and compare our implementation to the equation to better understand how contrast loss is implemented.

Creating our contrastive loss training script

We are now ready to implement our training roadmap! This script is responsible for:

1. Load the MNIST digit data set from disk
2. Preprocessing and creation of image pairs
3. Instantiation of the Siamese neural network architecture
4. Siamese network training with loss of contrast
5. Serialize the trained network and training history graph to disk

Most of this code is identical to our previous post tooSiamese Networks with Keras, TensorFlow and Deep Learning, so while I'll still cover our implementation in full, I'll defer a detailed discussion to a previous post (and point out the details, of course).

open thistrain_contrastive_siamese_network.pyFile in your project's directory structure and get to work:

# importiere die erforderlichen Pakete von pyimagesearch.siamese_network import build_siamese_modelfrom pyimagesearch importmetricsfrompyimagesearch import configfrom pyimagesearch import utilsfrom tensorflow.keras.models import Modelfrom tensorflow.keras.layers import Densefrom tensorflow.keras.layers import Inputfrom tensorflow.keras.layers import Lambdafrom tensorflow.keras . Datensätze importa mnistimport numpy como np

Lines 2-11Import our required Python packages. Notice how we import thosemetricssubmodule ofimage search, which includes oursloss of contrastImplementation.

From there we can load the MNIST dataset from disk:

# load the MNIST dataset and scale the pixel values ​​to the range of [0, 1]print("[INFO] loading the MNIST dataset...")(trainX, trainY), (testX, testY) = mnist. load_data()trainX = trainX / 255.0testX = testX / 255.0# add a channel dimension to the images trainX = np.expand_dims(trainX, axis=-1)testX = np.expand_dims(testX, axis=-1)# prepare positive and negative pairsprint( "[INFO] prepare positive and negative pairs...")(pairTrain, labelTrain) = utils.make_pairs(trainX, trainY)(pairTest, labelTest) = utils.make_pairs(testX, testY)

line 15loads the MNIST dataset with the pre-provided training and test splits.

We then pre-process the dataset by:

1. Scaling input pixel intensities in range images[0, 255]for[0, 1](Lines 16 and 17)
2. Add a channel dimension (Lines 20 and 21)
3. Constructing our image pairs (Lines 25 and 26)

Next, we can instantiate the Siamese network architecture:

# konfiguriere ein rede siamesaprint("[INFO] konstruiere ein rede siamesa...")imgA = Input(shape=config.IMG_SHAPE)imgB = Input(shape=config.IMG_SHAPE)featureExtractor = build_siamese_model(config.IMG_SHAPE)featsA = featureExtractor (imgA)featsB = featureExtractor(imgB)# finalmente, construa a rede siamesadistance = Lambda(utils.euclidean_distance)([featsA, featsB])model = Model(inputs=[imgA, imgB], output=distance)

Lines 30-34Create our sister networks:

• We start by creating two entries, one for each image in the image pair (Lines 30 and 31).
• Then we create the sister network architecture that will act as our resource extractor (line 32).
• Each image in the pair goes through our feature extractor, resulting in a vector that quantifies each image (Lines 33 and 34).

Using the 48-d vector generated by the sister networks, we proceed with the calculationEuclidean distancebetween our two vectors (Line 37) – this distance serves as our output from the Siamese network:

line 38define oModelspecifyimgAeimgB, our two images in the image pair as inputs, and ourdistancelayer as output.

Finally, we can train our Siamese network with loss of contrast:

# Compile modelprint("[INFO] Compile model...")model.compile(loss=metrics.contrastive_loss, optimizer="adam")# Train modelprint("[INFO] Train model...") History = model. fit([pairTrain[:, 0], pairTrain[:, 1]], labelTrain[:],validation_data=([pairTest[:, 0], pairTest[:, 1]], labelTest[:] ) , batch_size= config.BATCH_SIZE,epochs=config.EPOCHS)# Serialize model to diskprint("[INFO] Save Siamese model...")model.save(config.MODEL_PATH)# Plot training historyprint("[ INFO] Graph training history... ")utils.plot_training(history, config.PLOT_PATH)

Line 42compiles our model architecture with theloss of contrastFunction.

We then train the model with our training/validation image pairs (Lines 46-50) and then serialize the model to disk (Line 54) and track the training history (Line 58).

Training a Siamese network with loss of contrast

We are now ready to train our contrast-losing Siamese neural network using Keras and TensorFlow.

Be sure to use the"Transfers"Section of this manual to download source code, utilities and implementation of loss of contrast.

From there you can run the following command:

$ python train_contrastive_siamese_network.py[INFO] Loading MNIST dataset...[INFO] preparing positive and negative pairs...[INFO] building Siamese network...[INFO] compiling model...[INFO] training model . .. Epoch 1/1001875/1875 [=============================] - 81s 43ms/step - Loss: 0 .2038 - val_loss: 0.1755Epoch 2/1001875/1875 [=============================] - 80s 43ms/step - Loss: 0.1756 - val_loss : 0.1571Epoch 3/1001875/1875 [====================================] - 80s 43ms/step - loss: 0.1619 - val_loss: 0.1394Epoch 4/1001875/1875 [========================== == ] - 81s 43ms/step - loss: 0.1548 - val_loss: 0.1356Epoch 5/1001875/1875 [======================== ==== ==] - 81s 43ms/step - loss: 0.1501 - val_loss: 0.1262...Epoch 96/1001875/1875 [============= ===] - 81s 43ms/step - loss : 0.1264 - val_loss: 0.1066Epoch 97/1001875/1875 [=================== ===========] - 80s 43ms /step - loss : 0.1262 - val_loss: 0.1100epoch 98/1001875/1875 [======== ======================] - 82s 44ms/st ep - loss: 0.1262 - loss_val: 0.1078Epoch 99/1001875/1875 [=== === =================] - 81s 43ms /step - loss: 0.1268 - val_loss: 0.1067Epoch 100/1001875/ 1875 [== ========== ==================] - 80s 43ms/step - loss: 0.1261 - val_loss: 0.1107 [INFO] Save Siamese Model...[INFO] Track Training History...

Each epoch lasted about 80 seconds on my 3GHz Intel Xeon W processor. the training wouldeven fastermit GPU.

Our formative history can be seen inFigure 7.Notice how our validation loss really islowerthan our training loss,a phenomenon that I discuss in this tutorial.

If our validation loss is less than our training loss, it means we can “train harder” to improve the accuracy of our Siamese network, typically by relaxing regularization constraints, digging deeper into the model, and using a more aggressive learning rate.

But for now, our training model is more than enough.

Implementation of our contrast loss test script

The last script we need to implement istest_contrastive_siamese_network.py. This script is essentiallyidenticalto that covered in our previous tutorialCompare images for similarity with Siamese networks, Keras and TensorFlow,So while I'll cover the entire script today, I'll defer a detailed discussion to my earlier guide.

Let us begin:

# Import packages required for pyimagesearch import configfrom pyimagesearch import utilsfrom tensorflow.keras.models import load_modelfrom imutils.paths import list_imagesimport matplotlib.pyplot as pltimport numpy as npimport argparseimport cv2

Lines 2-9Import our required Python packages.

we will useCharging modelto load our serialized Siamese network from disk. Olist_imagesThe function is used to preserve image paths and facilitate the construction of pattern image pairs.

Let's get to our command line arguments:

# Build the argument parser and parse the arguments .parse_args())

The only command line argument we need is--Verboten, the path to our directory containing sample images that we want to create pairs from (e.g. theexamplesdirectory in our project directory).

Speaking of creating image pairs, let's do this now:

# Take the image paths from the test dataset and randomly generate # a total of 10 image pairs print("[INFO] loading test dataset...")testImagePaths = list(list_images(args["input" ]))np.random. seed( 42)pairs = np.random.choice(testImagePaths, size=(10, 2))# Load model from diskprint("[INFO] Loading Siamese model...") model = load_model(config.MODEL_PATH, compile = FALSE )

line 20takes the paths to all the images in our--VerbotenDirectory. We then randomly generate a total of 10 image pairs (line 22).

line 26carries our trained Siamese discus net.

With the Siamese network loaded from disk, we can now compare the images:

# loop over all image pairs for (i, (pathA, pathB)) in enumerate(pairs):# load both images and convert them to grayscale imageA = cv2.imread(pathA, 0)imageB = cv2.imread(pathB, 0) # Make a copy of both images for display origA = imageA.copy()origB = imageB.copy()# add a channel dimension to both images imageA = np.expand_dims(imageA, axis=-1 )imageB = np.expand_dims (imageB , axis=-1)# add a batch dimension to both imagesimageA = np.expand_dims(imageA, axis=0)imageB = np.expand_dims(imageB, axis=0)# scale the pixel values ​​for the range of [0 , 1] imageA = imageA / 255.0imageB = imageB / 255.0# Use our Siamese model to make predictions about the pair of images# indicating whether the images belong to the same class or not preds = model. predict([imageA, imageB] )proba = pred[0][0]

line 29goes around everythingparen. For each pair:

1. Load the two disk images (Lines 31 and 32)
2. Clone the images so we can display/draw them (Lines 35 and 36)
3. Add a channel dimension to both images, a requirement for the inference (Lines 39 and 40)
4. Add a batch dimension to the images, which again is a requirement for inference (Lines 43 and 44)
5. Scale range pixel intensities[0, 255]for[0, 1], just like we did during training

The pairs of images are then passed through our Siamese networkLines 52 and 53, resulting in the Euclidean distance calculated between the vectors generated by the sister networks.

Again, remember that thekleinerthe distance is thatmore likelythe two pictures are. On the other hand, thegreaterthe distance thatless similarthe pictures are.

The last block of code handles the display of the two images in the pair along with their calculated distance:

# initialize a figureafig = plt.figure("Pair #{}".format(i + 1), figsize=(4, 2))plt.suptitle("Distância: {:.2f}".format(proba)) # preferably a first class imagemax = fig.add_subplot(1, 2, 1)plt.imshow(origA, cmap=plt.cm.gray)plt.axis("off")# preferably a second imagemax = fig.add_subplot( 1, 2 , 2)plt.imshow(origB, cmap=plt.cm.gray)plt.axis("off")# more or plotplt.show()

Congratulations on implementing an inference script for Siamese networks! See my previous tutorial for more details on this implementation.Comparison of images for similarity with Siamese networks, Keras and TensorFlow.

Predict using our Siamese network with contrastive loss model

Let's put ourstest_contrastive_siamse_network.pyscript to work. Be sure to use the"Transfers"Section of this tutorial to download the source code, pre-trained model and example images.

From there you can run the following command:

$ python test_contrastive_siamese_network.py --input-examples[INFO] Loading test data set...[INFO] Loading Siamese model...

Look atFigure 8, you will see that we have sample sets of image pairs presented to our trained Siamese network with loss of contrast.

Images that are fromsame classTershorter distanceswhile pictures ofdifferent classesTerlarger classes.

So you can set a thresholdT, to act as a cut in the distance. If the calculated distance,D, It is<T, the image pair must belong to the same class. Otherwise ifD >= T, then the picturesandersKlassen.

set the limitTmust be done empirically through experimentation:

• Train the network.
• Calculates distances for image pairs.
• Manually display the pairs and their corresponding differences.
• Find a threshold that maximizes correct classifications and minimizes incorrect classifications.

In this case adjustT=0,16would be a reasonable limit since it allows us to properly label all pairs of images belonging to the same class, while all pairs of images are offandersClasses are treated as such.

Summary

In this tutorial, you learned about contrastive loss, including that it is a better loss function than binary cross-entropy for training Siamese networks.

What you need to keep in mind here is that a Siamese network is not specifically designed for thisClassification.Instead it is used todifferentiation, which means that it can not only detect whether an image pair belongs to the same class or not, but also whether the two images areidentical/similaror not.

Loss of contrast works much better in this situation.

I recommend you try out the binary cross-entropy and contrast loss when training your own Siamese neural networks, but I think you'll find that the general contrast loss works much better.

To download the source code for this post (and be notified when future tutorials are published here on PyImageSearch),Simply enter your email address in the form below!