import graphlab
We will use a popular benchmark dataset in computer vision called CIFAR-10.
(We've reduced the data to just 4 categories = {'cat','bird','automobile','dog'}.)
This dataset is already split into a training set and test set. In this simple retrieval example, there is no notion of "testing", so we will only use the training data.
image_train = graphlab.SFrame('image_train_data/')
The two lines below allow us to compute deep features. This computation takes a little while, so we have already computed them and saved the results as a column in the data you loaded.
(Note that if you would like to compute such deep features and have a GPU on your machine, you should use the GPU enabled GraphLab Create, which will be significantly faster for this task.)
#deep_learning_model = graphlab.load_model('http://s3.amazonaws.com/GraphLab-Datasets/deeplearning/imagenet_model_iter45')
#image_train['deep_features'] = deep_learning_model.extract_features(image_train)
image_train.head()
We will now build a simple image retrieval system that finds the nearest neighbors for any image.
Let's find similar images to this car picture.
knn_model = graphlab.nearest_neighbors.create(image_train,features=['deep_features'],
label='id')
graphlab.canvas.set_target('ipynb')
We are going to create a simple function to view the nearest neighbors to save typing:
def get_images_from_ids(query_result):
return image_train.filter_by(query_result['reference_label'],'id')
Very cool results showing similar cats.
car = image_train[8:9]
car['image'].show()
get_images_from_ids(knn_model.query(car))['image'].show()
show_neighbors = lambda i: get_images_from_ids(knn_model.query(image_train[i:i+1]))['image'].show()
show_neighbors(83)
show_neighbors(24)
knn_model.query(cat)
Split the SFrame with the training data into 4 different SFrames. Each of these will contain data for 1 of the 4 categories above. Hint: if you use a logical filter to select the rows where the ‘label’ column equals ‘dog’, you can create an SFrame with only the data for images labeled ‘dog’.
image_train['label'].sketch_summary()
Accuracy of predicting dog in the test data: Using the work you did in this question, what is the accuracy of the 1-nearest neighbor classifier at classifying ‘dog’ images from the test set?
Similarly to the image retrieval notebook you downloaded, you are going to create a nearest neighbor model using the 'deep_features' as the features, but this time create one such model for each category, using the corresponding subset of the training_data. You can call the model with the ‘dog’ data the dog_model, the one with the ‘cat’ data the cat_model, as so on.
dog_train=image_train[image_train['label']=='dog']
cat_train=image_train[image_train['label']=='cat']
bird_train=image_train[image_train['label']=='bird']
automobile_train=image_train[image_train['label']=='automobile']
dog_knn_model = graphlab.nearest_neighbors.create(dog_train,features=['deep_features'],
label='id')
cat_knn_model = graphlab.nearest_neighbors.create(cat_train,features=['deep_features'],
label='id')
bird_knn_model = graphlab.nearest_neighbors.create(bird_train,features=['deep_features'],
label='id')
automobile_knn_model = graphlab.nearest_neighbors.create(automobile_train,features=['deep_features'],
label='id')
You now have a nearest neighbors model that can find the nearest ‘dog’ to any image you give it, the dog_model; one that can find the nearest ‘cat’, the cat_model; and so on.
Using these models, answer the following questions. The cat image below is the first in the test data:
You can access this image, similarly to what we did in the iPython notebooks above, with this command:
image_test[0:1]
What is the nearest ‘cat’ labeled image in the training data to the cat image above (the first image in the test data)?
image_test = graphlab.SFrame('image_test_data/')
mycat=image_test[0:1]
image_test[0:1]['image'].show()
mycat_neighbors = get_images_from_ids(cat_knn_model.query(mycat))
mycat_neighbors['image'].show()
mycat_neighbors['id']
mycat_neighbors
cat_knn_model.query(mycat)
What is the nearest ‘dog’ labeled image in the training data to the cat image above (the first image in the test data)?
dog_knn_model.query(mycat)
For the first image in the test data (image_test[0:1]), which we used above, compute the mean distance between this image at its 5 nearest neighbors that were labeled ‘cat’ in the training data (similarly to what you did in the previous question). Save this result.
Similarly, for the first image in the test data (image_test[0:1]), which we used above, compute the mean distance between this image at its 5 nearest neighbors that were labeled ‘dog’ in the training data (similarly to what you did in the previous question). Save this result.
On average, is the first image in the test data closer to its 5 nearest neighbors in the ‘cat’ data or in the ‘dog’ data? (In a later course, we will see that this is an example of what is called a k-nearest neighbors classifier, where we use the label of neighboring points to predict the label of a test point.)
cat_knn_model.query(mycat)['distance'].mean()
get_images_from_ids(dog_knn_model.query(mycat))['image'].show()
cat_knn_model.query(mycat)['distance']
dog_knn_model.query(mycat)['distance'].mean()
Spliting test data by label: Above, you split the train data SFrame into one SFrame for images labeled ‘dog’, another for those labeled ‘cat’, etc. Now, do the same for the test data. You can call the resulting SFrames
image_test_cat, image_test_dog, image_test_bird, image_test_automobile
image_test_dog=image_test[image_test['label']=='dog']
image_test_cat=image_test[image_test['label']=='cat']
image_test_bird=image_test[image_test['label']=='bird']
image_test_automobile=image_test[image_test['label']=='automobile']
Finding nearest neighbors in the training set for each part of the test set: Thus far, we have queried, e.g.,
dog_model.query()
our nearest neighbors models with a single image as the input, but you can actually query with a whole set of data, and it will find the nearest neighbors for each data point. Note that the input index will be stored in the ‘query_label’ column of the output SFrame.
dog_query=dog_knn_model.query(image_test_dog)
dog_query
(We've reduced the data to just 4 categories = {'cat','bird','automobile','dog'}.)
dog_query[dog_query['query_label']==4]
cat_query=cat_knn_model.query(image_test_cat)
cat_query[cat_query['query_label']==1]
bird_query=bird_knn_model.query(image_test_bird)
bird_query[bird_query['query_label']==2]
automobile_query=automobile_knn_model.query(image_test_automobile)
automobile_query[automobile_query['query_label']==2]
Using this knowledge find the closest neighbor in to the dog test data using each of the trained models, e.g.,
dog_cat_neighbors = cat_model.query(image_test_dog, k=1)
finds 1 neighbor (that’s what k=1 does) to the dog test images (image_test_dog) in the cat portion of the training data (used to train the cat_model).
Now, do this for every combination of the labels in the training and test data.
dog_cat_neighbors = cat_knn_model.query(image_test_dog, k=1)
dog_cat_neighbors[0]
dog_bird_neighbors = bird_knn_model.query(image_test_dog, k=1)
dog_bird_neighbors[0]
Create an SFrame with the distances from ‘dog’ test examples to the respective nearest neighbors in each class in the training data: The ‘distance’ column in dog_cat_neighbors above contains the distance between each ‘dog’ image in the test set and its nearest ‘cat’ image in the training set. The question we want to answer is how many of the test set ‘dog’ images are closer to a ‘dog’ in the training set than to a ‘cat’, ‘automobile’ or ‘bird’. So, next we will create an SFrame containing just these distances per data point. The goal is to create an SFrame called dog_distances with 4 columns:
i. dog_distances[‘dog-dog’] ---- storing dog_dog_neighbors[‘distance’]
ii. dog_distances[‘dog-cat’] ---- storing dog_cat_neighbors[‘distance’]
iii. dog_distances[‘dog-automobile’] ---- storing dog_automobile_neighbors[‘distance’]
iv. dog_distances[‘dog-bird’] ---- storing dog_bird_neighbors[‘distance’]
dog_cat_neighbors = cat_knn_model.query(image_test_dog, k=5)
dog_bird_neighbors = bird_knn_model.query(image_test_dog, k=5)
dog_dog_neighbors = dog_knn_model.query(image_test_dog, k=5)
dog_automobile_neighbors = automobile_knn_model.query(image_test_dog, k=5)
dog_distances = graphlab.SFrame({'dog-dog': dog_dog_neighbors['distance'],
'dog-automobile': dog_automobile_neighbors['distance'],
'dog-bird': dog_bird_neighbors['distance'],
'dog-cat': dog_cat_neighbors['distance']})
dog_distances
def is_dog_correct(row):
if row['dog-dog']<row['dog-cat'] and row['dog-dog']<row['dog-bird'] and row['dog-dog']<row['dog-automobile']:
return (1)
else:
return (0)
dog_distances.apply(is_dog_correct).sum()
Accuracy of predicting dog in the test data: Using the work you did in this question, what is the accuracy of the 1-nearest neighbor classifier at classifying ‘dog’ images from the test set?
3554./5000