Depth map

I could find a way to access google map deth data. Now we can download the panoramic street view images with their depth buffer data. Here is an example:

We explore the depth map of the following panorama:

https://www.google.ca/maps/@43.2596533,-79.8718058,3a,75y,96.6h,77.92t/data=!3m6!1e1!3m4!1sC-69GfttuBMYQwcBzFfZwQ!2e0!7i13312!8i6656!6m1!1e1

We can use web scraping algorithm to download the panoramic image and its depth map:

As you can see, the panoramic image is equirectangular. So we need to convert it to cube first:

front

Back

right

left

bottom

top

And we need to execute the same transformation on the depth map:

Now we need to register the depth information on the images and convert them to 3D point cloud:

library(EBImage)
library(rgl)
library(raster)

## Loading required package: sp

## 
## Attaching package: 'raster'

## The following objects are masked from 'package:EBImage':
## 
##     flip, rotate

img2points=function(img,flip=1){
  width=dim(img)[1]
  height=dim(img)[2]
  if(flip==1){
         w=data.frame(x=sort(rep(1:height,width)),y=rep(1:width,height))
  }else{
    w=data.frame(x=rep(1:height,width),y=sort(rep(1:width,height)))
  }
  w$red=as.vector(img[,,1])
  w$green=as.vector(img[,,2])
  w$blue=as.vector(img[,,3])
  w$color=rgb(w$red,w$green,w$blue)
  w
}


img=readImage("right.png")
depth=readImage("rightd.png")

img=img2points(img,flip = 1)
depth=img2points(depth,flip = 1)

img$xcorp=(img$x/512)*depth$red
img$ycorp=(img$y/512)*depth$red
img$z=depth$red
img=img[img$z!=1,]
head(img)

##     x   y       red     green      blue   color       xcorp     ycorp
## 382 1 382 0.7254902 0.7607843 0.7921569 #B9C2CA 0.001884191 0.7197610
## 383 1 383 0.7098039 0.7411765 0.7725490 #B5BDC5 0.001799939 0.6893765
## 384 1 384 0.7019608 0.7333333 0.7686275 #B3BBC4 0.001692708 0.6500000
## 385 1 385 0.7058824 0.7411765 0.7725490 #B4BDC5 0.001554841 0.5986137
## 386 1 386 0.7137255 0.7450980 0.7764706 #B6BEC6 0.001416973 0.5469516
## 387 1 387 0.7294118 0.7647059 0.7960784 #BAC3CB 0.001401654 0.5424403
##             z
## 382 0.9647059
## 383 0.9215686
## 384 0.8666667
## 385 0.7960784
## 386 0.7254902
## 387 0.7176471

As you can see, we have x,y and z coordinate and the color. We had to convert prespective projection to orthographic projection. The result is saved in xcorp and ycorp variable.

Now let’s visualize the points

Result