Can you make Google Streetview-like images quickly and cheaply? That’s an R&D question I worked on while at Telenav after putting together the original OpenStreetView pitch.
Producing street view images isn’t trivial. Typically these are captured with dedicated hardware on dedicated vehicles driven around by paid employees.
I remember years ago talking to people building things like this. You couldn’t use standard DSLRs because the shutters were only rated for something like 100,000 exposures. This is fine for your typical prosumer but street view vehicles would be burning through a camera every week or something. Then, the car needs a bucket-load of data storage and you put a lot of miles on the vehicle very quickly. It gets expensive quick!
OpenStreetView’s solution, and Mapillary’s, is to put a phone on the dashboard pointing forward. This gets a lot of useful information but nowhere near the 360-degree view we’re used to in street view. But, the hardware is readily available (everyone has a phone) and the people using it are working for free. So it’s a great tradeoff, really.
How to get from there to 360-degree views?
At the time, dedicated spherical hardware cameras were expensive and hard to use. Think $500+ and you couldn’t talk to them. Most had a built-in SD card and could do a few preset recording modes without GPS (because, why would you need GPS?) For a half-decent camera the costs were more like $1k+
These prices were too high for even pro volunteers to spend. How could we drop the cost so that anybody could start taking 360-degree photos?
The obvious place to start is phones since they contain everything you need: cameras, compass, processing and a variety of radios. So, of course, I took an old iPhone and taped it to the roof of my truck, with a panoramic lens on it:
Old iPhone 4 devices can be found in bulk for ~$10 each which pulls the cost down. Taking photos as you drive or walk around resulted in images like this:
Notice that most of the image space is unused. If you unroll the donut you get a 360 strip, like this:
One of the few advantages of this approach is that “real” street view needs to blur things like faces and license plates. Since this strip is so low resolution, it comes pre-blurred!
You could in theory drive a car around like this and the phone could take photos and GPS points, unroll the image and upload it all in one. But… the images are pretty low resolution.
The answer is to use more than one phone:
We can use many phones in a mount. If they all take a photo at the same time then we can stitch them together and build a panorama. There turns out to be quite a lot of subtlety in the timing, capture, upload and stitching. The fundamental limit is the lens geometry of the camera. iPhones, like other devices, vary around 40ish degrees field of view. Since you need lots of overlap for a good panorama, you start to need something like 9 phones.
You can get to less phones by using wide angle lenses and changing the geometry a little:
Because of the CCD layout you get more pixels and a wider PoV in landscape.
The mounts were built with OpenSCAD. You write snippets of code (on the left) which outputs 3D shapes on the right. Here, we make some boxes and then subtract out another box to make a phone holder. Then we rotate and build many copies of them. To hold it together, there’s a thin cylinder (in blue) at the bottom. This will output a 3D file for printing.
Actually printing this in to a piece of plastic turns out to be surprisingly painful. Simplify3D helps a lot. The 3D model needs to be turned in to a set of commands for the printer to execute (move here, print a little bit of plastic, move over here…). Every printer is different. It takes a long time. We’re a long way from “just print this file” as we’re used to with printing on paper.
Measurements in the 3D model don’t quite come out in real life, either. The plastic oozes and has a set of material properties, so that it doesn’t print exactly what you send it but may be a few millimeters off. If you print walls that are too thin they will snap. You need to print a “raft” which is a layer of plastic on the print bed to print on top of, that you later snap off.
The cycle time is pretty long. Printing something can take 5-10 hours. Then you fix something, wait another 5-10 hours and so on.
The whole process is entertaining and educational, and reminds me yet again that manufacturing physical things is hard.
The resulting panoramas aren’t too bad, as you can see above. Each phone gets different lighting conditions and the photos are projected on the inside of a sphere. What you see above is just 5 phones, or about half a pano.
The software does some magic to try and sync timing. Initially I’d hoped that since the phones are (probably, hopefully) running ntpd they’d have pretty synchronous clocks. Wrong! Instead, a server (laptop) running a thin client is running the wifi network all the phones are connected to. Each phone runs an app which wakes up and connects. The server says something like “lets take a photo in 4 seconds” and the cameras all sync to this and take a photo at the same time.
They then connect again and upload their picture and a GPS point. This is nice as you get, say, 9 GPS readings per pano. Then they start again to take another set of photos.
The server software would then (and this is where it’s incomplete) take all these photos, build a pano and upload it somewhere. The panos I built were using autopano SIFT to find overlaps in the images but we could have taken compass readings too and used those alone or in conjunction to build the panoramas.
The finished image doesn’t look bad, as you can see. But it’s long and thin and has to crop the top and bottom off the images. The full pano would be much longer and thinner.
As the project progressed, two things happened.
- We started getting further from our goal (cheap, simple panos) not closer. Long thin pano image strips aren’t 360 views; you can’t look up and down. The cost and complexity kept going up with 3D printing, (old iOS version since it’s an iPhone 4) software to hang everything together, car mounts, charging 9 phones at once…
- Readily available commercial solutions came down in price and complexity. Moto and Essential phones now have cheap panorama attachments, for example. They tend to use two fisheye lenses back-to-back in a small consumer package.
So, while this was an interesting R&D experiment and a lot was learned it ultimately didn’t work out. You can find all the code for the server, iOS client and 3D files here.