Image Based Geolocalization and Sensor Network

This past week, there has been much issue with the fact the iPhone and Android phone stored the location of these phones in the clear. Another possibility of geolocation from one of these smartphones or any webcam come from their ability to locate where a photo was taken by correlating it with satellite imagery or correlating it with others. Two projects along those lines come to my mind: Webcam Geolocalization and IM2GPS. The first technique uses fixed webcams, but I think eventually it could be extended to mobile cameras. The second technique uses Flickr. These results can only be created because there are many cameras and webcams, i.e. the sensor network needs to be dense in some fashion. Let us note that 1 billion cameras will be sold in phones this year. One can definitely think of this sensor network made out of these cameras as a way to gather meteorological information and more...

From the respective project's websites:
1. Webcam Geolocalization

It is possible to geolocate an outdoor camera using natural scene variations, even when no recognizable features are visible. (left) Example images from three of the cameras from the AMOS dataset. (right) Correlation maps with satellite imagery; a measure of the temporal similarity of the camera variations to satellite pixel variations is color coded in red. The cross shows the maximum correlation point, the star shows the known GPS coordinate.

Overview

A key problem in widely distributed camera networks is geolocating the cameras. In this work we consider three scenarios for camera localization: localizing a camera in an unknown environment using the diurnal cycle, localizing a camera using weather variations by finding correlations with satellite imagery, and adding a new camera in a region with many other cameras.  We find that simple summary statistics (the time course of principal component coefficients) are sufficient to geolocate cameras without determining correspondences between cameras or explicitly reasoning about weather in the scene.

Let us note their use of AMOS: Archive of Many Outdoor Scenes
The relevant paper is: Participatory Integration of Live Webcams into GIS by Austin Abrams, Nick Fridrich, Nathan Jacobs, and Robert Pless. The abstract reads:

Global satellite imagery provides nearly ubiquitous views of the Earth’s surface, and the tens of thousands of webcams provide live views from near Earth viewpoints. Combining these into a single application creates live views in the global context, where cars move through intersections, trees sway in the wind, and students walk across campus in realtime. This integration of the camera requires registration, which takes time, effort, and expertise. Here we report on two participatory interfaces that simplify this registration by providing applications which allow anyone to use live webcam streams to create virtual overhead views or to map live texture onto 3D models. We highlight system design issues that affect the scalability of such a service, and offer a case study of how we overcame these in building a system which is publicly available and integrated with Google Maps and the Google Earth Plug-in. Imagery registered to features in GIS applications can be considered as richly geotagged, and we discuss opportunities for this rich geotagging.

2. IM2GPS

IM2GPS: estimating geographic information from a single image by James Hays, Alexei Efros. The abstract reads:

Estimating geographic information from an image is an excellent, difficult high-level computer vision problem whose time has come. The emergence of vast amounts of geographically-calibrated image data is a great reason for computer vision to start looking globally — on the scale of the entire planet! In this paper, we propose a simple algorithm for estimating a distribution over geographic locations from a single image using a purely data-driven scene matching approach. For this task, we will leverage a dataset of over 6 million GPS-tagged images from the Internet. We represent the estimated image location as a probability distribution over the Earth's surface. We quantitatively evaluate our approach in several geolocation tasks and demonstrate encouraging performance (up to 30 times better than chance). We show that geolocation estimates can provide the basis for numerous other image understanding tasks such as population density estimation, land cover estimation or urban/rural classification.

Other relevant sites:
What Do the Sun and the Sky Tell Us About the Camera?

The Quake Catcher Network

The Quake Catcher Network is a sensor network made up of people's computers. The data comes from the internal accelerometers of these computers. The map of this crowdsourced sensors is here. You can join the network here.

From the introduction section:

The Quake-Catcher Network: Cyberinfrastructure Bringing Seismology to Homes and Schools

 

Earthquake safety is a responsibility shared by billions worldwide. The Quake-Catcher Network (QCN) provides software so that individuals can join together to improve earthquake monitoring, earthquake awareness, and the science of earthquakes. The Quake-Catcher Network (QCN) links existing networked laptops and desktops in hopes to form the world’s largest and densest earthquake monitoring system. Costs for this network are minimal because the QCN uses 1) strong motion sensors (accelerometers) already internal to many laptops and 2) low-cost universal serial bus (USB) accelerometers for use with desktops. The QCN is built around the Berkeley Open Infrastructure for Network Computing (BOINC!), which provides free software to link volunteer computers together for a single scientific goal. 

Learn more about the Desktop Network.

Learn more about the Laptop Network.

The QCN provides a natural way to engage students and the public in earthquake detection and research. This project places USB-connectable sensors in K-12 classrooms as an educational tool for teaching science and a scientific tool for studying and monitoring earthquakes. Through a variety of interactive experiments students can learn about earthquakes and the hazards that earthquakes pose. For example, students can learn how the vibrations of an earthquake decrease with distance by jumping up and down at increasing distances from the classroom sensor and plotting the decreased amplitude of the seismic signal displayed on their computer.

Here is a layman's description of the system.

The type of sensors being used are low cost three way accelerometers. If you take a look at the usage, you'll find that a quake better not hit on saturday or sunday.

A Continent Wide Frequency Monitoring Network

[Metafilter readers: Wondering Star is focused on planet sized sensor networks, if you have other examples please let me know. Thanks!]

From Min Kao's site here is The Frequency Monitoring network (FNET)

FNET is a low-cost, GPS-synchronized wide-area power system frequency measurement network. Highly accurate Frequency Disturbance Recorders (FDRs) developed at Virginia Tech are used to measure the frequency, phase angle, and voltage of the power signal found at ordinary 120-V electrical outlets. The measurement data are continuously transmitted over the Internet to the FNET server housed at the University of Tennessee.

Several applications have been developed which utilize the FNET data. These include:

    * Event detection and location
    * Oscillation detection
    * Animations of frequency and angle pertubations
    * Integration of renewables into the power grid
    * Detection of system breakup or islanding
    * Prediction of grid instability and reduction of blackouts
    * Providing grid control input signals

Currently, FNET collects data from approximately 80 FDRs located across the continent and around the world.  Additional FDRs are constantly being installed so as to provide better observation of the power system.

wikipedia entry on FNET.





One can also watch live the gradient map here:

GPS Constellation as a way to image the Ionosphere

While we all know about the GPS constellation of 32 satellites orbiting the Earth in MEO that provide coordinates to our smartphones (have we become too reliant on it ?), and is the mainstay of many of sensor networks on earth by providing them location and a universal time reference. One other use of this constellation is to image the ionosphere, i.e. perform an inverse problem from the measurements of the delay of message transmission between the satellites and some known antenna on the ground (here is a layman's description of the issue) to evaluate the concentration of electron in the ionosphere.

Here is an animated gif version of the solving of this inverse problem. The sensor network is made of 32 satellites plus any number of antennas on the ground.and its size scales directly with the size of the earth.

Using Unsuspecting Amateur Comet Chasers and Interplanetary Probes as a Sensor Network

With this arxiv blog entry entitled "Astronomers Calculate Comet's Orbit Using Amateur Images" one realizes that astrophotographs posted online by amateurs all around the world represent a massive sensor network.

The sensor network described here is embodied by you, me, and whoever can take nightsky images and upload their photographs on the web.

How does this work? In a simplified  instance,  for each photo where the comet is displayed, there are also stars surrounding it. These stars can be estimated using a similar algorithm as the one used on satellite cameras called star-trackers. In effect, given a good timeline, the algorithm estimates the attitude of the image i.e. the pointing direction of your camera. Using this information and performing the same computation for different images, one can infer the trajectory of the comet. I mentioned a similar idea a year ago. However, the algorithm of the paper (see below) goes further than just perform star identification in that it picks the right star image out of a series of photos downloaded from a search on a search engine (i.e. they are not necessarily images of stars). 

The crowdsourcing project used to calibrate the images in this project is the astrometry.net system, From the presentation of the project:

If you have astronomical imaging of the sky with celestial coordinates you do not know—or do not trust—then Astrometry.net is for you. Input an image and we'll give you back astrometric calibration meta-data, plus lists of known objects falling inside the field of view.

We have built this astrometric calibration service to create correct, standards-compliant astrometric meta-data for every useful astronomical image ever taken, past and future, in any state of archival disarray. We hope this will help organize, annotate and make searchable all the world's astronomical information.

So far,the sensor network really is the size of a planet but we can push this capability further. Instead of just using pictures taken from the ground, one can include pictures taken from space probes. For that, I asked the maintainer of astrometry.net to be an alpha tester while choosing this raw footage taken by the Cassini probe that is currently orbiting Saturn (about 1 billion kilometer from earth). N00171086.jpg was taken on March 31, 2011 and received on Earth March 31, 2011. According to the Cassini site "The camera was pointing toward SKATHI, and the image was taken using the CL1 and CL2 filters. This image has not been validated or calibrated."

The system was able to fit a pattern between their star catalog and the stars of the image:

The code puts your image in perspective as well as provide an evaluation the field of view (here it is evaluated at (RA, Dec) = (121.200, 30.085) degrees and spans 21.07 x 21.07 arcminutes .)

it also provides the photo  framed within a google maps GUI.

This is a very nice set-up and one can imagine how photos dating back from Mariner and Pioneer could be used to extract historical information of currently unknown comets.

Here is the paper: Searching for comets on the World Wide Web: The orbit of 17P/Holmes from the behavior of photographers by Dustin Lang, David W. Hogg. The abstract reads:

We performed an image search on Yahoo for "Comet Holmes" on 2010 April 1. Thousands of images were returned. We astrometrically calibrated---and therefore vetted---the images using the Astrometry.net system. The calibrated image pointings form a set of data points to which we can fit a test-particle orbit in the Solar System, marginalizing out image dates and catching outliers. The approach is Bayesian and the model is, in essence, a model of how comet astrophotographers point their instruments. We find very strong probabilistic constraints on the orbit, although slightly off the JPL ephemeris, probably because of limitations of the astronomer model. Hyper-parameters of the model constrain the reliability of date meta-data and where in the image astrophotographers place the comet; we find that ~70 percent of the meta-data are correct and that the comet typically appears in the central ~1/e of the image footprint. This project demonstrates that discoveries are possible with data of extreme heterogeneity and unknown provenance; or that the Web is possibly an enormous repository of astronomical information; or that if an object has been given a name and photographed thousands of times by observers who post their images on the Web, we can (re-)discover it and infer its dynamical properties!

Other projects mentioned in the paper include:

• GalaxyZoo
• the monitoring projects of the AAVSO (http://aavso.org),

Welcome to Wondering Star, the blog where sensor networks can be the size of a planet.

A wandering star is a poetic term for a planet, a wondering star is a planet that may have the ability to think through sensing. Over the past few years, we have seen the emergence of very large sensor networks that can sometimes scale to the size of a planet. I'll try to feature examples of these as time go by but I am also interested in how these sensor networks will allow us to do new things by merging data from each other. Of belated interest, I will include the measurement techniques and theoretical advances that allow us to make better sense of the gigantic amount of data produced by these sensors. From a societal standpoint, these networks can also pose a series of new issues that were never important enough in the past. I hope you enjoy Wondering Star.