One Thousand Nights of Northern Lights
How an app makes aurora timelapse videos more accessible and immersive
The Northern Lights, or “Aurora Borealis,” are something you never forget.
In a few moments, the whole night sky can unexpectedly fill with impossible light and motion — green, violet, magenta, and red. Adventurers take thousands of photos of the aurora every year, as do researchers trying to understand the connection between the aurora and space weather. Researchers collect their images using automated cameras, compiling archives of nightly time-lapse videos that can span years of aurora activity.
To make this resource more accessible, I set out to create a web app that lets anyone explore years of this amazing time-lapse aurora imagery by using data visualization practices. In this piece I will describe a few of the questions that arose during my project.
First, what is the shape of the data?
Second, how can the data be condensed from hundreds of hours of video into an interface that is quickly scannable and comprehensible?
Third, how do we make the data compelling and engaging, as if we were really there under the aurora?
Fourth, what lessons did I learn while putting together the web app and showing it to subject matter experts?
You can Try the app here, then read on to see some of the thinking that went into its development.
Natural Data Visualization
While you may not have thought of them this way, the aurora are a natural data visualization. They are a visual footprint of the vastly complex electromagnetic dance of space weather that stretches across the solar system. Very bright and active aurora are powered by space weather phenomena such as coronal mass ejections and coronal holes on the Sun. What we see as auroral light is a visualization of the Sun’s electromagnetic connections to the rest of the solar system — and this connection can get stormy.
Understanding space weather is as important as understanding hurricanes or earthquakes — it impacts communications, navigation, power grids and our infrastructure. Astronauts have storm shelters on the International Space Station to ride out dangerous solar storms. There is still a tremendous amount to learn about how space weather propagates and affects us, and monitoring the aurora has been a window into this mystery. You can learn more about space weather from NASA’s public overview.
Yellowknife — A Front Row Seat to the Aurora Show
The aurora are most often visible in far northern and southern latitudes. Very few cities are located in these regions where the aurora can regularly be seen directly overhead. One of the most well known is the city of Yellowknife in northern Canada, 2000km north of the border with the United States and a destination for thousands of adventurers who want to witness the magic of the Aurora Borealis.
But many people never make it to a place like Yellowknife to see the skies come alive, and as a data visualization practitioner and citizen scientist, I wanted to make the aurora experience more accessible and explorable for everyone.
How We Gather Space Weather Imagery
Since the 1957–1958 International Geophysical Year, researchers have used a special type of camera to get the big picture of aurora activity and so that it can be matched to other sources of data from satellites and ground instruments. This tool is called an “All sky camera”, and it allows researchers to record aurora activity from horizon to horizon, every night, every few seconds, for years on end.
An all sky camera has the unique feature of a 180 degree fisheye lens, producing a spherical view of the sky overhead. The thousands of photos taken every night are often combined into time-lapse movies for quicker reference, but even so, there is about one hundred hours of time-lapse movies that would need to be sorted through to find a particular auroral event.
One all-sky camera in particular is used for public outreach: the AuroraMAX camera. It is located in Yellowknife and operated by the University of Calgary’s Auroral imaging Group. They host an archive of time-lapse video from their camera going back 10 years. Unlike many research cameras which have special wavelength filters, it is a conventional digital camera, capturing photos as a person might see them: in full colour at high resolution. This makes it suitable for as a visual source for an interactive data visualization project.
The Project: Aurora in the Yellowknife Night Sky
My goal was to build an interface that would allow anyone to browse the thousands of nights of aurora video at a glance, and then be able to ‘dive in’ to the data and see the aurora the way someone visiting Yellowknife might see them — a first-person view, where they can look up overhead and watch the lights evolve and move. In other words, I wanted to take hundreds of hours of video and provide a big-picture view and a very personal experience for the viewer using the same data set.
The first task was to make over a thousand nights of video easy to triage at a glance and at a reasonable bandwidth for mobile devices. I wanted a user to be able to find the best nights throughout the year and best moments during those nights for aurora activity, which ebbs and flows on the scale of minutes. I also wanted to allow the user to see if any given night was cloudy, had too much moonlight, or if the camera dome was obscured by snow. I needed to remap the time element into a static image, much like a seismograph encodes earthquakes and geological activity over time.
What is a Keogram?
Thankfully, there is an existing visualization called a ‘Keogram” that transforms all-sky image sequences into a long, rectangular summary image, where the time axis runs left-to-right. A keogram samples a time-lapse sequence by taking a 1-pixel-wide slice out each frame, stacking these slices from left-to-right in a long image. The effect is like a bookshelf, where you see the spines of each book as a compact representation of the book’s content.
Generating Keograms with OpenCV and Python
To generate keograms, I turned to the OpenCV computer vision library. OpenCV has been around for a long time, originally as a C++ library and now as a very capable Python library that works well with existing Python data structures. In OpenCV, images are arrays and can be sliced and sampled easily. OpenCV has support for loading many video formats and saving to a wide range of image types.
This made it suitable to write a command-line tool that would load the aurora time-lapse, sample the centre vertical column of each frame, and stack these into a keogram image to be saved as an image file. Some of the resulting keograms were 480px by 3000px wide, so more compact .jpg versions were generated for use in the app that weighed in at 10–30KB per night. 3000 HD video frames could be processed in about 30 seconds on my 2015 Macbook Pro.
Machine Vision for Metadata
There is additional metadata that needs to be gathered — namely start time and end time. Due to constantly changing sunset and sunrise times, each time-lapse video starts and ends at a different time, and thus each keogram has a varying pixel width as a result of the video frame count. This information is needed if we want to scale and align the keograms relative to one another in an interface.
Fortunately the AuroraMAX time-lapse video includes a time stamp on each frame. The timestamp image on the first and last frame could be parsed by the PyTesseract text recognition library in Python as part of the keogram generation pipeline, and this data stored is a JSON file with an object for each night’s meta data.
“Walk-away” Pipeline
Like many data-cleaning processes, each stage in this pipeline has to be made robust to network errors, video corruption, file corruption, network timeouts, and failures of the machine vision step to grab a readable timestamp. The goal was to avoid errors that would halt an hours-long, unsupervised process. This involves wrapping essential function calls in proper error catching code, and skipping corrupted frames and retrying, rather than just giving up on a whole night’s time-lapse video file.
The lesson here: in automated data processing, test and re-test with real data; expect and plan for the unexpected.
Interface Design
One of the most critical aspects of effective data visualization is on-boarding. The introduction of the interface includes several examples of how to read keograms and spot patterns like clouds, moonlight, and aurora substorms. using more images and less inline text prevents reading fatigue.
The interface has to display over 1300 keograms, or about 40,000 pixels of scrolling. These were subdivided by year using tabs and arranged chronologically as stacks of about 220 images for a year from January through December.
A keogram stack like this can be thought of as a case of the ‘small multiples’ visualization pattern, since there is a grid of smaller visualizations we want to compare with a common horizontal time axis. Rather than repeat the time axis ticks and labels for each row, I chose to use a ‘sticky-header’ pattern so that the timescale has minimum interference with the keograms, and used a loading="lazy" attribute in the image html to prevent load of images not yet in the viewport.
Linear Heatmap for Spotting Storm Correlation
Auroral activity is associated with increased geomagnetic activity, as measured by magnetometers across the globe. Forecasts of this activity include a scale called “Kp index” — analogous to the Richter scale for seismic activity. I wanted to align this data with the keogram images using a heat-map visualization strip that runs along the bottom of each keogram. This way, the visual aurora can be correlated with the magnetic activity in a more intuitive way. The Kp data was added to the keogram meta-data JSON file.
The heatmap color scale was chosen to include three distinct hue ranges, each with 3–4 levels of brightness and saturation. This reflected the common categorization of geomagnetic activity into 3 broad categories, each with strength variations.
Implementing the Web-App
The Keogram Index app was build with HTML, CSS, and ES2015 standard JavaScript managed by webpack. Since this primarily deals with stacking and aligning rectangular images, JS and CSS were not too unwieldy without a library like D3 to manage the data. A key to performance was to fully realize layout calculations in CSS, so that browser resize and reflow would be handled by the browser’s layout engine and not a JavaScript resize listener. When a specific year is selected from the filter bar, the app uses a pattern from game development, where a flag is set in JavaScript that the keogram view is invalidated, prompting a removal of that DOM node and a full re-render of it and its child nodes. This way, the initial render of the interface and subsequent re-renders can use the same code.
Although efficient for exploration, this still is an abstract way of looking at the aurora, since the dimensional reduction also takes away from the experience of really seeing them first hand. Fortunately the same fish-eye camera data could be transformed to create a first-person immersive viewpoint.
Making an Immersive Video Experience
When a user selects a keogram, I wanted to be able to show them what it was like standing in Yellowknife under that night’s aurora. This meant looking at how all-sky camera video can be made to fill a virtual sky so a user could look up and around as the aurora moved and changed.
All-sky time-lapse video looks like a highly distorted perspective on the sky, but it can be re-projected to create a fully immersive point of view. Much like a planetarium uses a dome-shaped screen overhead to immerse viewers, a virtual dome can be used to re-project the circular all-sky video. A virtual camera positioned in the middle of this dome would have the same point of view as a person standing under the aurora-filled sky.
This requires 3D geometry and texture-mapping of the video onto a hemispherical dome geometry, a perfect job for the ThreeJS JavaScript library. This library is a powerful and highly usable interface to the high performance WebGL 3D capability built into modern browsers.
Texture-mapping involves assigning each pixel of the time-lapse video to a pixel on the virtual dome. Since the video is in HD format, which is a 16:9 ratio, it must be distorted to a square to ‘drape’ over the dome. The dome’s exact shape must also be slightly distorted to ‘undo’ any fish-eye lens artifacts, especially near the horizon. In ThreeJS, texture mapping the dome means assigning each vertex of the dome’ mesh geometry a number between 0 and 1 across the width and length of the video, using the projection equation that describes the original glass camera lens. This ‘pins’ each part of the video to a specific spot on the dome, un-distorting the fish-eye view to a first-person view.
The result is fairly convincing, especially looking up and around. ThreeJS is performant enough that this immersive effect works on a mobile device, too. The video is streamed, since it is over 30MB per file; fortunately devices are optimized for video decompression so performance on an iPhone 6S was between 30 and 60 frames per second.
I wanted the user to be able to have access to any part of the night’s video, so added the keogram as a ‘scrub’ bar along the bottom of the interface, with a subtle position indicator. This created a familiar connection between the keogram index interface and the first-person view. A compass rose is also placed like a hula-hoop just below the virtual viewer’s waistline so there is an unobtrusive way to see what direction they are looking. A line of trees and snow, and a tent glowing warmly make the experience more immersive, like the viewer was out camping for the night. In order to assist in sharing a given night and moment, all time parameters are stored as URL parameters.
Right now the interface is being tested with Oculus hardware using the WebXR standard, making it a VR experience.
Lessons Learned
1. You can get too close to the content
As the explorer of the dataset you are likely already familiar with the terminology and shorthand of the domain. Test frequently with audiences since what you assume is obvious, important, or impactful may not be. What you assume is trivial or not worth the polish, can be a big factor in positive audience engagement.
2. Don’t be afraid to put a new perspective on data
Although the first-person perspective may not be a way of distilling patterns in the data or revealing insights directly, it does put the user right in the middle of the data. The point of exploration is insight, but also a more personal understanding of the dataset, and sometimes that comes from experiencing the data as it replays above you.
3. Ask the experts
I shared the progress of this app on Aurora chaser groups, which got the attention of the aurora research community. From them I received valuable feedback on terminology — such as “we use UTC timezone” — and questions that led me to discover new aspects of the subject I may never have found on my own.
4. Assess engagement for drop-offs
It is not difficult to add some anonymized engagement tracking to a web app. Although I could track hits on the keogram index and the aurora dome viewer, I still wondered where the drop-offs would be in engagement. Is loading taking too long? Are the instructions taking up too much space? Is it clear that clicking on a keogram goes to a first-person view? Does the user realize they can look around in 3D? By tracking each of these actions I could have gained more insight into effective UI design for a rather complex subject.
5. Beware of caching on web apps
One of the reasons I used webpack is the ability to automatically append often-changed file names with a hash text, for example ‘app.js’ becomes ‘app76571263...js’. This prevents browser caching of older versions of your code, styles, or assets on user devices, which can be a significant but invisible problem for engagement.
6. Delight and progression matter
Exploration of a large dataset needs guideposts, just like a tourist destination. On-boarding to a new topic and a new type of visualization can be engaging and fun — or it can be a barrier that stops users cold. I learned that writing short, concise, action oriented copy made a difference, and that providing hints and overlays for an interface were necessary and effective. Learning has to be bite-sized until the user feels they have the tools they need to explore on their own.
More to Explore
Introduction to ThreeJS at threejsfundamentals.org
The AuroraMAX website and the University of Calgary Aurora Imaging Group
Aurorasaurus citizen science project
About the Author
Jeremy Kuzub is an interactive data visualization practitioner in Canada and developer at the Capture North Project.
