Building the Digitally Fused Thermal/Night Vision Gunsight
The FMWS was too science fiction for Tom Clancy
Litton (later acquired by Northrop Grumman) approached MTSI to help engineer the first truly digitally fused thermal/night vision camera. Soldiers who can see in the dark have an advantage over the adversary – combine night-vision with thermal sensors and a soldier can also spot an otherwise well-hidden enemy from their body heat and still see the trees and other cold objects. They can detect objects not only at night but also in the day, in rain, haze, dust or smoke.
Earlier video imaging systems faced challenges such as parallax problems, or differences in what the camera sees and the final picture at long range, short range, and indoors and outdoors. There were problems with detecting light from separate apertures. Reflective and refractive elements split up the light and direct it toward different sensors, so each sensor would detect a specific wavelength, but a better image required more sensors which added bulk and weight.
That is easier when mounted to fixed- and rotary-wing aircraft, but individual soldiers on foot needed lighter cameras.
Enter the U.S. Army’s Fused Multispectral Weapon Sight (FMWS).
A few years earlier, military technothriller author Tom Clancy was reportedly skeptical about combining thermal and night-vision in a single device as too advanced for the fictional Sam Fisher to wear in the Tom Clancy’s Splinter Cell series. In a few years, Guinness World Records would record the FMWS as the first video game concept to become real-world hardware.
MTSI’s solution was to develop a military-grade system to eliminate disadvantages – coaxially fusing images together with sub-pixel resolution, synchronizing their frames, and using image correction tables to apply correction factors to individual pixels in an image.
In the first version of the digitally fused thermal/night vision camera, MTSI developed an algorithm that allowed the system to shift from 100 percent night vision to 100 percent thermal, and then a blend of both. Later versions had more complex automatic blending, selecting elements of an image which has more detail in thermal, and other elements that have more detail in night vision, and then mixing the two for the best possible image.
"The story goes that Tom Clancy initially rejected the idea of Sam Fisher’s trifocal goggles on account of the technology being unlikely to ever see light of day. However, since it was such a cool concept, the best-selling author gave Ubisoft the green light. Wind forward the clock a couple years, and in 2004 Northrop Grumman Electronic Systems launched its so-called Fused Multispectral Weapon Sight (FMWS), boasting thermal- and night-vision. Hmmm…"
The project also included a Schmidt-Cassegrain assembly. This is commonly used in telescopes to bounce and focus light between two mirrors inside the telescope tube.
In such an assembly, light enters the front of the telescope and travels through a corrector plate – reducing distortions and allowing inexpensive spherical mirrors to be used – and then bounces off a spherical mirror at the back of the assembly, and then moves toward (and bounces again) off a second mirror at the front. Once light bounces off that second mirror, it moves through a hole in the center of the first mirror – effectively folding the light path.
“The sight yielded very impressive results, producing clear images on both image-intensifier and infrared channels with surprising flesh tones. It will provide the U.S. warfighter with a great advantage over the enemy in situational awareness, based on the best of both imaging worlds."
The result is a combination of powerful magnification in a compact design.
But the thermal camera lens is made from germanium, and opaque to the visible spectrum (although thermal energy goes right through it). MTSI’s design for the FMWS was a combination of that thermal lens surrounded by a ring compatible with the visible spectrum. Electronics provided image synchronization to fuse the images together – separately clipping, extending and scaling them as needed to a common output resolution for producing the desired output data stream.
MTSI also designed an adaptive lookup table to provide a fast, hardware-efficient method for applying correction factors to individual pixels and selected data records. These factors included gamma correction, brightness adjustment, contrast adjustment, pixel bit-width conversion, and image fusion ratio. An adaptive lookup table looks at the part of the thermal image with useful information, then expands and compresses it to fit within the display capability.
An essential element of this method is knowing how visible and thermal sensors interact with light. The smallest element in a digital sensor is a photodiode, while a pixel is the smallest piece of information in the image. The more light that touches a photodiode, the higher the voltage. There is also integration time – the time interval to capture light.
For example, if it is really bright outside, a sensor will require a shorter amount of time. In very bright or dark areas, the sensor’s integration time needs to match the amount of light coming in. However, thermal works differently.
“On the thermal, it doesn’t work that way,” MTSI Senior Design Engineer Jeff Short said. “Think of a resistor which changes its resistance based on the thermal image focused on it. When the sensor reaches a certain wavelength of thermal energy in the infrared part of the spectrum, it will change resistance. You can design that to be sensitive to a range of temperatures, such as from -100 degree C to several hundred degrees C, which gives you a huge range on a thermal sensor.”
Low-power operation was achieved even at a fast 100 frames per second frame rate. The modular design consisted of three boards (Imager, FPGA, and I/O) based around an Altera Cyclone IV FPGA.
Northrop Grumman delivered the FMWS to the U.S. Army in 2004. More than augmenting conventional night-vision capability, the infrared sensor detected “hot” targets such as recently driven vehicles or recently fired weapons. Thermal also cannot see through glass (which acts like a mirror for infrared radiation), but night-vision can – overcoming one obstacle previous thermal sensors had, say, in spotting an adversary through a window.
A later variation of the digitally fused thermal/night vision camera sight was revealed in the December 2007 edition of Special Weapons for Military & Police mounted on a .50-caliber Barrett M82A1/M107 CQ anti-material rifle. Learn more about MTSI’s work on video imaging systems.