In an earlier post I mentioned we were in the middle of a big investigation into the degradation of one of our instruments. My active role in the investigation finished up Friday, so I’m breathing a big sigh of relief. Meanwhile I started processing all the photographs I made in the course of the investigation to document the work done. I ran a couple of time lapse cameras, so the photos number in the thousands. I only used a couple of the time lapse photos, though, so the number of “keepers” was a lot smaller.
I picked a subset of the keepers to narrarate the story thus far:
A couple of months back one of our astronomers analyzed the images coming off of our wide field imager to measure the zero point of each filter. A zero point is the measurement of the transmission of the instrument is in a given band. Because of dirt and grime building up on our primary mirror, we expect some drop in the zero point over time. What we didn’t expect was what he saw: a 20% drop in performance in g’ band with slightly less extreme drops in all the other bands. What made this worse was that we had just re-coated the mirror. We expected to get a marginal gain, not a dramatic loss. Clearly something was wrong. When the instrument came off the telescope we went up to take a look.
I’ve done plenty of sunset photography at the beach. Most days I get away with some good frames and a camera that needs a damp wipe. Other times I get stuck in an onshore and my gear winds up covered in salt spray. But even at the worst of times, I’d never seen a lens that bad.
One of my co-workers spent an entire day trying to clean the lens. Stuff seemed to move around, but he found it almost impossible to actually take stuff off. By the end of the day the lens looked cleaner, but it was still bad. We all took bets on what the performance would be the next time it went on sky. I thought we’d see at least some of the performance restored from his cleaning efforts. None of us expected what happened, though. That 20% increased to a 50% loss of light. We’d made things worse. We pulled the instrument off the telescope and went up the next day to take another look at that lens.
We’d just had the worst ice storm of the season. The roads were still closed to the public while the road crews took the snow blowers up to clear the roads. When we finally got to the observatory and took a look at the instrument, what we saw wasn’t promising. After a very brief hands-off investigation, we started taking things apart.
Our first thought was that we were dealing with a contaminant, so that’s how we proceeded. We tried every solvent we could think of to take the “stuff” off the lens: DI water, methanol, isopropanol, acetone (I cringed at this one), toluene… None of it seemed to do a thing.
We did eventually find some chemicals that had an effect, but the amount of scrubbing necessary to make them do anything at all was really discouraging. Eventually we found that weak acids had a stronger effect, and required less scrubbing.
We used two techniques to image the glass throughout the process: reflected light photography and dark field photography. Reflected light photography highlights differences in thickness in thin film layers through color change. Dark field illumination highlights contaminants like dust and other particulates. We used reflected light to show the spot of “clean” we were able to make with a weak acid.
As splotchy as the reflected light photos were, it wasn’t until we used dark field illumination that the true extent of the “contamination” became apparent.
We made a comprehensive dark field survey of the lens to see what we were dealing with. A couple of features stood out: The center of the lens appeared to be cleaner than the rest of the surface. This is likely because of the cleaning technique used earlier – radial strokes from the center to the edge. The center was the starting point of each pass across the glass, so it got the most cleaning.
Another feature we saw was the dark ring about 1/3 of the way out from the center. Our optical engineer said this is the mark left by the vacuum fixture the lens manufacturer used to manipulate the glass during final polishing and coating.
The rest of the glass appeared to be covered by a blotchy layer of… something. We took samples and sent them off to a company so they could perform an assay to tell us what the heck was going on.
Meanwhile we borrowed a digital microscope from one of the other observatories on the mountain and set it up to take a closer look at what this contaminant was. Under that much magnification we were running into issues with vibration, so we put everything on the table and tried not to breathe when we were taking exposures.
(As a quick side note, that’s my Bogen 3021 legs and 3047 head I got for doing large format photography work back in ’95! It’s the most solid tripod I’ve ever owned. Despite having one of its feet burned off in lava in 2001, it’s still going strong. I love good gear!)
What we saw under the microscope wasn’t encouraging. Either the entire lens was covered in bumps of… something… or we had a bigger problem. Were those craters?
We tried illuminating the patch under the microscope from various angles to see if we could figure out what was tall and what was short. But it wasn’t until we moved the microscope to the edge of the glass that we knew for sure.
We were looking at pits in the coating. Something had killed the anti-reflection coating on our lens.
This is when the investigation divided forces. One side of the investigation pursued the question of what had caused the damage. The other side pursued the question of how we were going to recover and get the instrument back on the sky. We collected a new set of samples to send out for assay and moved the lens to another lab to begin the next stage: fixing whatever had gone wrong.
After an understandable amount of deliberation and debate, the powers that be decided the best course of action was to remove the coating from the lens. At its very best, an anti-reflection coating cuts the reflectivity of an air-glass interface by about 4%. Typical coatings recover more like 2% of the light that is normally reflected. We were looking at a 50% loss due to scattering from the damaged coating. Removing the coating would get us back in the 2-4% range. It was an irrevocable step to take, but it was clearly a win.
So we went back to the weak acids that had worked earlier, and began to experiment with concentration. The trick was to strike a balance between an acid that was strong enough to remove the coating in no more than a handful of applications, but not so strong it would attack the underlying glass. Our optical engineer identified a handful of acids that would be reasonably safe to use on the glass, and maximum exposure times we could use without damage. Hydrochloric seemed to be the best match, so we went back to testing.
At 4% concentration, HCl was clearly removing the coating after only ten seconds of light scrubbing with a swab. The only problem was that even at 4% it was stronger than the concentration used by the glass manufacturer during their own testing.
We switched to a weaker solution: 50:1. At the same time we also wanted to minimize mechanical abrasion of the glass, so rather than going for a more aggressive swabbing action with the weaker acid, we tried a prolonged soak using a lens tissue saturated with acid.
After five minutes with an approximately pH 1.0 solution, we finally had something we could turn into a real procedure for stripping the lens.
At these concentrations the acid was about as strong as the white vinegar we’d tried earlier, and vapor concentrations were low enough not to require respirators. Respirators can be uncomfortable at the best of times, but at 14,000′ of altitude the restricted breathing is more than just a matter of comfort. It can mean having to break up work shifts to give people the chance to breathe. Working with just gloves as PPEs made life a lot easier.
There is something incredibly wrong about deliberately pouring acid onto a coated optic. Even knowing that the coating was shot, and that we couldn’t operate with that coating in place, I felt dirty when we loaded the first of several “patches”. Some part of me whispered, “You’re going to optical hell for this, boy.” With my heart in my throat I poured spoonful after spoonful of acid onto the patch.
All of this was complicated by the fact that I was still doing photography of the procedure as we went. (Now you understand why I used time lapse cameras!) At times the cameras got in the way and slowed things down, but we all agreed that having a good documented record of the work was more important in case we missed something and had to re-establish what we’d done at some later date.
It certainly helped that the technique worked. It wasn’t perfect, but it worked. We set up the next patch and began working in earnest.
There were a couple of features on the glass we weren’t certain of. One was the dark ring a third of the way out from the center – a ring left by the vacuum fixture the manufacturer had used to handle the glass. Would the acid work on that area? When we looked at it under the microscope the coating appeared to be less damaged than in other areas. Would that make it tougher to remove? Or easier?
The answer, as we learned, was tougher. The coating at the ring was less damaged than in other areas, but it was still damaged. It still needed to be removed. We only hoped that longer exposure to acid would eventually take it off.
By the end of the first day we’d made some real progress. We were clear out to the dark ring, and had started on another potential problem area of the glass. After doing a DI water wipe to remove any remaining acid we made a dark field image to see how we were doing. The prognosis? GOOD!
For the record, the apparent scratches in the center of the lens are actually on the lens cover we bolted to the bottom of the lens. The glass itself is scratch-free.
Day two was more of the same. We worked our way out toward the edge of the lens and continued to work on problem areas like the dark ring and a couple of other spots. As the problem areas got smaller, our patches also got smaller. At one point someone remarked that it looked like the lens had had a massive shaving accident. By then we knew the technique was clearly working, so the joke wasn’t as forced as it might’ve been earlier in the week. We could actually laugh without wincing.
As we neared the outer edge of the lens, the slope of the glass made it more likely that acid would run off our patches and down to the RTV that held the lens in its cell. Rather than spooning it on as I had the previous day, I switched to a pipette. Over the course of the day I put about a third of a liter of acid on our lens 0.09ml at a time. By the end of it my thumb was tired.
The lens has a 20mm wide baffle that fits around the outside diameter of the glass. This shadows the transition onto the AR coated surface, and covers the first 10mm of coating. We decided to leave the coating on the glass in this area. It wouldn’t affect the lens’s performance on sky, and it meant we had something of the old coating left in case we decided to get more chemical assays done in the future.
In the areas we did remove, though, we saw a curious thing: We could still tell where the AR coating had been. Something was still there. Whether it was an undamaged underlying coating or just a chemical stain on the glass we couldn’t tell. It didn’t seem to scatter light the way the damaged coating had, though, so we left it alone.
At the end of the second day the lens looked like… well… like a lens again. We did a final rinse with DI water and a final clean with methanol, then reinstalled the baffle and called it good.
The next day we blew it off with our version of canned air: high purity nitrogen and a regulator. In volume it’s cheaper than canned air, runs no risk of depositing stuff on the glass by accident, and lets us cover a lens this size in a matter of minutes. It’s a heckuvalot faster than the little Rocket Blower I use on my camera gear!
With the lens back on the instrument and the baffle back in place, we took one last look before putting it on the telescope. This is almost the same lighting that we used the first time we looked at the lens several weeks prior. The difference is striking. Before, I couldn’t even get my camera to focus on the bolt heads inside the optical tube assembly. This time? Not only could I focus on each of the bolts in the OTA, I could see all the way up to the pickoff mirrors for the guide cameras, and focus on the optics of the guide cameras themselves. Now that’s what a lens is supposed to look like!
Of course the day ran late. We were all getting used to leaving the mountain later than normal. With the instrument reassembled, we all sighed a big sigh of relief. And I finally got back to doing the kind of photography I prefer: landscapes.
A little less snow and ice than when we started, but I couldn’t pass up the light.
We went up the next day to put the instrument back on the telescope. Everyone was eager to hear the news: Did we get the light back? A couple of people stuck around headquarters after the exchange to watch the first images come off at the beginning of the night. I opted to go home and spend the evening with my family. About 7:30pm I got a call from the remote observing room: The images looked great! The light was back! It was the best news I’d heard in weeks.
At 8:30pm I got a second call, this time from my boss. The filter mechanism had jammed. “When do we go up?” I asked. I knew the answer. It was bound to be some version of “Right now!” We drove back up and cleared the filter jam. While we were working on it the glycol chiller system stopped working, so we purged that. Then our all-sky infrared camera stopped working, so we worked on that. It was almost 2am by the time we finally headed home for the second time.
“Hey, is that the volcano?” my boss asked. Off in the distance we could see the Halemaumau vent lighting up its cloud of volcanic gases. “Sure is!” I replied. “Pull over and take a picture,” he said.
This is the thing I love about the people I work with. My boss had been up the night before doing on-sky engineering. We’d just been through weeks of hell trying to get this instrument back on sky. Its first night back, two more systems on it fail. By the time we finish it’s two in the morning and we’re zonked. But people still take the time to appreciate the beauty of the place we live, work in, and call home. I pulled over and set my camera down on a rock so I could take a picture of the volcano at night.
It was a good day.