The View Up Here

Random scribblings about kites, photography, machining, and anything else

  • Flickr Gallery

Archive for the ‘Astronomy’ Category

Job Satisfaction

Posted by Tom Benedict on 20/09/2016

One of the questions I’ve asked myself over the years is whether I would be proud to tell my kids what I do for a living.

Much of the time the answer has been yes. When I worked for Academic Computing at the University of Texas I spent close to a year working in the Student Health Center. We took care of the servers that maintained medical records, handled customer satisfaction surveys, and made sure the desktop and handheld computers of the doctors, nurses, and administrators all worked. We helped them help the student body of UT stay healthy. Feel good about it? You bet!

Some of the time the answer has been a resounding no. The last year I was with IBM my job was to spy on my co-workers. I was the weenie who read all the logs from all of our servers and flagged “security risks”. During that time we never had an actual outside attack, and I think we had fewer than ten internal “ethical hacks”, all of which we caught. But I lost count of how many times my co-workers had a typo or tried to do something as root out of innocent ignorance. I had to report them all. Not one was a malicious act, and yet my job was to ding them for it anyway. Feel good about it? You gotta be kidding me…

Working at CFHT is solidly in the yes category. I’ve had downer days. Heck, I’ve had downer months, if not years. But at the root of it all we’re in the business of exploring the universe to better understand how the whole thing works. When people ask what I do for a living I tell them that my job description basically amounts to doing whatever is required so we can collect science-grade photons at night. Sometimes this means designing and building new instruments; sometimes it means sweeping the floors. When things get floor-sweepy it’s easy to lose sight of the fact that all of the things we do here contribute to our understanding of the universe.

Yesterday I brought in my camera bag and lighting gear so I could photograph a set of filters for customs paperwork. We’re shipping the filters to France to be scanned on a better spectrophotometer than the one we have here. Customs had a set of requirements for the photographs, so I was taking my time to make sure I got everything right. Toward the tail end one of our resident astronomers came in to see what I was doing. I explained about customs, about their need for documentation and serial numbers, etc. Not exactly sweeping floors, but documentation photography is pretty mundane stuff.

He listened patiently, then said, “You understand the importance of what you’re doing?”

“What do you mean?” I asked as I moved the last of the filters from the lighting scoop, back to its packing crate.

“Right now the tightest constraint on the cosmological constant is the SNLS survey, made with these filters. The scans they’re planning to do will further refine our understanding of the cosmological constant.” He pointed to the filter I was holding, the r’ filter. “That filter is key.”


Yeah. Even the act of photographing these filters so the customs agents can identify them and their serial numbers was helping to contribute to our understanding of the universe. No “Eureka!” moment. No lone genius. Just a lot of people doing a lot of seemingly mundane tasks, all of which is further refining our knowledge of how the universe works.

Today I’m processing the pictures, putting together documentation packets for customs, and packing the filters lovingly in their cases for shipment to France. Am I proud to tell my kids what I’m doing for a living?

You bet your ass.


Posted in Astronomy | Tagged: , , , , , , | Leave a Comment »

SPIE 2016 – Poster Done Too

Posted by Tom Benedict on 09/06/2016

And now the poster’s in the bag, too.

SPIE 2016 Astronomical Instruments and Telescopes - Poster

It’s not my most visually appealing poster, but the subject matter doesn’t really call for a lot of elaboration. It’s mostly a data dump of all the spectra I took over the past several months. Just for grins, the bar at the bottom is a gallery of all of the samples photographed with my NIR-converted A2200 point ‘n shoot. (Yes, this actually factors into the paper.)

The two columns on the left contain spectra from all of the samples, scaled from 0-50% reflectivity. The two columns on the right are where the good stuff is: With the exception of the bottom two graphs, it’s only the materials that reflected less than 10% of the light across the whole spectrum. That’s where the useful materials are.

So why include the others? Those are the ones to avoid! The paper wouldn’t be complete if I didn’t include them. Unfortunately, some of the materials we’ve been using for years for stray light control fell into the “avoid at all cost” columns. Bummer. But now we know better.

The poster is printed, and I shoved it in the mailing tube with all of the other posters from our group this morning. All that’s left now is to get my butt on a plane to Edinburgh and present the thing.

Hip hip hooray! Scotland, here I come!


Posted in Astronomy | Tagged: , , , , , , , | Leave a Comment »

SPIE 2016 Manuscript Done

Posted by Tom Benedict on 30/05/2016

Last week didn’t turn out quite the way I’d intended. Right after writing my last post I got a call from my sister to tell me my father had to go to the emergency room. Neither of my siblings were in a position to fly in to help him, so I offered. I spent last week helping him get back on his feet, get to all the doctor’s appointments my sister set up for him, and figure out his next move. This meant I wasn’t spending that time making further edits on my SPIE paper or flying kites and cameras for World Wide KAP Week, but I wouldn’t have wanted it any other way. WWKW rolls around once a year, and I knew I could submit the manuscript to SPIE remotely no matter where I was in the world. I needed to be at my father’s side, so that’s where I was.

Turns out I didn’t need to submit the manuscript remotely, though. I got back two nights ago, a couple of days before manuscripts were due. I gave the paper one last looking over just in case. Just… In… Case… Yeah.

Here are some lessons I learned from “just in case”:

  1. No matter how many times you check your spacing, there’s always a space somewhere you don’t want it. (Yes, I’m using this to justify how anal I am when editing.)
  2. When proofing a paper, also check captions and figure titles. I had one graph labeled “Diffuse Reflectance of Bulk Materiaw 1.5ls”. Um… What?! (Global search and replace can be a real bitch at times.)
  3. Be sure to catch all your little place-holders and fix them. One sentence included “…overall reflectivity between 6-?% across the full range…” In three rounds of editing by multiple people, no one caught that. Not even me.
  4. I always put in too many commas when writing a first draft.
  5. I always leave too many in during subsequent edits. There’s always one more comma to kill.
  6. Above all else, listen to the input from your co-authors. Right before flying out to be with my father I had a frenzied text conversation with one of the co-authors on the paper who insisted on a particular change in the paper. I disagreed, but I had to drop it when I got on the plane. When I got back I found I agreed with him. I made the change, and the paper was stronger for it.

That last one really applies to all forms of writing, not just technical and scientific papers. Listen to your editors. Listen to your co-authors. Listen to people who tell you something doesn’t make sense, doesn’t flow, or is just plain wrong. Even if it means a complete re-write it means you’re connecting with at least one more person when you finally publish.

I submitted the manuscript this morning. I’ll start designing the poster tomorrow.


Posted in Astronomy, Writing | Tagged: , , , , | Leave a Comment »

SPIE 2016 – Manuscript (almost) In The Bag! / World Wide KAP Week 2016 / Visitors

Posted by Tom Benedict on 20/05/2016

Manuscripts for the 2016 SPIE Astronomical Telescopes and Instrumentation Conference are due May 30th, so a little over a week off. With my usual level of good planning I didn’t associate that date with a visit from my friend and his wife (May 27th-29th) or the dates for World Wide KAP Week 2016 (May 13th-22nd). So no, of course I didn’t get an early start on things! I left things ’till this week. >sigh<

But I think I’ve got a workable rev of my manuscript in the bag. The graphs took a while to sort out, but everything came together this afternoon. I still have at least one round of editing left to go, but at this point it doesn’t have to occupy my every waking moment.

I think I avoided impacting my friend’s trip, and I still have this weekend for WWKW. Not ideal, but not a complete loss, either. I’m picking out a couple of subjects for WWKW, and tonight I’m going home to clear my chips and charge my batteries. Kite flying and kite photography, here I come!!



Posted in Astronomy, Kite Aerial Photography | Leave a Comment »

SPIE 2016 – Surprises Along The Way

Posted by Tom Benedict on 06/05/2016

I spent the last week scanning all the samples I’ve been prepping for the last several months. We’d scanned some of these materials before, so some of the results didn’t come as a surprise, but a couple did.

The scans I’m running are what’s called “Total Integrated Reflectance”, or TIR. The idea is to illuminate a sample with a particular wavelength of light, collect all the light reflecting off of it at that wavelength, and measure it. By doing this at a bunch of different wavelengths you build up a TIR spectrum. I’m running spectra from 250nm to 2500nm, or from the UV to the near-IR. This covers any instrument that uses a CCD detector along with many instruments that use near-IR detectors like the H2RG from Teledyne.

What you want to see is a flat spectrum around 5% or below from 250nm to 2500nm. That indicates a material that’s good to use to control stray light across that entire range. Unfortunately that’s true for some materials but not for all of them, even if they look black to the human eye. One of the more surprising sets of materials were the black flock papers from Edmund Optical. These are a mainstay for controlling stray light in instruments and amateur telescopes. I was caught off-guard when the reflectance of all three of the flock samples popped up long of 720nm! They’re black to the human eye, but in the near-IR they’re closer to a light gray.

Here’s a look at four of the materials I’m scanning, photographed using an IR-converted CCD camera. The filter in this camera cuts on gradually from about 650nm to around 720nm so it’s not ideal as a measurement tool, but it gives a good idea of how the materials behave from 720nm out to about 1000nm.

SPIE 2016 - Black Samples in the NIR

The blackest material in this set of four is the Industrial Strength Velcro. But only the hook side! The loop side is made from some other material that’s remarkably reflective in the NIR. The black flock paper to the left of the Velcro is considerably more reflective, even though to the human eye it looks much darker. One of the other surprise materials was the Black Tak tape from City Theatrical. Even though it appears not to be very black in this photo, that’s only around 5% reflectance. Even better, it maintains that reflectance across the entire spectral range. It’s neat stuff!

The half-tone bar at the top was printed on a normal laser printer. Toner, which is almost straight carbon, is quite good at absorbing light across the entire range of the tests. (Yes, toner-covered paper is one of the samples I’m running.)

I’ve run into some other surprises along the way, but those will have to wait for the paper. I’ve got another couple of days of scans still ahead of me before the real work begins: analysis and writing. Then I get to design a poster!


Posted in Astronomy, Engineering | Leave a Comment »

SPIE 2016

Posted by Tom Benedict on 08/04/2016

Turns out I’m going to the SPIE Astronomical Telescopes and Instrumentation 2016 conference in Edinburgh, Scotland, from June 26th through July 1st! I’m holding out hope my neck will have healed by then and I can do some kite aerial photography while I’m there. Either way I’m planning on bringing sound gear so I can play in a new environment and come home with some fresh sounds to play with.

I’m still writing the manuscript for my paper, and the poster hasn’t even made it to the sketch pad yet, but the research has been underway for a couple of years. As boring as this may sound, the paper is all about black stuff.

Whether you’re building an optical experiment on a workbench, modifying or building a telescope for your own use, or working at an observatory, at some point you’ll need some way to control stray light. The bulk of stray light control happens in the design: adding baffles and stops, planning light traps, or controlling the output cone of any light source. But inevitably least one of those will involve making some surface black.

The question is: What does “black” mean? And once you define it, what materials fit the definition?

From the standpoint of stray light control, “black” means “doesn’t reflect much”. But what’s “much”? 5%? 1%? 0.01%? And what range of wavelengths do you care about? CCDs can detect light well outside the range of human vision. What about infrared arrays like the H2RG? They can see well past human vision into the near edges of the thermal infrared.

At first glance the obvious answer is that it shouldn’t reflect anything at any wavelength. Unfortunately such as beast doesn’t exist.

The next best thing would be that it shouldn’t reflect much at all (0.01%?) at all the wavelengths we care about. For the sake of making it a tractable problem let’s say from the UV to K-band infrared, or about 250nm out to 2500nm. The problem is that 0.01% is tough to hit. It’s possible to get close using an exotic surface like carbon nanotubes, but such surfaces are fragile and tough to impossible to clean without damaging them. Let’s face it: astronomy is an outdoor sport. We try to close the domes when it rains, but things do get dirty. They do occasionally get wet. Eventually they’ll need to be cleaned. Ultra fragile exotic surfaces simply aren’t practical much of the time.

But maybe we don’t need 0.01% reflectivity. If we’re careful with our design we can make sure that any stray light has to reflect off of at least two surfaces. Let’s say we’re even more careful with our design and make it so light has to reflect off of at least three. That was the design rule I used when I designed the optical baffle for the Megacam Wide Field Corrector, shown here without its outer skin.

Megacam Baffle Internals - Rendered

If light has to bounce off of at least three surfaces, that means each reflection has the opportunity to absorb light. If each surface reflects a whopping 5% of the light, by the time you’ve bounced off three surfaces you’re down at the 0.01% level. 5% on three surfaces is a heckuvalot easier to deal with than 0.01% on one surface, and it brings our material selection into the realm of common off-the-shelf materials.

Now that we have a good working definition of a practical, workable “black”, that leads us to the second half of the question: What materials fit the bill?

At work I have the good fortune of having a spectrophotometer at my disposal. I’ve used it to scan instrument filters, optical assemblies, even the odd pair of sunglasses. But I’ve also scanned a bunch of stuff that looks black. In the process I’ve run into some surprises. Some years back we made a wrap for one of our instruments in an effort to keep light from LEDs and motor encoders from encroaching on the beam. Out of our own ignorance we used a synthetic black cloth that looks extremely black to the human eye, but turns out to be blindingly reflective above 690nm. We hadn’t fixed the situation at all. We’d made it worse! We replaced the wrap with a different fabric that’s absorptive out past 1100nm (the limit of what a CCD can see), and that solved it.

Over the years our growing catalog of things that are and aren’t black has come in handy. We’ve added paints, papers, tapes, and bulk materials like plastics. We figured it was time to share.

Of course we’re not the only ones doing this. Two years ago a group from Texas A&M published a paper at the SPIE Astronomical Telescopes and Instruments 2014 conference on exactly the same topic. They covered a number of metal treatments – anodizing, black electroless nickel plating, oxide treatments, etc. – in addition to a number of paints and a handful of other materials. But as it turns out our catalogs have almost no overlap, so our paper will be a good follow-up to theirs.

I’m in the process of finishing off the last of our scans over the next few weeks, and finishing the manuscript over the next month. Then it’s on to poster time!

Now I just need to figure out a way to make a visibly engaging poster that’s mostly… black.


Posted in Astronomy, Engineering | Tagged: , , , , | Leave a Comment »

Sitelle – Here at Last

Posted by Tom Benedict on 08/07/2015

A little over four years ago I posted a rendering to Flickr.

Cryogenic Detector Internals

I’d been tasked with designing and building new internals for a pair of matched cameras. The rendering was what I thought would be the final design of these parts. I try to save my Flickr stream for photos, but it was an important milestone in my professional life. It was also visual, aesthetically pleasing, and geeky, so I figured it was fair game.

I was wrong, of course. Shortly after posting that rendering I was told we couldn’t use the cryostats I’d design these to fit into, so I had to design two entire cameras. I kept most of the internals the same, but the outer part was completely new. Over the course of several months the rest of the camera design took shape.

Sitelle Camera September, 2012

We held a final design review, I got the go-ahead to start cutting metal, and a few months after that we had our first prototype camera.

Sitelle Camera Pumping

Early in the project we decided to build three cameras. The first was the prototype. The other two were the final cameras. This let work continue on the prototype as the final cameras were manufactured and finished. As each part of the prototype camera tested out ok, we made those parts for the final pair of cameras.

Sitelle Parts

Once we had complete sets of parts, they all went off to be anodized. Marc, the guy designing and building the electronics for the cameras, liked gold. So gold they became.

CFHT Sitelle Parts Anodized

Since it’s a porous surface, anodizing doesn’t work well in vacuum. So the insides of the camera bodies were machined clean and polished. The internal components were never anodized at all.

CFHT Sitelle Camera Assembly

Two internal components went through a number of revisions along the way: the cold strap and the getter – the cryosorb pump that maintains vacuum while the cameras are in use. We went through a number of designs for each of these. The first generation is shown above. We went through two more before we finalized the design for the getters, and almost that many cold strap designs. In addition to the box of unused getters and cold strap assemblies we now have in our clean room, we eventually wound up with two fully functioning cameras.

Two For Sitelle

We boxed the cameras up and shipped them off to Quebec for integration with the rest of the instrument. We had a number of issues that still needed to be addressed, the largest of which was that the hermetic space-rated connectors we used for the feed-throughs leaked once they were potted. We’re still dealing with this, though we’ve got what I hope is a real solution for the next generation of feed-throughs.

The cameras lived in Quebec for almost two years. A little over a month ago the team building Sitelle bid it farewell, loaded it into an airplane, and shipped it to Hawaii. After several weeks of unpacking, re-assembly, and testing, we finally put Sitelle on the telescope.

One Instrument - Two Cameras

And there at the end of each beam of the instrument is one of the cameras Marc and I built, beginning with that early design of the internals a little over four years ago. Designing and building the cameras was an exciting, if exhausting, process. But seeing them installed as part of a new instrument is indescribable.

Home At Last

As one might guess, I did a bunch of photography of the installation for first light. Toward the end, after balancing, after checkout, while we were testing the long instrument limits on the telescope, Marc and a number of other people involved in the project stood, watching the clearance between the instrument and the south bearing. I was struck how similar it was to the final scene of Ocean’s Eleven. I had to smile as I tripped the shutter. It seemed a fitting end to a wonderful caper.

Sitelle's Eleven

And the beginning of an even better one. Sitelle had a successful first light last night. Marc and the rest of the team are taking it through its commissioning tests, which will take several more nights. After that it will be unleashed on the sky, taking four million spectra at a time.

I can’t wait!

– Tom

Posted in Astronomy, Engineering | Leave a Comment »

Know Where Your Filters Are

Posted by Tom Benedict on 13/02/2015

In an earlier post I described some of what we did at work to fix the damage to the optics on one of our instruments. This instrument has been a constant source of fun and excitement. Prior to the lens damage it had another catastrophic failure in which one of its filters fell out of the filter jukebox and destroyed part of the filter change mechanism. That prompted an immediate instrument change so we could take it apart and fix the damage. But it also prompted a longer-term investigation into how the fault occurred so we could make sure it never happened again.

Something Wrong

As near as we could tell, one of the latches that keep the filters in the jukebox failed, which allowed the filter to slip out and into the space the guide rails occupy. When the jukebox came down, it smashed the tips off the guide rails. We held a brainstorming session to come up with ideas to stop this from happening and to detect if it ever does. The preventative measures were relatively straightforward. The sensing took more work. Some of the approaches were relatively easy to implement, but didn’t give us the level of sensing we were after. One of the more difficult and invasive approaches gave us all of the information we wanted, but would take the most time to implement. Needless to say that’s the option we ran with.

The filter jukebox has eight slots, so the instrument can hold eight filters. We added proximity switches to each slot so we can tell if it’s occupied by a filter. We also added sense lines to each of the latches that act on the filters so we can tell what state they’re in. Finally, we added a ninth proximity switch in the in-beam position so we can tell if a filter is in the beam. This lets us detect filters falling out of the jukebox, like the one that caused the earlier failure. It also lets us sense if the filter change mechanism had a mechanical failure and let go of a filter before it was either fully deployed in-beam or fully stowed in the jukebox. Basically, it lets us know if the mechanism does anything it’s  not supposed to do.

Installing Electronics

The electronics were designed and built by one of my co-workers. I did most of the mechanical work – the proximity switch plate, the light baffle, the modifications to the bulkheads to pass the latch sense signals through, etc. My boss and my co-worker did all  of the cabling and came up with the test plan.

New Filter Sensing Instrumentation

It took almost a week to install the electronics. Some of that was spent building and testing cables. Most of it was spent dealing with what we call “summit issues”: things that go wrong simply because they go wrong. My one contribution to the electronics was to build a ribbon cable. Seems simple enough. I’ve never messed up a ribbon cable. I mean, how hard can it be, right?

When I plugged in my new cable one of the proximity switches self destructed in a most spectacular way. Hissing, a bit of flame, and gouts of brown smoke gushing out from under the proximity switch plate. We took the entire system back to headquarters to replace the damaged parts and figure out what went wrong.

Back at HQ I took the ribbon cable apart, expecting to see a bent pin, a flawed ribbon cable, or any of the other likely causes for such a failure. What I found was an aluminum shaving wedged between two of the pins. Apparently when the mounting holes for the new electronics were being drilled, some metal chips fell into the box of spare parts and one of them wedge inside the IDC connector I used to build the cable. Including the drive to and from HQ and the time I spent cleaning the smoke damage off of all of the filters, total time lost was close to a day… for a metal chip. “Summit issue.”

Happiness is a Working Instrument

We finally finished all the work yesterday. We tested it thoroughly, then tested it again. Then tested it again. My boss was kind enough to pose for me at the end of what for him has been seven straight weeks of work on this instrument. (So if he looks a little stressed, cut the guy some slack.)

Back on the Telescope

With all new rails, jukebox rails, filter frames, and a wonderful new sensing system, it was finally time to put it back on the telescope.

– Tom

Posted in Astronomy, Engineering | Leave a Comment »

Heavens and (Optical) Hell

Posted by Tom Benedict on 19/01/2015

In an earlier post I mentioned we were in the middle of a big investigation into the degradation of one of our instruments. My active role in the investigation finished up Friday, so I’m breathing a big sigh of relief. Meanwhile I started processing all the photographs I made in the course of the investigation to document the work done. I ran a couple of time lapse cameras, so the photos number in the thousands. I only used a couple of the time lapse photos, though, so the number of “keepers” was a lot smaller.

I picked a subset of the keepers to narrarate the story thus far:

CFHT with Megacam

A couple of months back one of our astronomers analyzed the images coming off of our wide field imager to measure the zero point of each filter. A zero point is the measurement of the transmission of the instrument is in a given band. Because of dirt and grime building up on our primary mirror, we expect some drop in the zero point over time. What we didn’t expect was what he saw: a 20% drop in performance in g’ band with slightly less extreme drops in all the other bands. What made this worse was that we had just re-coated the mirror. We expected to get a marginal gain, not a dramatic loss. Clearly something was wrong. When the instrument came off the telescope we went up to take a look.

First Look - Something Wrong

I’ve done plenty of sunset photography at the beach. Most days I get away with some good frames and a camera that needs a damp wipe. Other times I get stuck in an onshore and my gear winds up covered in salt spray. But even at the worst of times, I’d never seen a lens that bad.

One of my co-workers spent an entire day trying to clean the lens. Stuff seemed to move around, but he found it almost impossible to actually take stuff off. By the end of the day the lens looked cleaner, but it was still bad. We all took bets on what the performance would be the next time it went on sky. I thought we’d see at least some of the performance restored from his cleaning efforts. None of us expected what happened, though. That 20% increased to a 50% loss of light. We’d made things worse. We pulled the instrument off the telescope and went up the next day to take another look at that lens.

Winter on the Mountain

We’d just had the worst ice storm of the season. The roads were still closed to the public while the road crews took the snow blowers up to clear the roads. When we finally got to the observatory and took a look at the instrument, what we saw wasn’t promising. After a very brief hands-off investigation, we started taking things apart.

Removing L1

Our first thought was that we were dealing with a contaminant, so that’s how we proceeded. We tried every solvent we could think of to take the “stuff” off the lens: DI water, methanol, isopropanol, acetone (I cringed at this one), toluene… None of it seemed to do a thing.

Early Testing

We did eventually find some chemicals that had an effect, but the amount of scrubbing necessary to make them do anything at all was really discouraging. Eventually we found that weak acids had a stronger effect, and required less scrubbing.

Finding what Cuts the Goo

We used two techniques to image the glass throughout the process: reflected light photography and dark field photography. Reflected light photography highlights differences in thickness in thin film layers through color change. Dark field illumination highlights contaminants like dust and other particulates. We used reflected light to show the spot of “clean” we were able to make with a weak acid.

Dark Field Illumination Technique

As splotchy as the reflected light photos were, it wasn’t until we used dark field illumination that the true extent of the “contamination” became apparent.

Dark Field Image

We made a comprehensive dark field survey of the lens to see what we were dealing with. A couple of features stood out: The center of the lens appeared to be cleaner than the rest of the surface. This is likely because of the cleaning technique used earlier – radial strokes from the center to the edge. The center was the starting point of each pass across the glass, so it got the most cleaning.

Another feature we saw was the dark ring about 1/3 of the way out from the center. Our optical engineer said this is the mark left by the vacuum fixture the lens manufacturer used to manipulate the glass during final polishing and coating.

The rest of the glass appeared to be covered by a blotchy layer of… something. We took samples and sent them off to a company so they could perform an assay to tell us what the heck was going on.

Using the Microscope

Meanwhile we borrowed a digital microscope from one of the other observatories on the mountain and set it up to take a closer look at what this contaminant was. Under that much magnification we were running into issues with vibration, so we put everything on the table and tried not to breathe when we were taking exposures.

(As a quick side note, that’s my Bogen 3021 legs and 3047 head I got for doing large format photography work back in ’95! It’s the most solid tripod I’ve ever owned. Despite having one of its feet burned off in lava in 2001, it’s still going strong. I love good gear!)

Microscopy - The Problem

What we saw under the microscope wasn’t encouraging. Either the entire lens was covered in bumps of… something… or we had a bigger problem. Were those craters?

We tried illuminating the patch under the microscope from various angles to see if we could figure out what was tall and what was short. But it wasn’t until we moved the microscope to the edge of the glass that we knew for sure.

Microscopy - Even the Edge is Bad

We were looking at pits in the coating. Something had killed the anti-reflection coating on our lens.

This is when the investigation divided forces. One side of the investigation pursued the question of what had caused the damage. The other side pursued the question of how we were going to recover and get the instrument back on the sky. We collected a new set of samples to send out for assay and moved the lens to another lab to begin the next stage: fixing whatever had gone wrong.

Test Swabbing

After an understandable amount of deliberation and debate, the powers that be decided the best course of action was to remove the coating from the lens. At its very best, an anti-reflection coating cuts the reflectivity of an air-glass interface by about 4%. Typical coatings recover more like 2% of the light that is normally reflected. We were looking at a 50% loss due to scattering from the damaged coating. Removing the coating would get us back in the 2-4% range. It was an irrevocable step to take, but it was clearly a win.

So we went back to the weak acids that had worked earlier, and began to experiment with concentration. The trick was to strike a balance between an acid that was strong enough to remove the coating in no more than a handful of applications, but not so strong it would attack the underlying glass. Our optical engineer identified a handful of acids that would be reasonably safe to use on the glass, and maximum exposure times we could use without damage. Hydrochloric seemed to be the best match, so we went back to testing.

Test Swab Technique

At 4% concentration, HCl was clearly removing the coating after only ten seconds of light scrubbing with a swab. The only problem was that even at 4% it was stronger than the concentration used by the glass manufacturer during their own testing.

Reflected Light Image - HCl Concentrations

We switched to a weaker solution: 50:1. At the same time we also wanted to minimize mechanical abrasion of the glass, so rather than going for a more aggressive swabbing action with the weaker acid, we tried a prolonged soak using a lens tissue saturated with acid.

Reflected Light Image - First Successful Soak Removal

After five minutes with an approximately pH 1.0 solution, we finally had something we could turn into a real procedure for stripping the lens.

At these concentrations the acid was about as strong as the white vinegar we’d tried earlier, and vapor concentrations were low enough not to require respirators. Respirators can be uncomfortable at the best of times, but at 14,000′ of altitude the restricted breathing is more than just a matter of comfort. It can mean having to break up work shifts to give people the chance to breathe. Working with just gloves as PPEs made life a lot easier.

Highway to Hell - Putting Acid on a Lens

There is something incredibly wrong about deliberately pouring acid onto a coated optic. Even knowing that the coating was shot, and that we couldn’t operate with that coating in place, I felt dirty when we loaded the first of several “patches”. Some part of me whispered, “You’re going to optical hell for this, boy.” With my heart in my throat I poured spoonful after spoonful of acid onto the patch.

Making of - The Work Begins

All of this was complicated by the fact that I was still doing photography of the procedure as we went. (Now you understand why I used time lapse cameras!) At times the cameras got in the way and slowed things down, but we all agreed that having a good documented record of the work was more important in case we missed something and had to re-establish what we’d done at some later date.

Dark Field Image - The Work Begins

It certainly helped that the technique worked. It wasn’t perfect, but it worked. We set up the next patch and began working in earnest.

Staying the Course - The Work Continues

There were a couple of features on the glass we weren’t certain of. One was the dark ring a third of the way out from the center – a ring left by the vacuum fixture the manufacturer had used to handle the glass. Would the acid work on that area? When we looked at it under the microscope the coating appeared to be less damaged than in other areas. Would that make it tougher to remove? Or easier?

Reflected Light Image - The Ring

The answer, as we learned, was tougher. The coating at the ring was less damaged than in other areas, but it was still damaged. It still needed to be removed. We only hoped that longer exposure to acid would eventually take it off.

Dark Field Image - End of Day 1

By the end of the first day we’d made some real progress. We were clear out to the dark ring, and had started on another potential problem area of the glass. After doing a DI water wipe to remove any remaining acid we made a dark field image to see how we were doing. The prognosis? GOOD!

For the record, the apparent scratches in the center of the lens are actually on the lens cover we bolted to the bottom of the lens. The glass itself is scratch-free.

Finishing Touches - Pipette Works Better

Day two was more of the same. We worked our way out toward the edge of the lens and continued to work on problem areas like the dark ring and a couple of other spots. As the problem areas got smaller, our patches also got smaller. At one point someone remarked that it looked like the lens had had a massive shaving accident. By then we knew the technique was clearly working, so the joke wasn’t as forced as it might’ve been earlier in the week. We could actually laugh without wincing.

As we neared the outer edge of the lens, the slope of the glass made it more likely that acid would run off our patches and down to the RTV that held the lens in its cell. Rather than spooning it on as I had the previous day, I switched to a pipette. Over the course of the day I put about a third of a liter of acid on our lens 0.09ml at a time. By the end of it my thumb was tired.

The lens has a 20mm wide baffle that fits around the outside diameter of the glass. This shadows the transition onto the AR coated surface, and covers the first 10mm of coating. We decided to leave the coating on the glass in this area. It wouldn’t affect the lens’s performance on sky, and it meant we had something of the old coating left in case we decided to get more chemical assays done in the future.

Edge Effects - Something Still there

In the areas we did remove, though, we saw a curious thing: We could still tell where the AR coating had been. Something was still there. Whether it was an undamaged underlying coating or just a chemical stain on the glass we couldn’t tell. It didn’t seem to scatter light the way the damaged coating had, though, so we left it alone.

Reflected Light Image - Coating Gone Lens Back

At the end of the second day the lens looked like… well… like a lens again. We did a final rinse with DI water and a final clean with methanol, then reinstalled the baffle and called it good.

Last Dustoff

The next day we blew it off with our version of canned air: high purity nitrogen and a regulator. In volume it’s cheaper than canned air, runs no risk of depositing stuff on the glass by accident, and lets us cover a lens this size in a matter of minutes. It’s a heckuvalot faster than the little Rocket Blower I use on my camera gear!

Last Look - Something Right

With the lens back on the instrument and the baffle back in place, we took one last look before putting it on the telescope. This is almost the same lighting that we used the first time we looked at the lens several weeks prior. The difference is striking. Before, I couldn’t even get my camera to focus on the bolt heads inside the optical tube assembly. This time? Not only could I focus on each of the bolts in the OTA, I could see all the way up to the pickoff mirrors for the guide cameras, and focus on the optics of the guide cameras themselves. Now that’s what a lens is supposed to look like!

Of course the day ran late. We were all getting used to leaving the mountain later than normal. With the instrument reassembled, we all sighed a big sigh of relief. And I finally got back to doing the kind of photography I prefer: landscapes.

Winter Ice

A little less snow and ice than when we started, but I couldn’t pass up the light.

We went up the next day to put the instrument back on the telescope. Everyone was eager to hear the news: Did we get the light back? A couple of people stuck around headquarters after the exchange to watch the first images come off at the beginning of the night. I opted to go home and spend the evening with my family. About 7:30pm I got a call from the remote observing room: The images looked great! The light was back! It was the best news I’d heard in weeks.

At 8:30pm I got a second call, this time from my boss. The filter mechanism had jammed. “When do we go up?” I asked. I knew the answer. It was bound to be some version of “Right now!” We drove back up and cleared the filter jam. While we were working on it the glycol chiller system stopped working, so we purged that. Then our all-sky infrared camera stopped working, so we worked on that. It was almost 2am by the time we finally headed home for the second time.

“Hey, is that the volcano?” my boss asked. Off in the distance we could see the Halemaumau vent lighting up its cloud of volcanic gases. “Sure is!” I replied. “Pull over and take a picture,” he said.

This is the thing I love about the people I work with. My boss had been up the night before doing on-sky engineering. We’d just been through weeks of hell trying to get this instrument back on sky. Its first night back, two more systems on it fail. By the time we finish it’s two in the morning and we’re zonked. But people still take the time to appreciate the beauty of the place we live, work in, and call home. I pulled over and set my camera down on a rock so I could take a picture of the volcano at night.

Volcano and Stars

It was a good day.

– Tom

Posted in Astronomy, Engineering, Hawaii, Photography | 6 Comments »

Even More Cowbell

Posted by Tom Benedict on 11/01/2015

I still haven’t used my new strobe gear to do portraiture. Which is kind of odd, now that I think about it, because that was one of the things I had in mind when I got it. I put some of this down to portraiture being a relatively new form of photography for me, so I’m not as driven to jump in. And some of it has to do with my only having one light and no reflectors. It’s tough to do good portraiture without at least a bounce card. (I know, I know… A bounce card is no more complicated than a piece of posterboard. I’ll get to that in a second.)

The real reason? NO ONE wants to be a subject, much less hold a piece of posterboard I tell them is actually a bounce card. (See? I told you I’d get to it!)

But that didn’t stop me from hoarding all the gift cards my family gave me for Christmas and using them to get  a second strobe, light stand, and umbrella. And it sure didn’t stop me from adding a 42″ reflector and flash diffuser to my cart while I was at it. As I placed the order I told Rydra that she and our little minions were now fair game. Oh yeah, baby! The portraits will roll! (Unfortunately, all of them know where I sleep, and know how to pull the batteries out of camera gear. So maybe this won’t work out as well as I’d hoped.)

Meanwhile I’ve been enjoying my single flash a great deal. No portraits, but I’ve been using it at work a ton. Most recently I’ve been using it while we’ve been investigating an issue on a much larger camera than anything I carry in my bag.

Megacam on CFHT

This is Megacam. When it was built it was the largest digital camera in the world. It’s a 320 megapixel focal plane fitted to a wide field corrector that’s optimized for use on our telescope. (And yeah, it came with Linux drivers.)

My real involvement with Megacam started as it was being integrated for use on the sky. Shortly after it saw first light, people figured out that it needed something that wasn’t part of the original design: a light baffle. That was my first large-scale project at the observatory: design and build a baffle for Megacam. The big black can on the bottom that looks like a silencer is what I came up with. It’s about 2m high and about 1.2m wide. To date it’s the largest lens hood I’ve designed and built. Tucked up above it is the wide field corrector assembly: four lenses and one image stabilizer (heck yes it includes image stabilization!) And it’s there that the story begins.

A few months ago one of our astronomers alerted us to what looked like a steady degradation in instrument performance. We investigated and found that the bottom-most lens in the corrector was dirty. One of the guys at work tried valiantly to clean it, but even his best efforts left the lens looking… ooky. To make a long story short, we took everything apart, looked at it through a microscope, and figured out what was going wrong. To our dismay we found that the optical coating on the lens was literally falling apart.

I wind up doing a lot of the documentation photography at work. Any time something goes wrong that we need a record of, I’m asked to pull out my camera and get busy. As part of the investigation of this lens I had the opportunity to do ambient light photography, dark-field photography, and (you guessed it) flash photography.

I needed a set of reflected light photos of the outer edge of the lens, but the room lights were killing me. So I turned them all off and pointed my flash at the ceiling to use as a giant white card. It worked great.

MC WFC L1 Acids

It’s not perfect. You can see the light fixtures in the reflection of the ceiling. But we’re not trying to do image analysis with these. The point was simply to record the state of the coating at several points around the edge of the lens. I couldn’t have pulled it off in the time I had available without my strobe.

I’m sure my second flash head will find its way into the photography I do at work. But what I’m really excited about is finally finally jumping into portraiture.

– Tom

Posted in Astronomy, Photography | Tagged: , | 1 Comment »