The View Up Here

Random scribblings about kites, photography, machining, and anything else

When Things Suck: It’s Good?

Posted by Tom Benedict on 30/10/2010

In this instance they are, anyway:

Among the many systems we maintain at work, one of the most critical is our collection of vacuum systems.  Every instrument we use relies on vacuum to some extent, some more than others.  One of the easiest approaches for reducing noise on a CCD or CMOS detector is to cool them well below ambient temperatures.  But once a device is cooled below the ambient dew point, it will condense water onto it.  Water and electronics rarely mix well, and water and optics almost never do.  So you have to take away all the air around the device before it is cooled: vacuum.

The systems we run fall into the mid-range.  With low vacuum you can rely on mechanical pumps like roughing pumps or scroll pumps.  Achieving higher vacuum relies on more complicated systems: diffusion pumps, turbomolecular pumps, sublimation pumps, cryogenic pumps, you name it.  Higher vacuums also require a higher level of attention paid to things like dust, oil, water, and fingerprints.  Yep, fingerprints.  The out-gassing from a single fingerprint can outpace a fairly large pumping system at 10^-10 torr.  But I’m getting ahead of myself.  Our systems typically peak out at 10^-8 torr.  That’s good enough for me.

Pumps create a pressure differential.  Different pumps are suited for different absolute ranges of pressure, and can create only so much differential.  For example, a good dry scroll pump will get you into the 10^-3 range, but won’t draw you much beyond that.  A good turbomolecular pump will get you into the 10^-8 range, but it can’t do that if there’s atmosphere on the other side.  For mid and high level vacuum, pumps are stacked in series.  On our coating chamber we use a diffusion pump to pull our high vacuum, backed by a Roots blower, backed by a large roughing pump: three stages.  Our cryostat pumps use turbomolecular pumps backed by dry scroll pumps: two stages.

But no matter what kind of vacuum system you build, you will rely on pumps to get you there.  And you can only get there if they work.

A few months ago we had a test cryostat in our lab, and had it hooked up to our headquarters pump station.  Everything was fine, but after a while the vacuum gauge just wasn’t reading as low as it should’ve.  We’ve had issues with our gauges, so that’s where we checked first.  Low.  We compared that reading to the readings on the pump station’s gauges, and found they agreed.  The gauges were good.  The pump just wasn’t pulling vacuum.  Eventually we traced it to the dry scroll pump.  It had been in service for close to ten years with intermittent use, and it looked like it had finally been run too long.  When we uncoupled it from the system, it was clear why things had stopped working: half an inch of seal material was sticking out of the exhaust port.

Vacuum pumps come in two coarse varieties: those that use oil and those that don’t.  Our coating chamber system uses all oil pumps.  Our cryostat systems use all oil-free pumps.  The reason is simple:  If we have our primary mirror in the coating chamber and the vacuum system chokes, we just deposited oil on the glass.  The fix is to re-strip and clean the glass, scrub out the chamber, and do it again.  That’s a cost of roughly $15-20k.  To get an oil-free system that large is several hundreds of thousands of dollars.  Not going to happen.  But if we have a cryostat pump system on a camera and the vacuum system chokes, we could deposit oil onto several million dollars of detectors.  The cost of an oil-free system to pump our cryostats is several orders of magnitude cheaper.  That’s a no-brainer.

Dry scroll pumps like the ones we use on our detectors work because of a noxiously complex set of seals.  Blow the seals, the pump won’t suck air.  This pump had one horribly blown seal.  Once we opened it up and looked inside, we saw it wasn’t the only seal that had gone.  It had simply reached end of life.

Some years ago several of our engineers were trained in how to rebuild the dry scroll pumps we use.  They came home with rebuild kits, a field service tool kit, books, and knowledge.  Now we rebuild our own.  This was my first time rebuilding a pump, so I worked with one of the engineers who’d been trained.  The rebuild was fairly straightforward, but there were a number of tricks he showed me to make the job easier.  When a dry scroll pump is rebuilt, all the seals are replaced, but so are all the bearings and o-rings.  The bearings are all pressed in, so getting them out is a bit of a trick.  The manual calls for an oven that can swallow all the parts and go up to 350F.  The manual called for us to heat parts, remove bearings, cool parts, heat parts, install bearings, cool parts, etc.  The real trick, I was told, was to line up as much of the work as possible, heat them all at once, remove the bearings and wipe them clean, then install the new bearings before the parts were ever cooled.  One cycle, all done.  In this way the two-day job of rebuilding the pump can be shortened to a single day.

Of course things never work out the way you expect.  In the middle of the rebuild we had a failure on one of our cooling systems, and the guy who knew what he was doing was called away to deal with it.  >gulp<

So the two-day job of rebuilding the pump was shortened to one, then stretched back out into two.  This afternoon I installed the new seals, put in the new cam shafts, closed everything up, and crossed my fingers.  After a rebuild the pump needs to run for 24 hours to run the new seals in, so I put a thermocouple gauge on the input port, closed it up, and turned it on.  I was overjoyed to hear the air getting sucked out of the small volume of the gauge, and then to see things start to drop.  I won’t hit 10 millitorr until tomorrow, but by the time I walked out of the room it was down to 20 millitorr.  Not a bad start.

Sometimes it’s good when things suck.

– Tom


4 Responses to “When Things Suck: It’s Good?”

  1. Greg said

    Hey Tom,
    For years I was under the impression that rebuilding a scroll vac pump was a very-precise procedure, best left to ‘the professionals’. But when I finally found a (Varian) rebuild procedure I knew that I should be able to easily handle the job. We ordered the full maintenance kit from the manufacturer which included a hard copy of the procedure. Every step was followed as described. But now with the rebuild complete I can’t get the unit to pump below around 100 mTorr. I have re-opened the pump a number of times and ever-so-slightly tweaked the position of the orbiting plate lock nut, both tighter and looser, to no avail (the book directs to tighten the nut 0.007″ more than the measurement taken when no tip seals are in place). Obviously there are a million details that I can’t possibly include here, but I thought you might have some general input that might help me here ?

    • Tom Benedict said

      If you’d written this six months ago, I would’ve said, “I’ve never seen that!” But now? Yeah, I’ve seen that.

      A while back we had a pump fail pretty catastrophically. One of the cam rollers snapped in two places. The orbiting plate was no longer constrained, and it hit hard and locked. The pump was DEAD. So we rebuilt it.

      The rebuild kits come in two flavors: tip seal kits and full rebuild kits. Up until now we’ve been doing full rebuilds every single time. This includes new cam rollers, new bearings, new o-rings, tip seals, everything. Turns out you only need to use this kind of kit if you lose a bearing or a cam roller like we did. Otherwise the tip seal kit is sufficient. That didn’t really impact this rebuild since we’d crashed this pump so hard. But it was good to learn. Now we’re stocking tip kits as well as full rebuild kits.

      When we rebuilt the pump, though, we couldn’t get it to pump below 100 mTorr. We rebuilt the thing several times trying to figure out what was going on. Finally we pulled all the bearings back out and compared them against the old ones. One of the main bearings was different. In the end we re-installed one of the old bearings (always save your old bearings!) and got the pump back up and running. But now we’re wondering if the rebuild kits we’re getting are relying on some newer version of one of the plates with a deeper bore.

      In any case, because the orbiting plate fits between the two fixed plates, that 0.007″ measurement really is what you want to do. That makes sure the orbiting plate is balanced between the two fixed plates with equal space on both sides.

      One question for you: When you first rebuild one of these, they don’t seat well. They can’t. The tip seals haven’t seated. So for the first 24 hours they don’t pump all that hard. But after 24 hours they should be pulling pretty well. When you say you’re unable to get it to pump below 100 mTorr, is that after a 24 hour run-in? Or is that right after powering it on?

      If you don’t mind describing what you’re seeing in some more detail, I’d be happy to try to help you troubleshoot it. The guy I rebuilt our busted pump with is working with me today, so I can get his feedback as well.


      P.S. We managed to get our pump running again. I’m pretty confident you’ll get yours running, too.

      • Greg K said

        Thanks for the input Tom
        When I first rebuilt the pump I did let it run for the twenty four hour break-in period. But it was loud and hot – not good. I foolishly hoped that it was just a matter of the break-in; such was not the case. Opened it up and eventually realized that I had mis-read the depth micrometer and had mis-tightened the orbiting plate lock nut. So I started over ; removed the orbiting plate and removed the ‘inner’ tip seals. Reinstalled the orbiting plate with lock nut, re-measured with the depth micrometer. Then installed new tip seals and again installed the orbiting plate. That is pretty-much where I am now… I have been picking-away (many other duties to cover) at trying slightly-different torquing of the lock nut to no avail. I have removed the orbiting plate two or three times since and re-measured without tip seals to make sure I am not missing something…
        Your mention of questionable dimensions of the new bearings intrigues me. But I fear that our old bearings are pretty-well shot – I might be able to re-install them to see if pump pressure improves, but I will not be able to leave them in place.
        At this point I am entertaining the idea of completely disassembling the pump and starting over to see if I did-in-fact somehow miss something.

      • Tom Benedict said

        Aaaah! I remembered what happened with that funky pump:

        The dimensions of the inner race for one of the bearings was slightly different. It was small, but enough to put the ball cage for that bearing in contact with one of the spacers in that stack. This resulted in a melted ball cage, which more or less torched that bearing.

        The only indicator we had of this (aside from putting the new and old bearings side-by-side and measuring everything on them) was that the depth of the orbiting plate lock nut couldn’t go back to what we had measured prior to disassembly. The depths on all of our pumps has been around 0.0675″. This one came out in the mid 0.070’s (I don’t have the log book with me). Once we swapped back to the old bearing, everything went right back to where it was supposed to.

        When you say your old bearings are shot, is that all of them? Or just some of them? The needle bearings for the cams tend to go, but I’ve got a stack of older bearings from the main shafts of our pumps, and they tend to survive a little better.

        If you do wind up disassembling your pump, it’s worth checking the bearings side-by-side with the older ones to see if any of them have changed. Also check for any heat damage like melted seals or a melted ball cage. I’d be curious to hear if you’re having a similar issue to the one we’re having.

        Unfortunately we had to get our problem pump back into service ASAP, so we opted for expedient rather than answer all of our questions. So we’re trying to leave a reasonable window of time when we rebuild the next one to take a fresh look at all this and maybe come up with some better solutions (order different bearings, ask Varian for a new spacer, etc.)


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: