BPuhl’s Blog

A little bit of everything without actually being much of anything

Archive for December, 2007

Happy New Year!

Posted by BPuhl on December 31, 2007



Posted in Babbling and Blabbering, Randomness | Leave a Comment »

Where clouds are kept…

Posted by BPuhl on December 31, 2007

“Look daddy, the sky is opening up to let the clouds out!”

That’s what Anika (my 3yo) said this afternoon, on a beautifully clear and sunny (if a little cold) Seattle day as she looked up at the contrail left by a high flying jet.

Posted in Anika | Leave a Comment »

Replication Version Number for your KrbTGT account password?

Posted by BPuhl on December 19, 2007

When  we flipped our REDMOND domain to Server 2008 domain functional mode, we experienced an issue with some of our application servers suddenly failing to authenticate.  We found that this was because Kerberos authentication was failing, as the domain KerbTGT password was changed during the domain mode switch.  In fact, if you look at our domain KrbTGT account, you see it as the following replication metadata:

C:\>repadmin /showobjmeta red-dc-11 “CN=krbtgt (Key Distribution Center Service Account),CN=Users,DC=redmond,DC=corp,DC=microsoft,DC=com”

Loc.USN                          Originating DC   Org.USN  Org.Time/Date        Ver Attribute
=======                          =============== ========= =============        === =========

65585069                      NA-WA-RED\RED-DC-10 151889264 2007-11-01 16:06:02    4 pwdLastSet

The “version 4” indicates that our KrbTGT password has actually been changed a few times in the past 8 years.

Naturally, we all thought this was “bad” (outages usually are), and the dev’s weren’t quite sure why the system didn’t handle this much more gracefully.  Although it’s unlikely that anyone would actually change their KrbTGT password, the system is designed to handle it.

We suspected a bug, so collected a bunch of data, and tried to repro this in a lab.  Unfortunately, we couldn’t ever repro the outages, so we went to the next step and worked with the developers to get some instrumentation to use when we did our next production domain.  This time we chose an Exchange resource domain to move to 2008 DFM, and everything went smoothly.

Even though there aren’t any bugs to fix, the PG has agreed to include documentation indicating that the krbtgt password gets changed when you flip to domain functional mode.

Just another one of those random tidbits of information which is good to have in your back pocket.

Posted in Active Directory | Leave a Comment »

Change management and dogfood

Posted by BPuhl on December 18, 2007

Talking to some folks at IT Forum in Barcelona recently, and discussing some of the “fun” that we’ve had with dogfooding Server 2008.  Most of the discussion was about features – RODC, Fine Grained Password, Server Core, etc…  but in any good dogfooding discussion I always get asked about some of the challenges.

Sure, we’ve hit our share of issues, but that’s really what we do this whole dogfooding thing for.  I was talking about a bug we had recently hit which caused several internal applications to feel a bit under the weather, when the customer asked me why we didn’t do change control.


At first, I wanted to say that of course, we do change control.  Because it’s true, on a good day, when the planets are aligned and gods are in the heavens, we can start off in a lab with testing, understand every aspect of every change which is occurring in the system, and reconcile that to all downstream applications, communicate those changes broadly, blah, blah, blah…  excuse me, I think I had a MOFball caught in my throat.

Our reality though, is that we’re often bound to aggressive deployment schedules with an undocumented operating system upgrade incorporating changes across dozens of teams…some of who will leverage our experience in crafting the documentation which doesn’t exist at deployment time.

I’m a fan of change management, but I even more firmly believe that IT is not in place to support itself, but rather to service the needs of the business.  If the business requirements can’t support a high (or even mid) level process overhead for testing, communication, etc… then that’s when the negotiations about availability and impact have to start.

Posted in Active Directory, Random Tecnical Stuff | Leave a Comment »

Yes, it’s my meeting…no, I don’t really care if you accept

Posted by BPuhl on December 15, 2007

Have you ever scheduled a meeting with a large group of people, and then get flooded with all of the accepts, tentatives, and declines?

It was recently pointed out to me, that in Office 2007, when you schedule a meeting, you can opt out of these responses – BRILLIANT!


Just uncheck the box.

Welcome to the 21st century.  🙂

Posted in 21st Century | Leave a Comment »

Badgers, Badgers, Badgers…

Posted by BPuhl on December 14, 2007

It’s taken a while, but I think we’ve made the final turn towards full enforcement of smart cards for logon for Microsoft Employee’s.  We’ve had a pretty successful pilot for the past few months.

The biggest problem with using your badge to log on though, is that people are always forgetting their badges at home.  This morning, I saw that the receptionists are now handing out these little badger decals to people who need a temp badge to get into the building (I may have added an extra letter):


Of course, seeing the badger on the badge(r), reminded me of badgers, badgers, badgers…MUSHROOM MUSHROOM…

And then Dan threw out the clip from UHF:

So I guess apparently, if you’re looking to enforce smartcards for employees…you really DO need some stinkin’ badgers!

Posted in Digital Identity, Identity and Access, Randomness | 1 Comment »

Welcome to the 21st Century

Posted by BPuhl on December 14, 2007

There are a ton of new features in every piece of software, most of which go unused.  So “Welcome to the 21st Century”, Brian’s stories about how he’s discovered things that everybody else probably already knew. 

The bar for a 21st century post, is when I go “well, duh” when I discover some infinitely easier way to do something that i’ve been doing the hard way for a while.

For today, I would like to show you my latest discovery which is used to manage “time”.  Sure, we all know about the little clock down in the system tray, but I’ve been travelling a lot this year and so having multiple clocks for different time zones has been key.  In fact, until recently, I sacrificed a large chunk of my Vista sidebar to tracking “time”


The other night, I “discovered” that there is an “Additional Clocks” tab when you go into the date/time settings page – BRILLIANT!  (yeah, that was my “duh” moment).

After reclaiming my sidebar space, I can now mouse-over and to get a quick peek


or click and get


Posted in 21st Century | 1 Comment »

Last Widget Phenomenon

Posted by BPuhl on December 14, 2007

Since we dogfood Active Directory quite a bit, it probably makes sense that we’re pretty careful when it comes to introducing new changes in the environment.  But even though we do our “due diligence” before and during deployments, we are after all dogfooding – so more often than not we’re tied to product release schedules and the (very) aggressive deployment requirements.  So sometimes in the name of agility, we’ll keep moving forward when some more conservative shops might slow down.  Then again, that’s actually half the fun of dogfooding!

In the past, we learned that it was prudent to take extra precautions when we were making an initial change.  For example, the first Windows 2000 domain controller in the NT4 domain – PDC piling on anybody? – or some of the application commutability issues that came with the early beta Windows Server 2003 deployments.

Recently, we started to notice a trend with our Server 2008 deployments.  We’ve dubbed this the “last widget phenomenon”, because we’ve found that paying extra notice to the “last” of something in the environment needs the extra attention as well.  For example, during an early beta deployment of Windows Server 2008, we had upgraded 4 out the 5 DC’s in the empty root domain of our CORP forests (root + 8 child domains).  The time was right, and we wanted to run the EMPTY root domain on all Longhorn server.  No problem right?  When the first 4 upgrades went great, then the 5th should be a piece of cake.  Unfortunately, we didn’t consider the last widget, and when the last Server 2003 DC was demoted we exposed a (previously known) performance bug in Kerberos which was fixed in a future build.  Since this was the “empty root”, all transitive authentication between domains in the forest failed while the 2008 DC’s skyrocketed to 100% CPU utilization.  So NOW what do you do?  You want to re-promote the 2003 DC, but all of it’s potential replication partners are burning bits as fast as they can.  Not to mention, even on a good day it’s going to take a couple hours to do the DCPromo (even with IFM).

So what should we have done?  Well – it’s much easier to power-up a DC that was just turned off rather than re-promote it in a hurry when the barn is burning.

Of course, if we do our jobs right, then you’ll never have to experience these bugs we’ll hit them for you first, and the PG’s will fix them).  But as a general administrative mindset, it’s useful to remember that the last widget can be as important as the first.  It’s pretty easy to get a little complacent, when you over prepared for #1 and nothing happened, and then nothing continued to happen for numbers 2-99.  But when it’s time to do the last one, #100 – be on the lookout.

Is it just me, or has anyone else noticed that the more prepared you are to respond to a situation, the less likely you are to ever need to?

Posted in Active Directory, ADFS, Identity and Access, Random Tecnical Stuff | Leave a Comment »

Things that keep me up at night

Posted by BPuhl on December 13, 2007

Our Premiere Field Engineering team in the UK put on a customer event and I had the opportunity to go over and speak one day with a group of about 25 customers about Windows Server 2008 Active Directory.  Not uncommon in this format, I was asked the “What keeps you up at night?” question.  Thinking about it a little bit, I realized that there weren’t all that many things that were really bothering me about Windows Server 2008 (either a very good sign, or a very bad one).

Not specific to Server 2008, here are the way that I usually classify things that are going to keep me up at night:

  • Custom Tools – Ugh…it seems like every time we get something new, exciting, or innovative – it will take us 80% of the way to what we need and then we have to come up with some stupid custom code that costs $3 million, takes 6 months to build, and can never be reused for anything else.  Even worse, is when customers ask us “How does MS IT do that…” and we have to reply with “custom tool”. 
  • Operational Complexity – There are a lot of tools, features, etc… that seem great, but are darn near impossible to manage (huh…you’ve heard that one before too huh?)  We seem to run into this a lot, and in the most extreme cases results in a custom tool (see above).  But operational complexity causes errors, and for an infrastructure team, usually means wide scale outages.  Let’s try to avoid those, shall we?
  • The next version – You would think that with everything going on, we wouldn’t always seem to be waiting on “the next version” of the software to get what we need.  But we’re customers, and time neither stops nor accelerates for anyone.

So ok, then what is keeping me up at night?  With Server 2008, there are a couple of things:

1)  How are we going to manage the PRP for RODC’s at scale?  There’s another post coming about this one

2)  What’s the final deal with backups, and when are we going to seriously get cycles to do the BC/DR work that we need – and what is that going to look like?.  After all, we’ve got a pretty decent DR plan for 2003 already, so we’re not starting from scratch…but our largest domain is in  Server 2008 domain mode now.

There are a lot of other features that I could talk about.  But seriously, as cool as Admin Role Seperation, snapshot viewer, and Fine Grained Password Policy are (just to name a few), it’s not like these are changes to the way we do things today.  On the other hand, RODC’s are an important part of our server landscape, and backups…well…yup.

Posted in Active Directory | Leave a Comment »

Perspective (RODC’s and Full DC’s)

Posted by BPuhl on December 13, 2007

It’s always good to keep things in perspective.  This is especially true when you start troubleshooting replication issues.  One of the engineers came by, curious because he was looking in AD Sites and Services, and there were no connection objects for the RODC’s. 

It turned out, that his MMC snap-in was pointed at the full DC.  Since connection objects are created by the KCC locally…and an RODC cannot replicate anything back to the full DC… then the “view” of the connection objects is only available when you point the MMC at the RODC itself.  Sure enough, when he did that, there they were.

Perspective is going to become hugely important to remember as RODC’s are deployed with Windows Server 2008.  For the past 7 or 8 years, we’ve sort of taken for granted that the directory was “loosely consistent”, and in all but ugliest troubleshooting cases we were able to pop open a snap-in and look at the data “in the domain” or “in the forest”.  This isn’t true anymore, and for some things, like replication data, it’s going to be critical to make sure that you get the right “view” of the enterprise but peering into the correct window (err…ummm….Windows)

Posted in Active Directory | Leave a Comment »