Getting Single Label Name Resolution on MDM Enrolled Phones

If you’re using Mobile Device Manager, it’s likely that you’ll want to have your phones be able to resolve single label names.  If you do, then there are 2 basic options that you have, first you can make WINS servers accessible to the devices, or alternatively, you can configure the phones to append a DNS suffix to single label name queries.  In fact, as with the full OS’s, you can actually do both.

There is a Technet article located here which talks about some of this, and gives the following ADM template to be used to apply 2 registry settings to the phones – IMPORTANT DETAIL: AS OF 8/21/2008, ONE OF THE REG KEYS BELOW IS INCORRECT – SO KEEP READING!!!!

CLASS MACHINE
CATEGORY “Windows Mobile Settings”
       CATEGORY “Contoso DNS Settings”
            POLICY “Name Resolution Ordering”
                  KEYNAME “SOFTWARE\Policies\Microsoft\Windows Mobile Settings\Registry\HKLM\Comm\AFD”
                          VALUENAME “NameResolutionordering”
                          VALUEON NUMERIC 4
                          VALUEOFF NUMERIC 1
            END POLICY
            POLICY “DNS suffix”
                  KEYNAME “SOFTWARE\Policies\Microsoft\Windows Mobile Settings\Registry\HKLM\Comm\MSEC\IPSECVPNNIC1”
                  PART “Enter the dns suffix required” EDITTEXT REQUIRED
                         VALUENAME “Domain”
                        DEFAULT “dns.corp.contoso.com”
                         MAXLEN 32
                  END PART
            END POLICY
       END CATEGORY
END CATEGORY

The net effect of setting these 2 registry keys is *supposed* to be, that they change the single label name resolution behavior to WINS first, and then to DNS with the suffix appended.

But in my MDM environment, I don’t actually have WINS servers available.  So this article doesn’t fully apply to me, and possibly you.  The MSIT MDM deployment exclusively uses DNS, so the first thing that was important was to find that the NameResolutionOrdering registry key has the following settings:

The search order for name queries when set is:
          Default (or 1) –  DNS then WINS
          Value 4 – WINS then DNS

The DNS queries will append any suffix as configured

So with this handy bit of information in hand, the first thing I did was chop out the section of the ADM template that set that registry key.  In my case, default was good enough for me.  Then I applied the GPO linked to the OU which our devices are in, recalculated the policy manually (I’m impatient) by using the update-MobilePolicyCalculation cmdlet on the Device Management server, and reconnected my device (using the Connect Now utility from the MDM resource kit client tools)

At this point, everything was working great – EXCEPT for that fact that it didn’t work.

When I would sniff the traffic on the DNS server, the queries all came in as single label names and did not have the suffix appended.  After much thrashing about checking registry keys and investigating the client, I finally dragged one of the Program Managers from the product team over to my office to help.  That was about the time that he let me in on the secret, that there is a “documentation bug” filed on that page, because the registry key for the setting the suffix is wrong.  Oh great.

Rinse repeat the whole thing with the correct registry key, and here is what the ADM template which actually worked finally looked like:

CLASS MACHINE
CATEGORY “Windows Mobile Settings”
      CATEGORY “MSIT DNS Settings”
           POLICY “DNS suffix”
                 KEYNAME “SOFTWARE\Policies\Microsoft\Windows Mobile Settings\Registry\HKLM\Comm\MSEC\IPSECVPNVNIC1\Parms\TcpIp”
                 PART “Enter the dns suffix required” EDITTEXT REQUIRED
                        VALUENAME “Domain”
                        DEFAULT “mdm.microsoft.com”
                        MAXLEN 32
                 END PART           
            END POLICY
      END CATEGORY
END CATEGORY

And that ladies and gentleman, is all that’s required to get single label name resolution on a mobile device, by appending a DNS suffix.

Politics, Pieces, and Parts…

I suppose I should have expected this, but I didn’t.  In my first post on MDM, when I said:

Wow, it’s been a remarkably long road to getting System Center Mobile Device Manager deployed here internally at Microsoft – and to be honest, we really don’t have it released to the general population yet  – but since we’ve got the infrastructure stabilized, and enough pilot users to be “close enough”, then it’s probably a good time to start a series on “How Microsoft IT designed and deployed System Center Mobile Device Manager.”

No – I didn’t actually mean that MDM as a system was hard to deploy.  Thanks to the commenters, pingbackers, other otherers, that decided to interpret this as “Wow, this system must be really hard to deploy”.  Come on guys, gimme a break – it’s just servers, we’re not talking rocket science here.

One thing which occurred to me though, is that I wonder how many of these people actually work in an IT department?  You know, limited budget, lots of politics, too many projects and not enough resources…yeah, the life that 98% of the IT guys in the world live in.

Just to be clear – No, quite frankly, MDM wasn’t all that hard to deploy.  Especially for those of us in MSIT, we actually had the opportunity to do something which we haven’t done in a long time – Rather than dogfooding the initial deployment of Yona (the codename that became SCMDM), politics, resources, and priorities intervened.  So we started our deployment just prior to MDM releasing to the public.  This is the rare case, where we actually had the documentation and whitepapers to use to deploy from.  That was actually refreshing!  So I’ll stop ranting here, and say that if you’re looking to deploy MDM, take a look at the Architecture, Deployment, and Operations guides – and you’ll be rock’n and rollin’.  Then you just need to deal with your own internal organizational issues 🙂

That was politics, so here are the pieces and parts of the MDM service:

Gateway Server – This is the one which everyone calls the “MDM Server”, though it’s really only one component of the system.  The MDM gateway server is the box which you deploy on the edge of your network.  Mobile devices connect to the Internet through their wireless service provider, and establish a VPN-like connection to this machine.  The idea is that this server would sit on the edge next to your VPN or other remote access infrastructure, and from here, devices can access your internal network.

Device Management Server – This is the box which does all of the work.  The MDM DM (sorry…had to…I love that acronym), runs WSUS and is used to manage the GW servers.  Devices connect to the the DM (via the gateway) to do things like software inventory, patching, application publishing, and policy application.  The DM server will also interpret the GPO’s which apply to a device from AD, and translate them into phone-speak for application.

SQL Server – The DM needs one of these.  We had multiple DM’s, which use the same SQL server that we set up on a SAN (not required).  We also have a separate server to use as the reporting server.

Enrollment Server – Web portal used for tethered enrollment of phones into MDM.  Think of this as “domain joining” the machine.  This server touches AD and the CA.

AD and CA – There is a domain prep and some certificate templates which need to be installed.  More on these later.

Ooops… (alt title: How I screwed up MDM) (MDM pt. 2)

This should be fairly quick, (it doesn’t take long to fall on a sword) but I thought I would just come out with one of the first “big mistakes” that we made with our MDM deployment.  And by “we”, I really mean “me”…

So – let me ask you this.  Just how important do you think it is, that when someone receives an e-mail on their phone, and that e-mail contains a link…for example, to a doc on a Sharepoint site… and that Sharepoint site is accessible via MDM….  Just how important do you think it is, that they should be able to click on the link, and have that doc open?

Well my big mistake was to think that since providing that functionality AND getting user authentication in place would be insanely complicated and hard….  that it wasn’t THAT big of deal.  To me (being the AD/ADFS guy, single sign-on and user authentication were much more important.)  The trade off in our deployment which we made, was the assumption that users could hit the Sharepoint site, and drill down to the doc that they were looking for.

Holy jeepers Batman, was I wrong about that!  (this is the part, where you the reader, can go:  “DUH!”)

Turns out, that if you don’t provide this functionality, then river and seas boil, 40 years of darkness, earthquakes, volcanoes, dead rising from the grave, human sacrifice, dogs and cats living together, mass hysteria![1] You have no idea what kind of crap I have endured over this!!!!

Let me be clear – as of today, right now, no – we are NOT providing this functionality.  The next few (several?) posts on this topic, will be about our existing design, which does not allow for this.

But please, make no mistake – there are forces conspiring, virtual teams have been formed, e-mails have been launched, meetings are called, and engineers are standing at the ready – solely to correct this most egregious error!  (we just need to figure out how, while maintaining the rest of the integrity of the system)

 

[1] You soooo know where this came from…

Mobile Device Manager

Wow, it’s been a remarkably long road to getting System Center Mobile Device Manager deployed here internally at Microsoft – and to be honest, we really don’t have it released to the general population yet  – but since we’ve got the infrastructure stabilized, and enough pilot users to be “close enough”, then it’s probably a good time to start a series on “How Microsoft IT designed and deployed System Center Mobile Device Manager.”  Of course, that’s really long, so let’s just call it part 1 for now.

First off, we’re still actually in a little bit of flux in terms of the service offering and how everything is going to work.  There were some definite design details which were completely missed (by me), that we’re going to need to go back and fix.  So the first disclaimer is:

This series is based on my experience up to the date of this post.  I’ve found that I’m really bad at coming back and correcting blog posts when things change, but I’ll continue to do my best.  by the time you read this, it may or may not describe what is actually in production at Microsoft anymore.

(2nd disclaimer:  These are in the order I’m thinking about them, not necessarily the order in which they occurred or in which might make sense to anyone else)

Ok, so one of the hardest things that we had to figure out for our SCMDM deployment, was what we were going to do about user authentication.  This may or may not seem obvious, but it turns out to be a ridiculously complicated conversation with fairly passionate arguments on each side.  The basic question, was whether or not a local PIN and a strong PIN policy, is sufficient for “user authentication” in our environment.  Here are some of the things we thought about:

• If <configurable X> number of bad PIN’s are entered – the device explodes (well, ok, it wipes and all sensitive data is erased)
• PIN policy can be specified and pushed down to the device via MDM (or OWA), so you can make it whatever you like
• The PIN’s are local to the device
• There is no enforcement to periodically “change” the PIN’s
• There is an implicit assumption that since a person is able to use a device, they entered the right PIN, and must be the user
• The person entering the PIN has “physical access” to the device:  Law #3 of security – physical access
• When talking about “2 factor auth”, one of the factors cannot be the device/laptop/Computer itself – separate from the device, there would need to be a “something you have” and a “something you know”
• How does this extend, if you wanted to use the phone itself as the “something you have” to log onto a laptop?  (outside the scope of this deployment, but a fun conversation anyway)
• How do you manage “deprovisioning” of a device, if a user is no longer employed
• It’s difficult to know which user is associated with which device
• Remote Wipe functionality in MDM requires that the VPN connection be established
• Because Microsoft does not provide the devices, they are personal devices, therefore we cannot restrict the ability for users to turn off the VPN connection
• Active Directory user accounts are the enterprise acceptable form of user authentication for applications
• User credentials are already stored on the device anyway – since everyone uses ActiveSynch for Exchange/Outlook access

After batting these things around quite a bit, we decided that the “wild west” culture of our environment, where we have more of a laissez-faire culture with respect to users (while at the same time being responsible for protecting our data), required that we leverage our existing account provisioning and deprovisioning processes to enforce security.

So although the “standard” MDM deployment architecture allows for authenticated mobile devices to have unrestricted access to the corporate network, we needed to deploy in such a way that we could enforce user authentication, and only allow access to those resources which MS IT Security approved for mobile devices.  This basically translates to High Business Impact (HBI) classified data being excluded, and everything else was fair game.

 

 

Over the next few posts, we’ll talk about “how” we enforced this, and what this meant to our deployment…

What do you do, when you don’t know what to do?

[Edited 3/8]  Talk about a reminder!  I started to write this post back in February.  Like most things, it got tossed to the back burner when everything else in life started flying sideways.  I’ve started to develop the bad habit, that if I can’t throw a blog post out in the 2 minutes that I’m thinking of something, then I don’t seem to be getting back to them.  Not good.

While out to dinner with several friends at DEC the other night, the conversation turned to, “What do you do when you don’t know what to do?”  What was really funny, is that I thought I had already blogged about it…well…without further ado, here’s a fairly legitimate blog post

[Original from 2/11]

So what do you do when you don’t know what to do?  This happened recently, when one of the vendors that is helping us with Longhorn deployments comes running down the hallway because he thought he had accidentally seized the RID master in the REDMOND domain and then brought the original back online.  Of course, it’s 9am, there’s hardly anybody around, but fortunately he was able to find someone who suggested that he just turn off the DC until the DC guys came in.  Back down the hallway and that’s what he did.

Make whatever comments you like about the impacting of seizing the RID master, knowledge of what the role does, it’s impact, etc…  That’s not the point of this post.  The point of this post is, that few people will ever have 100% understanding of every aspect of the system, everything that could possibly change in it, and the downstream effect that every possible change will have on all dependent systems.  Without that impossible depth and breadth of knowledge – then sooner or later SOMETHING is going to happen that you don’t know what to do about.

Back to the AD team, it didn’t take long to realize that few people had actually considered what they would do if, say for example, they thought they accidentally seized the RID master, or they accidentally ran a script that performed some bulk change (or delete), or accidentally clicked “ok” when they should have hit cancel.  The problem is that if you’ve never thought about it, then you don’t know what’s in your toolbox.  And how can you fix something, if you don’t have the tools?

The basic idea:  Slow or stop the bleeding, until you can get help.

–  If it’s a server issue and you aren’t sure what happened – Turn it off! 

–  If you think some change happened in AD that could replicate around the world – REPADMIN /OPTIONS * +DISABLE_OUTBOUND_REPLICATION  It’s the big red button that will stop replication.

These are just a couple of things that you can do, but the point is, nobody is going to get mad when you stop forest wide replication or for turning off a server, if it saves them from needing to do a forest recovery.  Damage control is about accepting that you’re going to have an impact, so let’s just limit it.

And one last note – When you do finally get a hold of the cavalry, and they come rushing in to help – Tell them what happened!  Yes, it might be embarrassing to admit that you made a mistake…or even to find out that you reacted quickly and forcefully to something that wasn’t a mistake at all…but I haven’t heard of people being fired over embarrassment.  I have heard of jobs being lost when people tried to hide their actions.

 

[From 3/8] – Talking about this at dinner, Stuart told a story that he had read in a book about the CDC (I wish I could remember the name of the book!), and the people who work in labs with ultra-dangerous viruses.  What do they do if they accidentally nick their spacesuit and their finger, exposing themselves to deadly viruses?  They have 10-20 seconds to cut off their finger, otherwise they are as good as dead.  Kind of makes me glad that I only work on computer systems [1]

 

[1]  My memory of this story is a little bit clouded by the wine.  Any inaccuracies are mine, but it still makes for a good story!

Geo-Load Balancing

I’ve been spinning a lot of cycles lately on this SCMDM deployment.  Not going to go into it in detail here, but the short version of it would be to think about a near-constant VPN connection between a Windows Mobile phone, and your corporate network. 

The interesting data point that I found today was simply related to the way that our users experience the Internet.  The random thought had occurred to me, is that because of the way that SCMDM works, nearly all of a users web traffic is going to go from their phone, to the carrier, across one of the intertubes to the MDM gateway, then back out across the intertube to the desired website.  (Yeah, humor me here about all the interesting issues with THAT architecture).

Since all of the users web traffic flows from the gateway, what’s the behavior like when accessing sites that are regionally load balanced?  For example, if I’m in France, and put google.com into a browser, then Akamai redirects my traffic to my local google instance, google.fr.  If I’m in Germany, I go to google.de, etc…  You get the point, this redirection works all over the world (if a site wants it to). 

So now, if I am an employee in Germany, and take my SCMDM enrolled phone, and go to google.com, I’m redirected to….Yup, you guessed it:  google.ie   – wait huh?  why would they go to Ireland?  Well, for my employees, their phones would connect to the MDM gateway servers in our Dublin datacenter, so their internet egress point would be located in Dublin, and Akamai would direct them to their “local” instance – google.ie

Hmmm, so if I’m correct in understanding how all this behavior works, then that must also mean that the same weird redirection behavior occurs today for Microsoft users surfing the internet from the corporate network.  Regardless of where in the world they are located, all internet egress occurs from one of the few proxy arrays located in a data center.  So to validate this, I walked the 10 feet down the hallway, to where one of the ISA engineers sits.  I explained to JW what I thought was going to happen to mobile phones, and asked how they were managing it with the ISA servers – answer:  They aren’t.

Interesting data point for the day:  In 5+ years of managing the ISA servers, he’s NEVER heard of a single complaint from users, about the behavior of being redirected to the wrong localized version of a website because of where the proxies are located.

This is all amazing to me – I can generate 100 help desk calls [1] because a subnet shows up with a bad building label, yet he can redirect users around the world to get content and never hears a peep. 

 

 

 

[1]  A useful bit of perspective on Microsoft employees.  We publish quite a bit of information in Active Directory, and one thing we do is printer publishing.  So a user in a building can locate a printer that’s near them – accomplished through mapping and maintaining sites and subnets in AD.  If we have an incorrect subnet, we can slam the help desk with calls from users, declaring that “I’m in building 42 and trying to find a printer, and my laptop says I’m in building 34!!!!”