This is a text-only version of the following page on https://raymii.org:
---
Title       : 	Linux on Microsoft Azure? Disable this built-in root-access backdoor (wa-linux-agent)
Author      : 	Remy van Elst
Date        : 	22-08-2018
URL         : 	https://raymii.org/s/blog/Linux_on_Microsoft_Azure_Disable_this_built_in_root_access_backdoor.html
Format      : 	Markdown/HTML
---



![][1]

Are you running Linux on Microsoft Azure? Then by default anyone with access to
your Azure portal can run commands as root in your VM, reset SSH keys, user
passwords and SSH configuration. This article explains what the backdoor is,
what it is meant to do, how it can be disabled and removed and what the
implications are.

<p class="ad"> <b>Recently I removed all Google Ads from this site due to their invasive tracking, as well as Google Analytics. Please, if you found this content useful, consider a small donation using any of the options below:</b><br><br> <a href="https://leafnode.nl">I'm developing an open source monitoring app called  Leaf Node Monitoring, for windows, linux & android. Go check it out!</a><br><br> <a href="https://github.com/sponsors/RaymiiOrg/">Consider sponsoring me on Github. It means the world to me if you show your appreciation and you'll help pay the server costs.</a><br><br> <a href="https://www.digitalocean.com/?refcode=7435ae6b8212">You can also sponsor me by getting a Digital Ocean VPS. With this referral link you'll get $200 credit for 60 days. Spend $25 after your credit expires and I'll get $25!</a><br><br> </p>


Azure is Microsoft's Cloud platform. It provides virtual machines and related
datacenter virtualization, next to software as a service (hosted stuff like
databases). I currently work on a project where an hosted application platform
is built on Azure using Linux (Ubuntu, CentOS), and recently found out about
this backdoor. Or, useful feature, since it's not just a blatant and deliberate
backdoor

I have no idea how the situation on Windows on Azure is, and have not researched
this.

### The backdoor

Azure, as any other good Cloud provider, has images which you use to create new
VM's (sometimes calles VPSes, instances, droplets). It speeds up the rollout of
new VM's, because mounting an ISO and manually installing a VM is time-
consuming. In that image, they often change stuff to make it run better on their
cloud. When I created images for an OpenStack provider, we pre-installed `cloud-
init` and `haveged` for example. Microsoft does that as well, they install their
own agent, the `wa-linux-agent`. Later on in this article I explain what this
agent is meant to do. It is not just a backdoor, it provides actual useful
features. One of those features is root access outside of the VM.

In the [Azure portal][3] one can login and configure their Azure cloud. For this
project I use Ansible and Terraform, so I don't have to regularly interact with
this webpage (which is kind of slow to work with). However, I also have a
personal Azure account for playing around, which is where I found this feature.

Via the Azure portal one can execute commands as root inside a VM and change SSH
keys, user passwords and SSH configuration.

I'll let the pictures speak for themself:

#### Remote command execution as root

![][4]

#### Remote SSH key injection, user password reset and SSH configuration reset

![][4]

You can read more on the backdoor feature [here][5] ([mirror][6]) and [here][7] ([mirror][8]), Microsoft is very transparent about this, quoting from the pages:

> Scripts run by default as elevated user on Linux

>

> Think of the Azure VMAccess extension as that KVM switch that allows you to
access the console to reset access to Linux or perform disk level maintenance.

Microsoft puts this feature in a positive way so that it looks less like a
backdoor:

    
    
    The disk on your Linux VM is showing errors. You somehow reset the root password for your Linux VM or accidentally deleted your SSH private key. If that happened back in the days of the datacenter, you would need to drive there and then open the KVM to get at the server console. Think of the Azure VMAccess extension as that KVM switch that allows you to access the console to reset access to Linux or perform disk level maintenance.
    
    This article shows you how to use the Azure VMAccess Extension to check or repair a disk, reset user access, manage administrative user accounts, or update the SSH configuration on Linux when they are running as Azure Resource Manager virtual machines. 
    

For command execution as well:

    
    
    This capability is useful in all scenarios where you want to run a script within a virtual machines, and is one of the only ways to troubleshoot and remediate a virtual machine that doesn't have the RDP or SSH port open due to improper network or administrative user configuration.
    

I believe that this is a useful feature. I however also am of the opinion that
this is a backdoor. It is not made obvious that this agent is running in your VM
or that it provides root access. Only after looking (in the output of `ps aux`
and in the Microsoft docs) I found out what it is and what risks are connected
to it. Took me a good hour.

A tickbox when deploying VM's (or API flag) to disable this agent would be nice.

### Impact and implications

Anybody with access to VM's in the Azure portal is able to execute commands as
root inside any VM they have access to.

This also means anybody working for or on Microsoft (Azure) is able to run
commands as root inside your VM. (They can already take a live snapshot of your
VM with RAM included, but that is a know risk you take when using a cloud/VPS
provider. So consider all your private keys, certificates and data compromised
as soon as you don't control the entire chain of equipment).

Microsoft Azure however is audited regularly and [has an ISO 27001
certificate][9], so let's hope they don't abuse this power.

If you are the only one with an Azure portal account, the impact is probably not
that bad.

If you have multiple people working on a project, (multiple people having access
to the portal), the risk is larger. Any one of those people (and all that had
access to the portal in the past, like contractors or employees that moved on),
has (had) root access to all your VM's.

Any one (or all) of your VM's could be compromised. Maybe your manager has
access to the portal but not SSH, or not root, and want's to put you in a bad
position. Installing that rootkit or cryptocoin miner under your account and
removing all the logging just got way easier.

### Removing the agent/backdoor

The agent is just a package, so using your package manager you can remove it:

    
    
    # dpkg -l | grep walinuxagent
    ii  walinuxagent                        2.2.21+really2.2.20-0ubuntu1~16.04.1       amd64        Windows Azure Linux Agent
    
    
    # rpm -qa | grep LinuxAgent
    WALinuxAgent-2.2.18-1.el7.centos.noarch
    

On Debian/Ubuntu:

    
    
    apt-get purge walinuxagent
    

On CentOS/RHEL:

    
    
    yum remove WALinuxAgent
    

If you just want to stop the service (for example, to see what the impact is),
you can do so using your init system of choice:

CentOS/RHEL:

    
    
    # systemctl list-unit-files | grep agent
    waagent.service                               enabled 
    
    systemctl disable waagent
    systelctl stop waagent
    

Debian/Ubuntu:

    
    
    # systemctl list-unit-files | grep agent
    walinuxagent.service                       enabled 
    
    systemctl stop walinuxagent
    systemctl disable walinuxagent
    

[This guy][10] on Reddit, who claims to work on azure, took the time to write a comprehensive response with more workarounds and possible fixes for this issue. **[Quoting verbatim][11]:**

I believe your commentary has more to do with [RBAC][12] than with the agent,
but I'm trying to better understand your concerns.

Even if you remove [waagent][13] (link to the code on GitHub, thanks for mentioning the code is open source in your post) from an Azure VM, an administrator in your subscription could lock you out with a firewall rule, can restart or stop the VM or can delete it altogether. 

If your primary partition isn't encrypted, they can stop the VM and attach that
disk to another running VM they control and change user passwords, SSH
configuration, etc. And without getting into those weeds, even without `waagent`
you can pass custom data to `cloud-init` (`cloud-init` and `waagent` aren't
mutually exclusive in Azure) using, say, the Azure CLI.

In an organizational setting (a team, a company) it's likely you as the VM
operator have been granted _less_ permissions than the administrator of the
subscription, so it'd actually be expected that they (the administrator) can
perform operations on your VM. Your subscription (your team) shouldn't be an
adversarial scenario, but there are still ways the team can use RBAC here.

If you run `az provider operation list` you'll see that adding an extension to a
virtualMachine is an operation you can actually [write a custom role for][14].
If you're trying to enforce rather than delegate, you can also use a custom
[Azure Policy][15]. All of those methods are enforced at the API level, so the
end result is the same whether you use the portal, the CLI or 3rd party tools
like Ansible or Terraform.

It's also worth mentioning that when you run a custom script from the portal,
the operation is not only logged in the VM but also in the Activity Log for that
resource in Azure itself - even if that was someone else in your team that is a
subscription administrator.

If you open up SSH and have concerns that someone could manipulate SSH or PAM
configuration from outside the VM, there are Azure features such as [just in
time access][16] that are designed to help you exactly with that. But I still
think your commentary has more to do with RBAC than it has with `waagent` or the
`VMAccess` extension. Other redditors have commented, like you did, that there's
a troubleshooting aspect to this. It is true that many features you see in the
portal such as the ability to reset SSH configuration, run a particular command
or see the serial console output are used for troubleshooting (including when
guided by our own Linux Support Escalation team) but there are also there for
composability.

By that I mean someone that has heavily automated/scripted their Linux setup in Azure and instead of maintaining their own custom image (a [documented scenario][17] including extensive discussion on the agent) they rely on standard images and attach extensions with custom scripts, `cloud-init` custom data, pulling SSH keys from Azure Key Vault (or using a third-party tool like Hashicorp's Vault) or using the [AAD login extension][18] (in preview, and not much to do with the AD we love to hate) You said it's not made obvious this agent runs in the VM, and imply that the backdoor nature is concealed. 

I personally always make the point to introduce the agent at any public
presentation (including recently at OSCON) so I'm genuinely interested in your
suggestions for changes in documentation, portal prompts, etc., so people are
more aware that this agent is running and how it helps them? I work on Azure.

(Edit - adding details on Azure Policy and Activity Log)

**End quote.**

### Logging?

When executing a command, the following appears on Ubuntu in
`/var/log/waagent.log`:

    
    
    2018/08/22 11:58:07.546949 INFO [Microsoft.OSTCExtensions.VMAccessForLinux-1.4.7.1] Target handler state: enabled
    2018/08/22 11:58:07.657715 INFO [Microsoft.OSTCExtensions.VMAccessForLinux-1.4.7.1] [Enable] current handler state is: enabled
    2018/08/22 11:58:07.769027 INFO [Microsoft.OSTCExtensions.VMAccessForLinux-1.4.7.1] Update settings file: 1.settings
    2018/08/22 11:58:07.883364 INFO [Microsoft.OSTCExtensions.VMAccessForLinux-1.4.7.1] Enable extension [./vmaccess.py -enable]
    2018/08/22 11:58:08 VMAccess started to handle.
    2018/08/22 11:58:08 [Microsoft.OSTCExtensions.VMAccessForLinux-1.4.7.1] cwd is /var/lib/waagent/Microsoft.OSTCExtensions.VMAccessForLinux-1.4.7.1
    2018/08/22 11:58:08 [Microsoft.OSTCExtensions.VMAccessForLinux-1.4.7.1] Change log file to /var/log/azure/Microsoft.OSTCExtensions.VMAccessForLinux/1.4.7.1/extension.log
    2018/08/22 11:58:10.012703 INFO Event: name=Microsoft.OSTCExtensions.VMAccessForLinux, op=Enable, message=Launch command succeeded: ./vmaccess.py -enable, duration=2025
    2018/08/22 11:58:10.161467 INFO [Microsoft.CPlat.Core.RunCommandLinux-1.0.0] Target handler state: enabled
    2018/08/22 11:58:10.212205 INFO [Microsoft.CPlat.Core.RunCommandLinux-1.0.0] [Enable] current handler state is: enabled
    2018/08/22 11:58:10.262973 INFO [Microsoft.CPlat.Core.RunCommandLinux-1.0.0] Update settings file: 1.settings
    2018/08/22 11:58:10.313158 INFO [Microsoft.CPlat.Core.RunCommandLinux-1.0.0] Enable extension [bin/run-command-shim enable]
    2018/08/22 11:58:11.444806 INFO Event: name=Microsoft.CPlat.Core.RunCommandLinux, op=Enable, message=Launch command succeeded: bin/run-command-shim enable, duration=1031
    2018/08/22 11:58:11.704326 INFO Event: name=WALinuxAgent, op=ProcessGoalState, message=Incarnation 5, duration=4836
    

When changing an SSH key, the following is in that same log:

    
    
    2018/08/22 11:27:00.603980 INFO [Microsoft.OSTCExtensions.VMAccessForLinux-1.4.7.1] Target handler state: enabled
    2018/08/22 11:27:00.713193 INFO [Microsoft.OSTCExtensions.VMAccessForLinux-1.4.7.1] [Enable] current handler state is: notinstalled
    2018/08/22 11:27:01.042711 INFO Event: name=Microsoft.OSTCExtensions.VMAccessForLinux, op=Download, message=Download succeeded, duration=0
    2018/08/22 11:27:01.364254 INFO [Microsoft.OSTCExtensions.VMAccessForLinux-1.4.7.1] Initialize extension directory
    2018/08/22 11:27:01.509190 INFO [Microsoft.OSTCExtensions.VMAccessForLinux-1.4.7.1] Update settings file: 0.settings
    2018/08/22 11:27:01.656294 INFO [Microsoft.OSTCExtensions.VMAccessForLinux-1.4.7.1] Install extension [./vmaccess.py -install]
    2018/08/22 11:27:05 VMAccess started to handle.
    2018/08/22 11:27:05 [Microsoft.OSTCExtensions.VMAccessForLinux-1.4.7.1] cwd is /var/lib/waagent/Microsoft.OSTCExtensions.VMAccessForLinux-1.4.7.1
    2018/08/22 11:27:05 [Microsoft.OSTCExtensions.VMAccessForLinux-1.4.7.1] Change log file to /var/log/azure/Microsoft.OSTCExtensions.VMAccessForLinux/1.4.7.1/extension.log
    2018/08/22 11:27:05.788982 INFO Event: name=Microsoft.OSTCExtensions.VMAccessForLinux, op=Install, message=Launch command succeeded: ./vmaccess.py -install, duration=0
    2018/08/22 11:27:06.105463 INFO [Microsoft.OSTCExtensions.VMAccessForLinux-1.4.7.1] Enable extension [./vmaccess.py -enable]
    2018/08/22 11:27:06 VMAccess started to handle.
    2018/08/22 11:27:06 [Microsoft.OSTCExtensions.VMAccessForLinux-1.4.7.1] cwd is /var/lib/waagent/Microsoft.OSTCExtensions.VMAccessForLinux-1.4.7.1
    2018/08/22 11:27:06 [Microsoft.OSTCExtensions.VMAccessForLinux-1.4.7.1] Change log file to /var/log/azure/Microsoft.OSTCExtensions.VMAccessForLinux/1.4.7.1/extension.log
    2018/08/22 11:27:08.271645 INFO Event: name=Microsoft.OSTCExtensions.VMAccessForLinux, op=Enable, message=Launch command succeeded: ./vmaccess.py -enable, duration=0
    2018/08/22 11:27:08.438678 INFO Event: name=WALinuxAgent, op=ProcessGoalState, message=Incarnation 2, duration=8933
    

Since this is a root access backdoor, the logging on the VM can be compromised.
If you have a centralized logging system, now would be a good time to check if
any of your VM's could have been exploited with this feature.

### What is this (wa-linux-agent)?

This agent is a piece of software created by Microsoft to make life in "the
cloud" easier. OpenStack and Digital Ocean for example have a comparable piece
of software called [cloud-init][19] and the `qemu-guest-agent`. The code is open
source, can be found [here on github][20].

You can read more on the backdoor feature [here][5] ([mirror][6]) and [here][7] ([mirror][8]).

It states the following features:

  * Image Provisioning

    * Creation of a user account
    * Configuring SSH authentication types
    * Deployment of SSH public keys and key pairs
    * Setting the host name
    * Publishing the host name to the platform DNS
    * Reporting SSH host key fingerprint to the platform
    * Resource Disk Management
    * Formatting and mounting the resource disk
    * Configuring swap space
  * Networking

    * Manages routes to improve compatibility with platform DHCP servers
    * Ensures the stability of the network interface name
  * Kernel

    * Configure virtual NUMA (disable for kernel <2.6.37)
    * Consume Hyper-V entropy for /dev/random
    * Configure SCSI timeouts for the root device (which could be remote)
  * Diagnostics

    * Console redirection to the serial port
  * SCVMM Deployments

    * Detect and bootstrap the VMM agent for Linux when running in a System Center Virtual Machine Manager 2012R2 environment
  * VM Extension

    * Inject component authored by Microsoft and Partners into Linux VM (IaaS) to enable software and configuration automation
    * VM Extension reference implementation on [GitHub][21]

### OpenStack / QEMU

OpenStack/QEMU also has an agent, which could be in the image your cloud
provider uses. It's the [qemu-guest-agent][22]. More information on [the
features here][23].

Since OpenStack is not as widespread as Azure and cloud-providers all build
their own images, the impact of this is much lower.

The OpenStack provider I used to work for included this in their images (since
it can help freeze the VM when a snapshot is made, to keep data consistent).

Using the [nova set-password][24] command one can reset a user password via this
agent.

Following the [nova][25] [source code][26] we can see that in the case of
Libvirt (KVM/QEMU) it calls [virDomainSetUserPassword][27].

Bottom line, inspect the software running in your VM before you put it in
production. Check daemons and agents you don't know, check for rouge SSH
keys/users, make use of the firewall, build multiple layers of security, defense
in depth, and most important, use your head.

   [1]: https://raymii.org/s/inc/img/azure-logo.png
   [2]: https://www.digitalocean.com/?refcode=7435ae6b8212
   [3]: https://portal.azure.com
   [4]: https://raymii.org/s/inc/img/azure-backdoor-2.png
   [5]: https://docs.microsoft.com/en-gb/azure/virtual-machines/extensions/vmaccess
   [6]: http://web.archive.org/web/20180822113038/https://docs.microsoft.com/en-gb/azure/virtual-machines/extensions/vmaccess
   [7]: https://docs.microsoft.com/en-gb/azure/virtual-machines/linux/run-command
   [8]: http://web.archive.org/web/20180822121145/https://docs.microsoft.com/en-gb/azure/virtual-machines/linux/run-command
   [9]: http://web.archive.org/web/20180822121005/https://www.microsoft.com/en-us/TrustCenter/Compliance/ISO-IEC-27001
   [10]: https://www.reddit.com/user/bureado
   [11]: https://www.reddit.com/r/linux/comments/99dd5a/linux_on_microsoft_azure_disable_this_builtin/e4n1wr5/
   [12]: https://docs.microsoft.com/en-us/azure/role-based-access-control/
   [13]: https://github.com/azure/walinuxagent
   [14]: https://docs.microsoft.com/en-us/azure/role-based-access-control/custom-roles
   [15]: https://docs.microsoft.com/en-us/azure/azure-policy/create-manage-policy
   [16]: https://docs.microsoft.com/en-us/azure/security-center/security-center-just-in-time
   [17]: https://docs.microsoft.com/en-us/azure/virtual-machines/linux/create-upload-generic
   [18]: https://docs.microsoft.com/en-us/azure/virtual-machines/linux/login-using-aad
   [19]: https://wiki.archlinux.org/index.php/Cloud-init
   [20]: https://github.com/Azure/WALinuxAgent
   [21]: https://github.com/Azure/azure-linux-extensions
   [22]: https://wiki.libvirt.org/page/Qemu_guest_agent
   [23]: https://wiki.qemu.org/Features/GuestAgent
   [24]: https://raymii.org/s/tutorials/OpenStack_nova_get_-_password_set_-_password_and_post_encrypted_password_to_metadata_service.html
   [25]: https://github.com/openstack/nova/blob/master/nova/virt/libvirt/guest.py#L506
   [26]: https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L2072
   [27]: https://libvirt.org/html/libvirt-libvirt-domain.html#virDomainSetUserPassword

---

License:
All the text on this website is free as in freedom unless stated otherwise. 
This means you can use it in any way you want, you can copy it, change it 
the way you like and republish it, as long as you release the (modified) 
content under the same license to give others the same freedoms you've got 
and place my name and a link to this site with the article as source.

This site uses Google Analytics for statistics and Google Adwords for 
advertisements. You are tracked and Google knows everything about you. 
Use an adblocker like ublock-origin if you don't want it.

All the code on this website is licensed under the GNU GPL v3 license 
unless already licensed under a license which does not allows this form 
of licensing or if another license is stated on that page / in that software:

    This program is free software: you can redistribute it and/or modify
    it under the terms of the GNU General Public License as published by
    the Free Software Foundation, either version 3 of the License, or
    (at your option) any later version.

    This program is distributed in the hope that it will be useful,
    but WITHOUT ANY WARRANTY; without even the implied warranty of
    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
    GNU General Public License for more details.

    You should have received a copy of the GNU General Public License
    along with this program.  If not, see <http://www.gnu.org/licenses/>.

Just to be clear, the information on this website is for meant for educational 
purposes and you use it at your own risk. I do not take responsibility if you 
screw something up. Use common sense, do not 'rm -rf /' as root for example. 
If you have any questions then do not hesitate to contact me.

See https://raymii.org/s/static/About.html for details.