Security Operations with VMware vRealize Log Insight

I have been creating a security operations center (SOC) specific to VMware vSphere using VMware vRealize Log Insight (vRLI). This SOC project shows the power of vRLI and the wealth of data available within vSphere 6.5. The original goal was to just gain visibility into my own environment. However, after showing the simple views to a few folks, it has grown from there and continues to grow.

You can download the SOC by visiting my AAC Library GitHub Repository. The SOC is in the vli directory. Download the zip file, unzip, and import the resultant content pack into your VRLI installation. Here are the steps from a Microsoft Windows machine.

Update: 2017-03-17: The name has changed from Texiwill’s Security to Security Operations. There was a dashboard addition, updates to existing once, and a major cleanup.

1. Launch vRLI and log in as an administrative user.
2. Go to the content pack area by bringing up the user menu (red arrow) and selecting Content Packs (green arrow).
3. Select the Import Content Pack link at the bottom (red arrow).
4. This brings up the Import Content Pack Dialog. Click on the Browse button (red arrow).This will bring up the Microsoft Windows open dialog box, where you will select the content pack (green arrow) and click on Open (purple arrow).Then, you go back to the Import Content Pack dialog and click on Import (yellow arrow in previous image).
5. The content pack will be loaded. This will result in a screen representing the details of the content pack. To use the content pack, go back to Dashboards (red arrow).
6. The content pack is now ready to use.

There are six dashboards currently. I will update this post when there are more available. I am working on a seventh, with input from others,  related to API invocations and by which user, but it is not ready yet.

The six dashboards are:

• Login Events and Actions, where we track all actions and events by users within vCenter and vSphere. In effect, anywhere we can grab a username, we do so, whether it is within vCenter or vSphere. We are using the user fields that are part of the vSphere Content Pack (vc_username and vmw_user), not our own defined users. There are several alerts that fire when administrator (local, domain, or SSO) or root users log in and take action. This dashboard is mainly for visibility. You really want to know which users are doing what within your environment. There should be one user per service account, according to the hardening guide. Click to expand the image below and see details. The image above shows a default setup where vRLI and Horizon View are constantly logging in, taking some action, and logging out. We also see several vSphere actions taking place in the third graph, and we break out the vSphere logins by machine in the bottom graph. The topmost graph is about vSphere as well, but is a some of all users. The left-hand side of this dashboard is related to alerts and should mostly be blank or 0, as they are actions taken by administrators and root.
• Firewall Events is a dashboard related to NSX Edge firewalls. It will work for vCNS Edge firewalls as well. Actually, it will work for any firewall that has the proper log format. This dashboard is purely about visibility into what your firewalls are doing. We break out firewalls by activity—DROP, ACCEPT, etc.—and by firewall hostname and source of packets within your network. My biggest sources of data I will not show, but they are related to email and VMware Horizon View.
• VM Configuration Changes does just that: it tracks configuration changes to your VMs. There are still some issues with this dashboard related to the actual count of changes per event, but the dashboard for me only lights up when View is deploying desktops or backup is running. There are alarms for each modify, add, and delete action associated with an administrator.
• VMRC/MKS Events show remote console actions. There really should be no remote console actions from within vCenter or direct to the host unless it is a break-glass situation.  We track which user opened up the console as well as when and to which VM they attached. These graphs should normally be be empty with no results unless you have remote console activity. As an added bonus, we track this behavior if direct access to the host is made through its UI as well. There are alerts if administrator or root is used to access a console.
• DataStore Browser Events track usage and access to the vCenter and host datastore browsers. Once more, these graphs should show no results, as access to the datastore browser should not be allowed. However, if it is, we track by user file deletions, uploads, downloads, and copies (will be available in RC4). We also have appropriate alerts for each if the administrator or root users take any of these actions. In my environment, when Horizon View deletes or adds desktops, I see activity. Otherwise, it is blank.
• Permissions shows visibility into changes made to roles within vCenter.  These graphs and reports should also be blank or no result in normal usage. Unfortunately, we do not quite know which user made the change, but we do have the time a change was made, and a subsequent query can tell us a subset of who made the change. The goal is to see which roles change, and which ones change to include administrator privileges. There is an alert if roles are changed from their existing permissions to administrative permissions.
• Added 2017-03-17 – API Invocations tracks low level interaction with the vSphere and vCenter APIs. In addition, it looks at direct usage of the CLI against any given vSphere host.

At the moment, that constitutes the dashboards in release candidate 4 (RC4). There will be more added as we discover ways to find the necessary data. That is the hardest part. The data is a rich set of data, and as such there are many ways to view it. There are also many types of data.

Show this to your security folks and wow them! This is actually my primary dashboard, as I am looking for any activity by administrator or root—any activity that seems out of the norm. This is one tool to help you find the Admin Escape.

You can download the SOC by visiting my AAC Library GitHub Repository. The SOC is in the vli directory. Download the zip file!