Here is what happened: Early one morning, an employee’s Surface laptop was infected with some sort of malware which performed 1,754 file deletions in Windows Explorer. All of the affected files were in a shared OneDrive for Business folder that contained important data (e.g. contracts, estimates, project records, and proposals).
In this case, the customer has HubStor in place for their Microsoft 365 backup, so they were able to recover the data quickly and fully. HubStor restored the deleted files, and the customer rebuilt the Surface laptop.
However, we took this opportunity to consider just how important it is to have a data protection solution for your OneDrive for Business data.
What would this customer’s scenario be like if they didn’t have HubStor?
OneDrive for Business native backup and recovery tools
To fully appreciate the native capabilities of Microsoft 365, we have to consider that most organizations subscribing to Microsoft 365 make heavy use of OneDrive for Business. It’s a great tool for collaboration. They’ve likely replaced traditional home directories with OneDrive for Business sites. And most workers are probably heavy users of their OneDrive, frequently sharing folders and files as needed, with all sorts of data that is critical to the business.
The following is our assessment of what organizations face (the good, bad, and the ugly) when depending on Microsoft 365 native tools alone.
Mass file deletion notification
The OneDrive owner benefited from a new feature Microsoft introduced in August 2018 to receive email notifications of mass file deletion activity. Without this, the issue may not have been detected so easily. Here’s the email:
In our customer’s case, the notification email was initially dismissed as a phishing email by the recipient It was only upon closer inspection that the person decided it may be legit.
While it is a great feature, the problem is that the usual Microsoft 365 user is receiving several phishing emails each day that do a very good job at looking like official Microsoft alerts about their Microsoft 365 account.
These phishing emails have a numbing effect on the user — you start to see legitimate Microsoft emails as just more phishing emails.
And that’s a problem because, as the message states, “When files are deleted, they’re stored in your recycle bin and can be restored within 93 days. After 93 days, deleted files are gone forever.”
The OneDrive for Business recycle bin
SharePoint Online and OneDrive for Business data goes to a Recycle Bin when it is deleted. The Recycle Bin provides the list of deleted files within the past 90 days. You can select one or multiple files and either delete or restore them with the click of a button.
Problem solved, right?
We simulated some OneDrive for Business deletion scenarios and did some Recycle Bin testing of our own. Our restore from the Recycle Bin yielded a partial recovery. Approximately 20% of the files wouldn’t restore.
The restore error for each file said the file already existed in the location. However, we checked several files getting the error and they were NOT in the location.
Upon closer inspection, we noticed that the error details for each file referenced the same file and location. Confusing at first, after more investigation we pieced it together. Microsoft’s code is batching the restore jobs and they aren’t handling item errors elegantly — if one item in the batch fails, it appears that the remaining items in the batch fail.
If you encounter this problem, one workaround is to click through items in the Recycle Bin, performing a restore job for one item at a time. Painful.
Another option is to roll up your sleeves and write some code. Using the Graph API or SharePoint Client Object Model (SCOM), you could initiate restores of recycle bin items for a MySite and handle the errors properly yourself to ensure you initiate a restore for each item that needs to be restored.
(We also explored using SCOM or PowerShell to rip a copy of the data in the Recycle Bin out to an alternate location, but that appears to not be an option. Programmatically, just like in the GUI, your actions are to either restore (to original location) or delete from the Recycle Bin.)
What about the Windows recycle bin?
The inability to execute a flawless restore from the Recycle Bin is concerning, especially if you have a large number of files to recover. Posts in online forums reveal that OneDrive restore scenarios often involve 50,000 files deleted accidentally.
With larger numbers of files to restore, you’re more likely to hit the batch restore error scenario and initiating item-level restore jobs in the Recycle Bin GUI is just not practical.
Another place you could recover from is the Windows recycle bin on individual workstations. We tested this and it works but it’s almost impossible to understand what you’re going to be able to recover with this method.
For example, if several users are accessing files from the same OneDrive for Business site collection, and they’re doing so from the desktop, then they have some of the files locally.
If one of the users then performs a mass deletion, the sync engine on each local machine removes the files for the other users. If the other users have these files locally then they’ll go to the Windows recycle bin on that machine. If you restore from there, it’ll put it back into the local OneDrive folder scope and sync back up.
If one of the users disabled the ‘File On-Demand’ feature, then they should have a complete list of files for any folders they can access in their Windows recycle bin in a mass deletion scenario.
Not a huge fan of the OneDrive recycle bin and standard restore capabilities?
If you’re not thrilled about the Recycle Bin and Microsoft 365’s native recovery capabilities you’re not alone. The functionality is too basic.
The problem of over-privileged OneDrive permissions
Before we look at some recommendations, I thought it was pertinent to call out a major reason for the OneDrive for Business data recovery fire drills you might be having – permissions!
I wish SharePoint Online had a permission level that allowed read, write, and edit BUT NOT delete!
Unfortunately, there is only one permission level that denies delete permission: Read. This is a view-only level of access. There are multiple permission levels, but all except ‘Read’ include the ability to delete!
In my experience, when I share things with others from my OneDrive, I want their contribution to my files but I don’t want them deleting anything!
Think about this in practice: OneDrive makes it easy to share folders and files, and these shares likely are at the permission levels of Design, Edit, or Contribute – all of which include the power to delete.
Is anyone reviewing and trimming these entitlements as time passes? Not likely in most organizations. That means your OneDrive for Business data is likely exposed to more people than you realize, many of them having delete control over some of your OneDrive files!
Recommendations for how to protect OneDrive for Business
Although not achieving a true backup, here are some things you can do in Microsoft 365 to help protect your organization’s information:
- If you are suspicious of a user who may be prone to deleting files accidentally or maliciously, I suggest setting up an alert policy in the Microsoft 365 Security & Compliance Center. Alert policies allow you to get notification may events, including unusual mass deletions in SharePoint and OneDrive for Business from specific users or IP addresses. Alerts won’t protect your data, but they’ll let you know about possible issues early (better to know immediately than to find out after things expire from the Recycle Bin).
- If your organization uses a Security Information Event Management (SIEM) solution, consider feeding your Microsoft 365 logs into your SIEM for better activity monitoring and alerting.
- Other than the Recycle Bin (which is not a robust data protection solution), your other options are to set retention policies or place users on legal hold in Microsoft 365. Applying a legal hold to OneDrive for Business sites is a little more involved than you might expect. Both approaches will safeguard against accidental or malicious deletions, and they can work on other content, too (such as mailboxes and Team sites). However, they are by no means elegant approaches for a variety of reasons. First, deleted data from OneDrive for Business or SharePoint Online goes into a preservation hold library in SharePoint Online. While this appears viable at first, the problem is the ease by which you can perform large-scale recovery. Bottom line, native capabilities to recover data from the preservation hold library do not appear to be available. from which doing a recovery is unclear. (For mailboxes, deletions go into a deleted items recovery folder hidden in the user’s mailbox). Keep in mind, your legal department may not like setting a retention policy on data, and similarly using legal hold for data protection may be unwelcome.
Unfortunately, none of the above achieve a true, proper backup of your data, and most recovery scenarios are not simple to support.
What is HubStor’s data protection strategy for Microsoft 365?
At HubStor, we use a multifaceted approach to protect our Microsoft 365 data, which includes:
- HubStor’s Microsoft 365 backup — As you might expect, we use our own HubStor technology internally to back up Microsoft 365. This gives us assurance with a safe, segregated copy of our data that we use to perform fast recovery of items, folders, lists/libraries, and sites as needed. Our internal HubStor tenant protects our Office 365 data with the following configuration:
- OneDrive for Business sites connector — HubStor performs a nightly incremental backup of all OneDrive for Business sites.
- Group and Team sites connector — It also gets a nightly backup of all Group and Team sites across our organization with auto-enrollment of any new sites.
- SharePoint Sites connector — All normal SharePoint sites are backed up nightly.
- Exchange Online connector — And HubStor maintains a backup of all user mailboxes (active and archive).
- Long-term Retention of Audit Log — In addition, we use HubStor’s connector for the Microsoft 365 audit log to have long-term retention of our event history. This data is readily searchable in HubStor using any of the fields from the audit log.
- Microsoft 365 Retention Policy — We also have an indefinite (forever) retention policy configured in our Office 365 tenant and it applies to all possible data types.
- Sharing Security Rule — We set sharing links to expire in 60 days automatically.
- Alert Policies — Finally, we have several alert policies that give our executive team insight into possible concerning activities.
In closing, we’ve seen it enough times to know that accidental or malicious data deletion is a real threat to businesses. The problem with most Software-as-a-Service (SaaS) offerings — whether it be Microsoft 365, Slack Enterprise, or RingCentral, as examples — is that they don’t offer true backup and recovery capabilities that work at the enterprise level.