Check Your Veeam Cloud Connect Repository Storage Usage

Here are the steps to check your Managecast Veeam Cloud Connect repository usage through the Veeam Backup and Replication console:

    1. Open the Veeam console and select the ‘BACKUP INFRASTRUCTURE’ tab on the bottom left.
    2. Then, select the ‘Backup Repositories’ group on the top left.
    3. A list of existing repositories will be displayed, select the cloud repository. It will typically be named ‘Managecast Cloud Repository’ and the repository type will be labeled Cloud.
    4. Right click the ‘Managecast Cloud Repository’ and choose ‘Properties.’
    5. VEEAM CC REPO USAGE

    6. The window that is displayed will show the settings for the selected cloud connect repository including: Capacity, Used space, and Free space.

    VEEAM CC REPO USAGE PROPERTIES

Veeam and AWS Storage

Amazon cloud storage, S3 and Glacier, are top contenders when it comes to storing data offsite and Veeam is a behemoth when it comes to VM backup; you can use the two together but should you? The answer: well, it depends.

Integrating a Veeam Backup and Replication configuration with Amazon storage is done through an AWS storage gateway. An appliance that sits between your Veeam server and the AWS Cloud. You can connect this gateway to the Veeam server as a file server, direct attached storage, or a virtual tape library.

With the amazon storage gateway as a file server or direct attached storage you can backup directly from Veeam to the storage gateway. You can perform incremental backups this way, but in order to avoid the corruption possible with long backup chains periodic fulls will need to be taken. Synthetic operations are possible in this configuration, however without a proxy on the AWS side anytime they are performed Veeam will need to read the full backup and incrementals stored in the AWS storage, effectively downloading it from the Amazon servers, performing the synthetic operation locally, and re-uploading new synthetic data. This causes the synthetic operations to take incredibly long times to complete and is not recommended.

The other option is the virtual tape library(VTL) which allows you to present the storage gateway to Veeam as a tape server. This lets you to use Veeams tape backup jobs to create virtual tape backups on AWS storage. While Veeam tape backups allow for incremental backups, this method also requires periodic fulls to be taken any time a new tape is required. Which may end up happening frequently, depending on backup size and retention, as the maximum tape size for AWS storage gateway is 2.5TB. For restores requiring already archived/removed, it can take up to 24 hours for a tape to be retrieved, for a cost, and made available again in the tape gateway.

Alternatively, Veeam has an offsite backup method built into its distribution that comes in the form of Veeam Cloud Connect. Cloud Connect partners are third party Veeam service providers running Veeam cloud gateways and repositories who supply you with the necessary storage and compute tailored to performing offsite Veeam backups.

End users put in their service providers information and login information and are provided with a ‘Cloud Repository’ in the Veeam console. They can send backups, backup copies, etc. to the cloud repository.

Instead of having to perform periodic full backups, tenants can perform forever forever incremental backups with periodic synthetic fulls. Because the service provider is also running Veeam and has the compute required for synthetic operation, synthetic fulls and merges are able to be performed locally at the remote site—significantly reducing backup windows.

With a pre-configured appliance for cloud connect available from Veeam, Azure may be a better option compared to AWS. This lets you setup and manage your own cloud connect server in the Azure cloud to point offsite backups to. However, this adds another layer of complexity/management as well as costs for not only Azure storage but also hourly compute costs as well as additional Veeam Cloud Connect licensing. It is certainly more feasible to look for an already available Cloud Connect partner who already manages their own offsite Cloud Connect server. A Veeam Cloud Connect partner can provide the specialized backup service, offering true incremental forever backups without the complexity of creating your own service.

As far as using AWS for offsite backup with Veeam it is certainly doable, however due to the fact that full backups are required regularly it could only be recommended for those with less than 1 TB of data to backup, those with low frequency backup who can afford extremely large backup windows, or those with a lot of bandwidth available to the amazon storage servers.

Otherwise it may be more feasible to go with a cloud connect provider who can offer nightly incremental backups with regular synthetic fulls, significantly reducing backup times because they can perform the necessary synthetic operations on the remote repositories. It’s not only more time effective, but also often times more cost effective, and simpler from a management perspective.

VEEAM 9.5 ReFS Integration

Veeam v9.5 was recently released and with it came a large number of improvements and added features. Namely, the seamless integration of Microsoft Server 2016’s new ReFSv3.1 filesystem. Veeam’s integration with this version of ReFS adds the fast cloning ability and spaceless full technology, meaning merges and synthetic operations require less resources and time and synthetic full backups take up significantly less storage.

Veeam 9.5 ReFS Benefits

Veeam 9.5 integration with Windows server 2016 ReFS comes with 2 significant benefits to synthetic and merge operations: Fast Cloning and Spaceless Full Technology.

Both of these rely on Veeam v9.5’s integration with ReFS allowing Veeam to utilize ReFS block cloning. This allows Veeam backups to copy data blocks within files or even from one file to another very quickly. When data is copied, the file system will not create new copies of existing data blocks and instead creates pointers to the locations of existing data.

–Fast Cloning with ReFS ensures that because new full backups do not need to copy existing data and instead use pointers to existing backup data it significantly reduces the time required to perform synthetic operations which can normally be very resource intensive.

–Space-less Full Backups are possible due to the fact that new synthetic backups are primarily made up of pointers to existing full backup data, significantly reducing the space required to store the backups.

One of the downsides of space-less fulls when compared to a deduplicating storage device is that there is no global ded uplication. Space-less fulls with ReFS will only reduce storage usage between copies of the same full backup file. Even still however, the space savings are tremendous:

CaptureReFSSize

 

In the picture above we moved a customer with close to 1TB native backup size from an NTFS backup repository to an ReFS repository. After a month of weekly and monthly GFS backup copies the utilized storage is less than half of the native file size. The customer already had full backups, and as those older GFS restore points get removed due to retention and replaced with ReFS spaceless fulls the utilized storage will continue to decrease.

–Encryption is also possible with ReFS space-less fulls. One of the downsides of deduplication is that backup files cant be encrypted otherwise you lose the benefits of deduplication. Because the space-less fulls and encryption are both transparent to Veeam, it’s able to both encrypt the data while still providing the space saving benefits of space-less full backups on ReFS.

Adding ReFS Volumes as Veeam Repositories

In order to see the benefits of ReFS with Veeam older repositories will need to be attached to a Windows Server 2016 server and formatted as ReFS. If you had previously added a Windows Server 2016 ReFS volume as a repository, it will need to be readded after upgrading to Veeam v9.5 in order for Veeam to recognize it as an appropriate ReFS volume and allow the new features to be utilized.

Important: Veeam’s fast cloning and spaceless full technology only supports ReFS volumes created on Windows Server 2016, volumes formatted as ReFS in Windows Server 2012 will not see the benefits because Server 2012 uses an older version of ReFS.

Any restore points created prior to the v9.5 upgrade will not see the new benefits. In order to utilize fast cloning and spaceless fulls, all full and incremental backups involved in synthetic operations will need to have been created using Veeam v9.5 with a Windows Server 2016 ReFS repository. This means the benefits of fast cloning and spaceless fulls will not apply when you first copy older backup data into the new ReFS repository. Therefore, in order for existing backup or backup copy chains to begin seeing these benefits, either an active or synthetic full(including backup copy Synthetic GFS) will need to be performed. Then the next time a synthetic operation is performed the [fast clone] tag will be displayed next to the synthetic operation in the job activity logs, as well as a corresponding increase in the speed of the operation.

Update: Make sure to use 64K Block Size when formatting the Veeam repository volumes to avoid issues with 4K Block Size and ReFS. Read this post for more information.

Issues Running Backups or Rescanning Repository After Veeam Upgrade

Just recently, right after upgrading to Veeam 9.5, we ran into an error with one of our customers that would show up whenever backups started to run and when we tried to rescan the Veeam repository. The errors messages were:

Warning Failed to synchronize Backup Repository Details: Failed to synchronize 2 backups Failed to synchronize 2 backups
Warning Failed to import backup path E:|PATH|TO|BACKUP|FILES|BackupJobName.vbm Details: Incorrect file system item link received: BackupJobName

Based on the errors it looked like there were issues with the database entries for the backup job mentioned in the error. As a troubleshooting step we tried removing the backup files from the GUI under Backup & Replication>Backups by right clicking on the backup and selecting ‘Remove from Configuration.’ However, this ended up giving us the same error in a popup dialogue:

Incorrect file system item link received: BackupJobName

After opening a ticket with Veeam they informed us that this is a known issue caused by unexpected backup entries in the VeeamBackup SQL database. Namely, the issue backup jobs were listed in the database with a zero entry for the job ID or repository ID causing Veeam to not be able to locate the backup files.

Because this involves the Veeam SQL database we are going to be making changes to the database so it’s best to back it up before the next steps. Here’s a knowledge base article from Veeam that shows the suggested methods to backing up the database.

To determine whether there are any errant backup entries in the SQL database run the following query:

SELECT TOP 1000 [id]
 ,[job_id]
 ,[job_name]
 ,[repository_id]
 FROM [VeeamBackup].[dbo].[Backup.Model.Backups]

Under ‘repository_id’ you should see one or more of the backup jobs showing ‘00000000-0000-0000-0000-000000000000’ or ‘NULL’ as the id for job or repository. Any entries with this issue will need to be removed from the database in order to resolve the error.

It’s best practice to save a backup of the SQL database before making any changes. If you’re unsure of how to back up the SQL database, follow Veeam’s knowledge base article. 

After backing up the SQL database run the following query for each job with ‘00000000-0000-0000-0000-000000000000’ as the repository_id:

EXEC [dbo].[Backup.Model.DeleteBackupChildEntities] 'REPLACE_WITH_ID’
EXEC [dbo].[Backup.Model.DeleteBackup] 'REPLACE_WITH_ID’

Replacing the ‘job_id’ with that of any backups with the unexpected ‘repository_id’ found with the previous query.
After that issues with the local backup server was resolved.

However, we were still seeing errors when trying to connect the backup server to our cloud connect repository to do backup copies. We were still getting the following errors:

Warning Failed to synchronize Backup Repository Details: Failed to synchronize 2 backups Failed to synchronize 2 backups
Warning Failed to import backup path E:|PATH|TO|BACKUP|FILES|BackupJobName.vbm Details: Incorrect file system item link received: BackupJobName

In order to resolve this we had to remove the entries for the problem jobs from the cloud connect server database. For those using a cloud connect service provider you will have to have them make changes to the SQL database.

We had 2 VMs that were giving the ‘Incorrect file system item link received: JobName” error. So we had to remove any entries for those jobs from the SQL db.

We ran the following query to get the Job ID of both jobs mentioned in the errors:

SELECT TOP 1000 [id]
 ,[job_id]
 ,[job_name]
 FROM [VeeamBackup].[dbo].[Backup.Model.Backups]

Then we ran the same delete query as before using the new job_id’s:

EXEC [dbo].[Backup.Model.DeleteBackupChildEntities] 'REPLACE_WITH_ID’
EXEC [dbo].[Backup.Model.DeleteBackup] 'REPLACE_WITH_ID’

After those entries were deleted we were able to rescan the repository.

Lastly, once we rescanned our cloud repository and imported the existing backups we started getting the following error message:

Warning Failed to import backup path E:|PATH|TO|BACKUP|FILES|BackupJobName.vbm Details: Path E:|PATH|TO|BACKUP|FILES|BackupJobName2016-11-21T220000.vib is absolute and cannot be added to other path

This error indicates that there is a problem where the backup chain isn’t accurately resolving the location of the GFS restore points. In order to resolve this we had to manually remove the Imported backups from the cloud connect server by going to Backup & Replication>Backups and selecting the backup job and choosing ‘Remove from Configuration’ making sure to check ‘Include archived full backups.’ After the backups had been removed from the cloud connect repository we were able to manually rescan the repository on the local backup server and the backup files were imported again successfully.

Update:  After deleting these affected rows using the commands above you may get the following error message:

Unable to map tenant cache ID 'ID-GOES-HERE' to service provider side ID 'ID-GOES-HERE' because it is already mapped to ID 'ID-GOES-HERE'

If you do see this error a solution can be found in this Veeam KB article. Essentially, the data for these deleted rows still exists in the Veeam DB cache table and needs to be removed. In order to do this run the following query on the VeeamBackup database:

delete from [cachedobjectsidmapping]

This will delete the DB cache table. In order to re-populate it you will need to rescan the service provider and the Cloud Repository.