Possible to Restore Server Config?

[dagordon]

I'm currently using StoreGrid Pro 2.5.1, as opposed to 3.1, because of the issue described here. So my question pertains mostly to 2.5.1, but I'm also curious how it applies to 3.1, as I would ultimately upgrade to 3.1 if the issue described in the linked thread is resolved.

My question is simple: to what extent is it possible to restore a StoreGrid server (as opposed to client) config?

I have a computer running StoreGrid server. That computer backs up to a NAS. Say the computer dies, but the NAS, which contains all of the backup data, is fine. Would there be any way to restore the data?

I should say that I'm storing the StoreGrid "internal database" on the NAS itself. So the internal database would be completely intact.

Just as a test, I shut down my backup server and installed StoreGrid server on another machine, with the same StoreGrid ID as that of my server. I even temporarily changed the name of the machine to the name of the server. I then specified the same location for the "internal database" that I was using for the server.

Result: StoreGrid seemed to think that this was a brand new installation. It didn't find the backup jobs I had performed. I even tried the "Disaster Recovery" feature, and this didn't work, I think because this is only for restoring client configs.

So assuming the backup data is intact but the StoreGrid server installation disappears, is there anyway to get the data?

I sure hope so. It's a basic principle of data protection that you should minimize possible failure points. If the backup data itself gets destroyed, of course I won't be able to backup. But if the backup data itself is fine, I should really be able to use it!

Thanks

[kshama]

Hi dagordon,

In StoreGrid v2.5.1, if the backup server crashes but if you have the backup data and StoreGrid's DB in an external (NAS) drive, you can reconstruct the backup server by following the steps as outlined in this knowledge base article:

https://support.vembu.com/index.php?_m=knowledgebase&_a=viewarticle&kbarticleid=148

Please note that steps 4 and 5 described in the article are critical. When you tried to attach the data back after installing the StoreGrid backup server afresh, looks like you had not performed these two steps.

In StoreGrid v3.x,, if you have the MySQL data and backup data is in the external (NAS) drive and if your backup server crashes, all you need to do is rename (or delete) the ib_logfile0 and ib_logfile1 files in the MySQL Data location. Then, during MySQL installation in the new machine, just point it to the same data location as before. Once the MySQL server is up and running with the earlier data, install StoreGrid using the new MySQL installation's DSN.

Regards
Kshama
Vembu Technologies

[dagordon]

Question about restoring the server config w/ 3.x:

Exactly what files do I need to have in order to do this?

There seem to be three different types of files that might be needed:

1. The backup data (of course), stored wherever the server is configured to store the backup data;

2. The StoreGrid internal database, stored wherever "DBStorageLocation Path" points to in the server config file;

3. Various MySQL files, stored wherever MySQL is installed.

So which of these files to I need in order to restore my backup data, assuming my server dies? Clearly 1, and I'm pretty sure 2, but what about 3?

kshama's last email makes it sound like I need 3 as well. Is this true? So if I want to be protected against the scenario that my server dies, I should back up the entire MySQL installation folder, or have MySQL installed to the location of the backup data?

In any case, this is one of the single most important things that users should know -- i.e., what files are necessary in order to restore their data -- and it doesn't seem like this is clearly presented anywhere. This is very bad.

For every release of StoreGrid, there should be some place easily accessible that documents all the files that will be needed to restore your data.E

For one thing, can you please make a knowledge base article similar to the one you linked above but for migrating a StoreGrid 3.x installation?

Thank you.

[dagordon]

To add to this:

I did find this article:

https://support.vembu.com/index.php?_m=knowledgebase&_a=viewarticle&kbarticleid=146

It says that to restore a server config you need two things:

1. The '1' folder located in <Backup_Storagelocation>
2. The Backup of master database.

However, it's not clear what (2) is. The article suggests (in step 3) that it's not just the files stored in "DBStorageLocation Path", but rather is some kind of dump that must be created from within the old backup server. So this is not of any help if the old backup server dies.

I repeat, this information has to be more clearly stated somewhere.

[nik24]

I would also like a step by step disaster recovery process of the StoreGrid server. We store the backup data on a NFS share. So in the event of the disaster we will have all the data, but potentially no meta database. I have a couple of questions:

1. Is there any point in backup up the meta databases? The meta db backup will always be out of data by up to 24 hours (assuming we do daily backups). So any record of backups customers create after the database has been backed up will be lost if we need to restore from backup. This makes the database out of sync with the backup data and therefore useless?

2. What is the best way to restore the meta database?

3. Backing up the meta database is probably not practical anyway because it can be extremely large (we will have 14TB of back data, and millions of files).

B

[kshama]

Hi,

This forum thread has the discussion on backing up the backup server's MySQL data:

http://www.storegrid.com/forum/viewtopic.php?t=4975

Regards
Kshama
Vembu Technologies

[dagordon]

This is very frustrating for me. I read the thread linked to in the previous post, and it raised more questions than it answered. I've also emailed support about this issue and am still not clear about what is required in order to restore a server config.

I should say that I think StoreGrid is the best backup software currently in existence, but the lack of clarity on how to actually restore my data should I need to has me wondering whether I should be looking into other solutions.

I suppose these are my main questions:

1. Is it possible to restore my data if all I have is the backup data (whatever is stored in the backup location)?

2. Related to 1 -- do I need any files in <DBStorageLocation> in order to restore my backup data?

The second post in this thread seems to suggest that with StoreGrid 2.5.1, the answer to (1) is NO and (2) YES. (This is why with 2.5.1, I had <DBStorageLocation> actually pointing to the backup location (a NAS), so that if my server died I could still restore my data.)

Is this no longer the case, with StoreGrid 3.x?

--

More specifically, here's the response I got from support about how to restore the server config; I've inserted comments.

> To recover the backup server in case of server crash, you need to have the
> following data:
>
> 1. The '1' folder located in <Backup_Storagelocation>
> 2. The Dump of master database.
>
> The MySQL database of the backup server consists primarily of two types of
> data. One is the data related to the backup server itself (server DB)- its
> configurations, the reseller, customer and client accounts, its invoicing,
> replication settings etc. The second part is the meta data of the backup data
> (backup DB).
>
> Backup Server by default dumps the server DB data into a file once every day
> for a backup. This file to which the server DB is dumped will be named .sql
> and will be created in the location specified for backup data storage during
> installation.

So only the "server DB" portion is dumped -- what about the metadata portion? Do I need the metadata portion in order to do a restore?

>
> By default MySQL master database(which contains configurations, the
> reseller, customer and client accounts, its invoicing, replication settings etc)
> will be dumped in the location that is specified as backup location in the
> backup server ''Server Admin > Server Management > Server Settings'' UI
> page(From StoreGrid v3.1 SP1 onwards). If you have changed the backup
> location, then the ''DatabaseName.sql'' file will be stored in the new backup
> location configured in the ''Server Settings'' UI page. The new location will be
> applicable for the new clients connecting to the back server. The existing
> clients will continue to backup to the old backup location in your backup
> server.
>
> The dump will be in the .sql format. By default the dump will scheduled at
> 12.00A.M daily. You can change the dump path and the schedule by
> modyfing the above tags.

I'm currently looking in my backup data location and can't find any .sql files.


> Now using the '1' folder and this dump, you can recover the backup server by
> following the below steps.
>
> 1. Copy the clients' backed up data <Storage Location>/1 folder to a new
> backup server location <Storage Location>/1 folder in the same drive in the
> new backup server.
>
> If you would like to change the backup location in the new backup server,
> please copy the '1' folder in the required location, then we will update you
> the sql queries.(If you need those sql queries, please refer step 5.)
>
> 2. Make sure you have configured the machine name of your new backup
> server same as the old backup server's name.
>
> 3. Install and configure MySQL Server & MySQL Connectors in your new
> backup server as mentioned in the below URL,
>
> Windows:
> http://www.vembu.com/storegrid/sp-help/mysql-setup-windows.html
>
> Linux:
> http://www.vembu.com/storegrid/sp-help/mysql-setup-linux.html
>
> Once you have successfully installed the MySQL Server & MySQL Connectors,
> please start MySQL Server.
>
> 4. To import the StoreGrid Backup Server database to your new Backup
> Server MySQL, please follow the below comand in mysql prompt, After login
> to MySQL.
>
> mysql -u root -pvembu < dumpAllDatabase.sql
>
> dumpAllDatabase.sql file is the dump of database from the old server.

Am I supposed to enter the full path of the dumpAllDatabase.sql file here? How is it supposed to know where the file is located?

>
> Note: If the backup location is same in the new backup server as the existing
> backup server, you can skip the step 5.
>
> 5. If the backup location is going to be different in your new backup server, to
> update the new backup location all the client's backups in the new backup
> server master database, please follow the steps given below.
>
> 5A. Access the StoreGrid server database by executing following command
> in our command prompt,
>
> mysql -u<DB_USER_NAME> -p<DB_PASSWORD> --
> database=<DB_NAME>
>
> where <DB_USER_NAME> is your database user name, <DB_PASSWORD> is
> your database password and <DB_NAME> is your database name.
>
> 5B. To update the new backup location all the client's backups, please run
> the following SQL queries:
>
> update SERVER_CLIENT_GROUP_DETAILS set
> BACKUP_LOCATION="<NEW_LOCATION>";
>
>
> update BACKUP_GROUP_DETAILS set
> BACKUP_LOCATION="<New_Storage_Location>";
>
> update GROUP_DETAILS set
> BACKUP_LOCATION="<New_Storage_Location>";
>
> 6. Install StoreGrid in your new backup server with same StoreGrid ID and
> backup location as before.
>
> NOTE: If you have changed the public DNS Name of your backup server, then
> you need to point all your clients to the new backup server by doing the
> client side Local To Remote Server migration. For more details, please refer
> our forum post here
> http://www.vembu.com/forum/viewtopic.php?p=9157#9157
>
> 7. Start StoreGrid in your new backup server.
>
> 8. Check if the all the customers, clients & backups are listed properly in new
> backup server.
>
> 9. Schedule the backups from the StoreGrid clients and check if the clients
> are backing up incrementally to the new backup server.
>
> Hope this clear. If you have any further queries please feel free to >contact us.

In addition to the questions I've listed here, I have a more general concern that this process for restoring the backup data (specifically, #3, the process for installing MySQL) seems extremely cumbersome. Is there really no desire to have a simple application that will restore the server config?

If my backup server dies, the last thing I want to be doing is running manual MySQL commands via a command line... if people had to do this to install StoreGrid, I suspect it would have far fewer customers...

Does this concern no one else?

As I said, I really do love this product, but it seems to me that the very core functionality of being able to restore data is neither clear nor optimal!

Thank you.

[ottomark]

I'm currently deciding whether to purchase StoreGrid and the lack of response on this issue is disturbing... especially because in the forum thread that kshama linked to he said that he would update the discussion with a link to a help document but he never did.

The single most important thing for me is knowing that I can restore my backups. Is there really no easy process for doing this if there's a problem with the server installation???

[kshama]

Hi Ottomark,

We are still working on the getting the help document ready which describes in detail with screenshots what it would require to restore a backup server in case you lose the backup server's MySQL DB completely, no replication of the backup server was done and you only have the backed up data. Once the document is ready, we'll be giving you the link to it.

But as described in this forum post, StoreGrid backup server database consists basically of two sets of data. One is the details of the customers, clients etc and the other is the meta data of the backup data itself. You need both these restored before the clients can resume backing up from where they left off.

For the second set of data, StoreGrid backup server has an inbuilt process that can be run to reconstruct the meta data from the backup data.

For the first set of data, StoreGrid dumps the relevant MySQL DBs as [dbname].sql file in the storage location on daily basis. This file can be used to construct the customer/client details.

Now, in StoreGrid we have gone one step beyond this to provide redundancy of the MySQL data. We also stored Customer/Client related information in the backup storage location as sgclient.txt files under the respective client backup locations. If in case even the [dbname].sql file dump is not available or corrupted, the Customers/Clients related data can be reconstructed from the sgclient.txt files.

We are right now working on updating the inbuilt process (which currently can construct the meta data DBs from backup data) also to build the customer/client information DBs as well. Once this is ready, all the MySQL DB can be constructed from the backup data alone, you wont need the daily dump ([dbname].sql file) that the backup server creates.

Like mentioned before, we'll document all this in a help document and send the link to you all. We are trying to get this complete as quickly as possible and it should be available by tomorrow.

Meanwhile, please put your worries to rest. As long as your backup data is intact, your server can be brought back.

Regards
Kshama
Vembu Technologies

[kshama]

Hi dagordon,

[kshama]

Hi,

Here is the link to the document that describes the process to restore the backup server after a total crash with only backup data storage location intact.

http://www.storegrid.com/online-backup/sp-help/PageHelp/rebuildbackupserver.html

Regards
Kshama
Vembu Technologies

[dagordon]

OK, this is very helpful. Thank you!!

My only comment is regarding this point:

"If the backup storage location is changed from the 'Server Settings' page in the StoreGrid webconsole, then the database dump remains in the location given during the server installation. This is applicable for StoreGrid v3.0, 3.1 and 3.2."

I suggest this be changed, so that the location of the database dump follows any changes to the backup location.

As it is, someone could install StoreGrid with the default backup location (which is on the local machine), change their backup location and back up to the new location, the server dies, and they're screwed.

[kshama]

Hi dagordon,

Yes, it has been fixed in 3.1 SP1 and 3.2 SP1 releases. With this change, the MySQL DB will dump the .sql file in the backup location as configured in the Server Settings page.

Regards
Kshama
Vembu Technologies

[dagordon]

[dagordon]

OK, I have installed SP1 over StoreGrid 3.1 Pro.

It seems to be creating the folders for the dump, but it not actually dumping the files.

That is, in my backup location, I now have the folders /SGDatabase/SGDatabase/. But there are no files in these folders.

In the conf file, for:

<BackupServerMetaData Enabled="1" Path="C:\Program Files\DBDump" BackupDBEnabled="0"><Schedule Type="Daily" Day="0" Hour="0" Mins="0">

should BackupDBEnabled be 0 or 1? I have BackupServerMetadata Enabled set to 1, but BackupDBEnabled is 0, as in your example above.