|
| Author |
Message |
bcarlos
Joined: 24 Nov 2009
Posts: 11
|
 Posted: Fri Jul 23, 2010 1:39 pm Post subject: Impossible to move storage to new location |
 |
|
We are trying to move around 4TB of backup data to a new location but it seems to be impossible because of so many files Storegrid creates.
Why doesn't Storegrid create a single file per backup? We have currently millions of small files which make it impossible to copy with tools like rsync.
This setup has a small customer base (around 20 clients) and Storegrid mysql database is already 15GB large.
I don't see how this product can scale and allow for hundreds of customers, and how you can copy all backup data from one place to another in a fast way, without having to shut down the all thing down and do a block level copy. |
|
| Back to top |
|
 |
kshama
Joined: 04 Sep 2007
Posts: 939
|
| Posted: Tue Jul 27, 2010 12:21 am Post subject: |
|
|
Hi bcarlos,
StoreGrid creates its own multiple directory structure while storing the backup data. Each file that is backed up is stored as a separate individual file in the backup server. All files and folders of one backup are stored under a separate folder. It is therefore possible to move the backup location of a backup alone or for the entire client machine. Therefore, you could move backup data for each client one by one after deactivating just that client in backup server (so that is doesn't start backing up while the move is on), while the other clients continue to backup to the old location.
MySQL size would depend upon the number of files backed up. As a rough estimate, a million files backed up could consume about 1.5 GB. But StoreGrid uses sharded database - one DB for each of the backups, making it highly scalable.
To further enhance its scalability, we are working on supporting multiple MySQL servers in a cluster setup with multiple backup servers where you can add MySQL servers and backup servers into the cluster on the fly as you scale up.
We also working on aggregating the disk writes of multiple concurrent backups into a single stream which will write the data into a single file. With this change, you could have several concurrent backups running in backup server but yet only one thread will be writing the data into the disk, which avoids disk rotational latency. With this change, all the backups would be written into a single file till this single file grows to a certain size.
Regards
Kshama
Vembu Technologies |
|
| Back to top |
|
|
bcarlos
Joined: 24 Nov 2009
Posts: 11
|
| Posted: Fri Jul 30, 2010 4:38 pm Post subject: |
|
|
| Quote: | | Each file that is backed up is stored as a separate individual file in the backup server. |
That's what's causing most of our problems. There are millions of files. It's impossible to copy them or even do a fsck on the system like this.
| Quote: | | We also working on aggregating the disk writes of multiple concurrent backups into a single stream which will write the data into a single file. |
Do you have any timeframe for this? As it is currently it's impossible to manage a Storegrid storage with several TB so we have to find another solution until Storegrid supports this. |
|
| Back to top |
|
|
kshama
Joined: 04 Sep 2007
Posts: 939
|
| Posted: Mon Aug 02, 2010 9:19 am Post subject: |
|
|
Hi bcarlos,
| Quote: | | Do you have any timeframe for this? As it is currently it's impossible to manage a Storegrid storage with several TB so we have to find another solution until Storegrid supports this. |
We expect it to be available in about 2 to 3 months time frame.
Regards
Kshama
Vembu Technologies |
|
| Back to top |
|
|
|
