Deploying StoreGrid Online Backup on Amazon EC2 / S3

Home > Solutions > Deploying StoreGrid Online Backup on Amazon EC2 / S3

A 'How To' guide that helps you run your online backup service on Amazon's Cloud

Section 1 - Introduction

Vembu StoreGrid Cloud AMI which facilitates deploying StoreGrid on Amazon's Cloud Computing Infrastructure is available to partners for production use. The StoreGrid Cloud AMI is like a virtual appliance which can be instantiated as a backup or a replication server and run on Amazon Elastic Cloud Computing (EC2) platform. The StoreGrid Cloud AMI will use Amazon Simple Storage Service (S3) to store the backup data from StoreGrid Clients.

So far, we have primarily worked with partners who are willing to host StoreGrid in their own data centers and offer online backup services to their customers. With the StoreGrid Cloud AMI Virtual Appliance, any IT solution provider can now start an online backup service using Amazon Web Services without any upfront capital investment on servers and storage infrastructure. Besides the current practice of deploying the StoreGrid backup server and replication server in their own data centers, service providers now have the added options of:

The StoreGrid Cloud AMI virtual appliance is currently available for Windows Server and CentOS Linux server. Other requisite modules like the MySQL 5.1 back-end database are bundled together in the StoreGrid Cloud AMI to facilitate ease of deployment for partners. However, as detailed below, the Amazon deployment will currently require some work on your part.

If you have questions, please email us at storegrid-cloud@vembu.com

Section 2 - Overview of Amazon Deployment:

Vembu StoreGrid Cloud AMI is an Amazon Machine Instance of StoreGrid. This AMI would enable Service providers and MSPs to use the Amazon cloud infrastructure as their data center to launch their online backup service business. This document will guide you with the steps to run StoreGrid Backup/Replication Server as an Amazon EC2 Webservice. Vembu StoreGrid has public AMIs (called StoreGrid Cloud AMI) for both Windows and CentOS to allow Service Providers or MSPs to run their own EC2 instance of StoreGrid Cloud AMI in the Amazon cloud infrastructure.

The AMI ID of Vembu StoreGrid Cloud AMI are as follows:

StoreGrid v3.0 Cloud AMIs for the US Region
Vembu StoreGrid Cloud AMI ID for Windows [32 bit] : ami-21678a48
Vembu StoreGrid Cloud AMI ID for Windows [64 bit] : ami-eb678a82
Vembu StoreGrid Cloud AMI ID for Cent OS  [32 bit] : ami-b16588d8
StoreGrid v3.0 Cloud AMIs for the EU Region
Vembu StoreGrid Cloud AMI ID for Windows [32 bit] : ami-5fc3e82b
Vembu StoreGrid Cloud AMI ID for Windows [64 bit] : ami-c3c3e8b7
Vembu StoreGrid Cloud AMI ID for Cent OS  [32 bit] : ami-e1cce795

Note: You have to signup for the Amazon WebServices and to use Amazon Elastic Compute Cloud (EC2) and the Amazon Simple Storage Service (S3) Web Services. After signing up, you will be able to launch StoreGrid Cloud AMI instances with your Amazon Account's access key and secret key for the Web Services.

Launching and configuring your own instance of StoreGrid backup server or replication server using the StoreGrid Cloud AMI involves the following steps (each of these steps is detailed further down in this document):

  1. You need to install the Amazon EC2 command-line tools in your machine. This is required to execute commands in your Amazon Web Services account to start AMI instances and also to manage your AMI instances. Also, you need to generate a secure key-pair using the Amazon EC2 command-line tools. This key-pair is used for remotely accessing your EC2 instance in a secure manner.

  2. Launch your StoreGrid Cloud AMI instance and login to your instance. Mount Amazon Elastic Block Store [EBS] volumes for local caching of the backup data before uploading to Amazon S3 and configure it.

  3. Configure MySQL installation to use the mounted EBS Volumes for storage and start the MySQL Server.

  4. Customize StoreGrid installation configuration in your EC2 instance and update StoreGrid to use Amazon S3 for storage and the mounted EBS Volumes for local caching of backup data before uploading to Amazon S3. Also, configure StoreGrid Cloud AMI to accept backups from clients. Or if it is a replication server configure it to accept replication data from StoreGrid backup servers. And start StoreGrid Server.

Section 3 - Install Amazon EC2 command-line tools:

As the first step you need to install the Amazon EC2 command-line tools in your machine. This will enable you to start the StoreGrid Cloud AMI instance in Amazon EC2 and also manage it from your machine. This tool is available from the URL http://developer.amazonwebservices.com/connect/entry.jspa?externalID=351&categoryID=88

To know more about the Amazon EC2 command-line tools and the EC2 WebService, please follow the URL to view/download the Amazon EC2 Developer Guide, http://developer.amazonwebservices.com/connect/entry.jspa?externalID=1765&categoryID=118

For using Amazon EC2 command-line tools in your machine you need the following installed and setup in your machine as pre-requisites:

  1. Install Java JRE from http://java.sun.com and make sure the JAVA_HOME environment variable is updated with the Java installation path. This is required as Amazon EC2 command-line tools need Java to run.

  2. All command-line instructions to Amazon Web Services below will need a Certificate File and a Private Key File provided by Amazon while signing up for the Amazon WebServices (EC2, S3, ...). These files will be used for authenticating the User for doing the Amazon Web Services operations. Hence you would have to add the following options to all your command line instructions.

    -C <User's Certificate file> -K <User's Private Key File>

    OR, you could add the Certificate File and the Private key file locations to the environment variables EC2_CERT & EC2_PRIVATE_KEY respectively.

  3. Export EC2_HOME environment variable with the EC2 API Tools path.

    In Windows,
    set EC2_HOME=C:\ec2-api-tools
    In Linux,
    export EC2_HOME=/home/test/ec2-api-tools

  4. Export EC2_URL environment variable with the Amazon EC2 end-point to be used for the US/Europe Amazon EC2 datacenter

    In Windows,
    For US,
    set EC2_URL=http://us-east-1.ec2.amazonaws.com
    For Europe,
    set EC2_URL=http://eu-west-1.ec2.amazonaws.com
    In Linux,
    For US,
    export EC2_URL=http://us-east-1.ec2.amazonaws.com
    For Europe,
    export EC2_URL=http://eu-west-1.ec2.amazonaws.com

  5. For convenient usage of the Amazon EC2 command-line tools, add the bin folder to the PATH environment variable

    In Windows,
    set PATH=C:\ec2-api-tools\bin;%PATH%
    In Linux,
    export PATH=/home/test/ec2-api-tools/bin:$PATH
Section 3.1 - Generate a Secure Key-Pair:

In order to remotely login to your StoreGrid Cloud AMI instance running in Amazon EC2 through SSH or Remote Desktop (RDP) you need to generate a Secure Key-Pair for secure communication. Please note that, launching public AMIs (such as StoreGrid Cloud AMI) without a key pair ID will leave them inaccessible.

The Secure Key-Pair is a 2048 bit RSA key pair generated with a specified name. The generated key-pair should be specified while launching an Amazon EC2 instance and will be used for authenticating TCP connections such as SSH & RDP.

In Linux, the key pair content is used as an identity [SSH Authorization Key] to authenticate the root user connecting to the EC2 Instance.

In Windows, the key pair is used to generate and encrypt the initial random password for the EC2 Instance, such that the password is not preset in a public image. To fetch and decrypt the initial random password which is set in the EC2 instance, you have to use the Key-Pair. After fetching the password, you would be able to access the EC2 instance with UserName: Administrator and Password: <as fetched>

The key-pair can be generated by the following steps:

  1. In the EC2 command line tool, Run the command 'ec2-add-keypair <keypair-name>'.

    where, the <keypair-name> is the name you select for the key pair.

    The resulting private key is displayed as output of the command.

  2. Open a text editor.

  3. Paste the entire private key, starting with the line "-----BEGIN RSA PRIVATE KEY-----" and ending with the line "-----END RSA PRIVATE KEY-----".

  4. Save the file and preserve it for using while authenticating SSH or RDP connections to your Instance.

NOTE: This key-pair file will be used for fetching password in a Windows instance & authenticating root user in Linux installation. Hence, this key-pair file needs to be preserved.

Section 4 - Launching the StoreGrid Cloud AMI Instance:

You can launch a new StoreGrid Cloud AMI instance in EC2, using the command 'ec2-run-instances' as follows :

ec2-run-instances -k <key-pair name> <US/EU SG_AMI_ID> -z <availability zone>

where, key-pair name is the name specified while creating the Secure Key-Pair.

The AMI ID of Vembu StoreGrid Cloud AMI are as follows:

StoreGrid v3.0 Cloud AMIs for the US Region
Vembu StoreGrid Cloud AMI ID for Windows [32 bit] : ami-21678a48
Vembu StoreGrid Cloud AMI ID for Windows [64 bit] : ami-eb678a82
Vembu StoreGrid Cloud AMI ID for Cent OS  [32 bit] : ami-b16588d8
StoreGrid v3.0 Cloud AMIs for the EU Region
Vembu StoreGrid Cloud AMI ID for Windows [32 bit] : ami-5fc3e82b
Vembu StoreGrid Cloud AMI ID for Windows [64 bit] : ami-c3c3e8b7
Vembu StoreGrid Cloud AMI ID for Cent OS  [32 bit] : ami-e1cce795

Now, you will get an instance ID for the instance just launched [i-########]. Now, you can fetch the Public DNS name of the instance by executing the following command,

ec2-describe-instances i-########

This will fetch you all the details regarding the instance which is launched. Make sure the instance has entered running mode, as the Public DNS Name will be available only after the instance has entered running mode. The Public DNS Name will enable you to remotely access the EC2 instance.

Section 4.1 - Firewall Configuration:

To allow the remote access to your EC2 instance, you need to grant permission to the 'default' security group which determines the ports opened up for communication in your EC2 instance.

The TCP port for which the permission should be granted is,

Section 4.2 - Logging in to your StoreGrid Cloud AMI instance:

For Linux instance:

Once the instance is running, login to the Public DNS of the instance through SSH as follows :

ssh -i <key-pair file> root@<Public DNS Name of the instance>

-i <key-pair file> uses the key-pair file to authenticate the SSH Connection with the EC2 Instance

For Windows instance:

The key-pair used while starting the EC2 instance will generate a random password for the 'Administrator' user for your Windows instance. This one-time random password has to be fetched for the first login after the windows instance is started and ready. You can use the following command to fetch the random password,

ec2-get-password <Instance ID> -k <key-pair file path>

where, <key-pair file path> is the location of the file you saved with the RSA Key Pair data while creating the Key Pair.

Now, you can connect to the Windows instance using RDP (Remote Desktop or the Microsoft Terminal Services Client) with the UserName: Administrator and Password: <as fetched by the ec2-get-password command>

It is strongly recommended to change the password in the running instance after you first login to the instance.

NOTE: Once the password is changed, you will not be able to login with the password fetched using ec2-get-password. If ec2-get-password operation is performed again, it will not fetch the updated password.

Section 4.3 - Mount EBS Volumes:

To store the backup data of your clients in Amazon S3 and the metadata information of the StoreGrid Backup/Replication Server, you need to create EBS Volumes with a storage size as required. The EBS volume will be used by StoreGrid as a temporary cache location before uploading the clients' data to Amazon S3. After creating the EBS volumes, you can mount them to the EC2 instance that you have started. The attached EBS Volume also stores the MySQL Database data.

To do this, please follow the below steps :

  1. Create a new EBS volume as follows :
    ec2-create-volume -s <size in GB> -z <availability zone>

    Make sure the <availability zone> specified here is same as the availability zone of the instance started.

    Ex : ec2-create-volume -s 10 -z us-east-1a

    The above command will return a volume id.

  2. Now, attach the EBS volume to the instance as follows :
    ec2-attach-volume <volume id> -i <instance id> -d /dev/sdh

    where, /dev/sdh is the device which would get attached to your Linux instance. You can use /dev/sd# based on the instance-type used to start the EC2 instance.

    In Windows, please use device name as, xvd# where hash(#) can be replaced by a,b,c,...,p. In all types of instances, xvda, xvdb, xvdc are reserved. In large and extra-large instances, the xvdd & xvde are reserved.

  3. Once the EBS volume is attached,

    In Linux, login to instance via SSH and execute the following
    yes | mkfs -t xfs /dev/sdh
    mount /dev/sdh /storegrid

    where /dev/sdh is the device name and /storegrid is the mount point.

    In Windows,

    1. On the taskbar, click Start, and then click Run.
    2. Type diskmgmt.msc and click OK. The Disk Management utility opens.
    3. Right-click the Amazon EBS volume, select New Volume, and follow the on-screen prompts

    Any data written to this file system is written to the Amazon EBS volume and is transparent to applications using the device.

    NOTE: If the EBS Volume mounted is an existing volume which already contains data(StoreGrid Cloud AMI configuration xml files), the EBS Volume need not be formatted and filesystem created, instead EBS Volume can be just attached to the instance.

    In Linux, once attached, the EBS Volume should be mounted to the specified mount location.

        mount /dev/sdh /storegrid

    NOTE: In Linux, the Mounted Location should be '/storegrid' such that the StoreGrid automated script, recognizes the StoreGrid configurations.

Section 4.4 - Firewall Configuration for StoreGrid:

To allow StoreGrid clients to connect to the StoreGrid Backup Server, you need to grant permission to the 'default' security group which determines the ports opened up for communication in your EC2 instance.

The TCP ports for which the permission should be granted are 32004, 32007. For accessing StoreGrid WebConsole remotely, you can open up 6060 and 6061 TCP Ports. You can do this with the following commands :

ec2-authorize default -p 32004 [For default Backup Port]
ec2-authorize default -p 32007 [For default SSL Backup Port]
ec2-authorize default -p 6060 [For default HTTP Port of StoreGrid WebConsole]
ec2-authorize default -p 6061 [For default HTTPS Port of StoreGrid WebConsole]

Section 5 - StoreGrid Cloud AMI Configuration:

To have a persistent configuration of the StoreGrid Server, the StoreGrid configuration values and Amazon access credentials can be stored in the EBS Volume mounted for the local backup cache. This will enable the AMI when restarted as a fresh instance to load the configuration values from the EBS Volume and start the StoreGrid with minimal manual intervention. Please follow the below steps to create the configuration as needed by StoreGrid to work with Amazon S3 as the backend storage.

  1. Create a folder 'SGCloudConf' under the attached volume in your instance.

  2. Place the your key-pair, EC2 certificate and EC2 private key files under '<Mounted_Location>/SGCloudConf' in the name of ec2-key-pair, ec2-cert.pem and ec2-priv-key.pem respectively [You can skip this step if 'EBSSnapShots' is disabled in SGConf.xml file].

  3. Create a file 'SGConf.xml' under '<Mounted_Location</SGCloudConf' as follows: For example, if you are going to run StoreGrid as a "Backup Server" in your EC2 instance and the unique StoreGrid ID of your EC2 instance is "backupserver@datamaniacs.com" and you would like to configure "weekly" snapshot for the volume "vol-123a4bc0" on every "Sunday at 2 AM", then your SGConf.xml should be as below:

    <StoreGrid>
      <Configuration ID="backupserver@datamaniacs.com" StartModule="601" />
      <EBSSnapShots Enabled="1">
        <Schedule Type="Weekly" Day="0" Hour="2" Mins="0" />
        <Volume ID="vol-123a4bc0" />
      </EBSSnapShots>
    </StoreGrid>

    This xml file stores the values of StoreGrid Configuration and EBS Volume snapshot schedule.

    The ID attribute under Configuration tag represents the unique StoreGrid ID of the StoreGrid installation.

    The StartModule attribute specifies that the StoreGrid instance to be started as 601 - Backup Server or 609 - Replication Server. In the above example, if you want to run it as a Replication Server instead, then change 601 to 609.

    The Enabled attribute under EBSSnapShots tag represents the enable - 1 and disable - 0 of scheduled snapshots of the attached EBS Volume. It is recommended to have this value enabled, as this would make sure that there is a backup of the attached EBS Volume.

    The Schedule tag fills in the details for when to schedule the EBS Volume snapshot. For example, if you wish to do snapshot of the EBS volume daily on 2:10 AM, then you can configure as follows :

    <Schedule Type="Daily" Day="0" Hour="2" Mins="10" />

    The Volume tag's ID attribute provides the EBS Volume's id value for taking the snapshot.

  4. Create a file 'S3Conf.xml' under '<Mounted_Location>/SGCloudConf' as follows - For example, if your access key is 'SDFD4SFDSG43', secret key is '4GSDGTSYRSY32' and the bucket name is 'datamaniacsbucket' then your S3Conf.xml file should be as follows:

    <StoreGrid>
      <S3Configuration AccessKey="SDFD4SFDSG43" SecretKey="4GSDGTSYRSY32" BucketName="datamaniacsbucket" />
    </StoreGrid>

    Here, the 'BucketName' attribute specifies the bucket name in which the data will be stored in Amazon S3 storage. You need to create the specified bucket (should be in lower case) in S3 for your S3 storage before starting StoreGrid. You can create the bucket by using the S3Fox (Firefox addon) or Bucket Explorer.

    The amazon access credentials are encoded to a value understood by StoreGrid once after reading the values. The encoded values will be available in 'EncryptedS3Conf.xml' file.

  5. A log file can be located at the following location to debug the above steps,

    In Windows,
    "C:/Program Files/Vembu/StoreGrid/SGCloud/SGDaemon.log"
    In Linux,
    "/home/storegrid/Vembu/StoreGrid/SGCloud/SGDaemon.log"

    This will help identify the problem and resolve it.

  6. The logs of the Scheduled snapshots are located at,

    In Windows,
    "C:/Program Files/Vembu/StoreGrid/SGCloud/SnapShot.log"
    In Linux,
    "/home/storegrid/Vembu/StoreGrid/SGCloud/SnapShot.log"

After StoreGrid is started, login to StoreGrid webconsole by entering <http://Public_DNS_name_of_the_instance:6060> with the username of "admin" and password "storevembu".

Section 6 - Installing StoreGrid Client:

Download the StoreGrid v3.0 Client builds for installation in the machines you want to backup.

Supported OS Versions Size Download StoreGrid v3.0 Client Build
Windows 7 Compatibility
Windows® 7 / Vista / XP / 2000
Windows Server 2008/2003/2000, Windows SBS 2008/2003 & Windows EBS 2008
[Build No. : 3002009102219]		 20.85 MB
RedHat Linux 8.0 and later, RedHat Enterprise Linux, Fedora core 3 and later, CentOS 4.2 and later, SuSE Linux 9.x and Later
[Build No. : 3002009102219]
(Unzip the zip file and then execute the bin file) 		30.36 MB
Mandrake Linux 10.0 and Later
[Build No. : 3002009102219]
(Unzip the zip file and then execute the bin file) 		30.22 MB
Debian Linux 3.0 and later, Gentoo and Ubuntu 5.10 and later
[Build No. : 3002009102219]
(Unzip the zip file and then execute the bin file) 		30.59 MB
FREEBSD 5.4 and later
[Build No. : 3002009102219]
(Unzip the zip file and then execute the bin file) 		39.74 MB
Mac OS X 10.3.x (Panther) and 10.4.x (Tiger) for PowerPcs
Developer Tools not needed!
[Build No. : 3002009102219] 		45.31 MB
Mac OS X 10.4.5(Tiger), 10.5.x(Leopard) and 10.6.x (Snow Leopard) for Intel PCs
Developer Tools not needed!
[Build No. : 3002009102219] 		44.93 MB
Solaris (For Solaris10 - Sun and Intel)
Developer Tools not needed!
[Build No. : 3002009110216] 		32.82 MB
Installing StoreGrid Client in Amazon EC2 for Backups for mounted EBS volumes:

If you are looking at installing the StoreGrid Client build in your existing Amazon EC2 instances which you want to backup, then please do look at the instructions for installing StoreGrid Clients in an Amazon EC2 instance and configuring backups for the mounted EBS volumes.

If you have questions or run into any issues, please email us at storegrid-cloud@vembu.com

Section 7 - Seeding Backup Data:

When backup data is huge and the client cannot directly backup the data to the Backup Server running on Amazon EC2 due to limited network bandwidth in the client network, the backup data (client does a 'same machine' backup to an local external drive) can be taken to a datacenter/network which has higher upload bandwidth. The data can then be ftped into the Backup Server machine running on EC2. The backups can then be attached to the backup server and the data migrated to the S3 through the Server Side Local to Remote Migration feature. For information on this, please refer: Server Side Local to Remote Migration

Section 8 - Migrating from older version to StoreGrid v3.0:

Existing partners who are currently running StoreGrid v2.5 or v2.5.5 on EC2, cannot migrate to the latest v3.0. Migration from v2.x to v3.0 will be supported in the future.

Amazon Cloud
Pricing for StoreGrid Service Provider Edition
Call Me Back
© 2004-2010 Vembu Technologies Pvt. Ltd. All Rights Reserved. Trademarks | Privacy Policy