Deploying StoreGrid Online Backup on Amazon EC2 / S3

Home > Solutions > Deploying StoreGrid Online Backup on Amazon EC2 / S3

A 'How To' guide that helps you run your online backup service on Amazon's Cloud

Section 1 - Introduction

Vembu StoreGrid Cloud AMI which facilitates deploying StoreGrid on Amazon's Cloud Computing Infrastructure is available to partners for production use. The StoreGrid Cloud AMI is like a virtual appliance which can be instantiated as a backup or a replication server and run on Amazon Elastic Cloud Computing (EC2) platform. The StoreGrid Cloud AMI will use Amazon Simple Storage Service (S3) to store the backup data from StoreGrid Clients.

So far, we have primarily worked with partners who are willing to host StoreGrid in their own data centers and offer online backup services to their customers. With the StoreGrid Cloud AMI Virtual Appliance, any IT solution provider can now start an online backup service using Amazon Web Services without any upfront capital investment on servers and storage infrastructure. Besides the current practice of deploying the StoreGrid backup server and replication server in their own data centers, service providers now have the added options of:

The StoreGrid Cloud AMI virtual appliance is currently available for Windows Server and CentOS Linux server. Other requisite modules like the MySQL 5.1 back-end database are bundled together in the StoreGrid Cloud AMI to facilitate ease of deployment for partners. However, as detailed below, the Amazon deployment will currently require some work on your part.

If you have questions, please email us at storegrid-cloud@vembu.com

Section 2 - Get your Amazon AWS account

  1. Point your browser to http://aws.amazon.com and click on the button ‘Sign Up Now’

  2. Enter your Amazon account details, if you have one (or select ‘I am a new user’)

  3. Enter your name and email address to register for an Amazon Web Services Account and click ‘Continue’

  4. Please enter your contact information and accept the AWS Customer Agreement

  5. You will see this screen below if you have successfully created your AWS account.

  6. Login into Amazon Web Services and select Your Account > Access Identifiers from the top menu. Note down your Amazon AWS account number from the top-right corner.

  7. You will see the ‘Access Key ID’ and your ‘Secret Access Key’ listed below. Please note them down for future use.

To proceed further, you can use AWS Management Console, ElasticFox or Amazon Command Line tools. In this guide, we will proceed with the AWS Management Console to manage your Amazon Cloud instances.

Section 3 - Launching StoreGrid Cloud AMI Instance

Section 3.1 - Generating a Secure Key Pair for Authentication

In order to remotely login to your StoreGrid Cloud AMI instance running in Amazon EC2 through SSH or Remote Desktop (RDP) you need to generate a Secure Key-Pair for secure communication. Please note, launching public AMIs (such as StoreGrid Cloud AMI) without a key pair ID will leave them inaccessible.

The Secure Key-Pair is a 2048 bit RSA key pair generated with a specified name. The generated key-pair should be specified while launching an Amazon EC2 instance and will be used for authenticating TCP connections such as SSH & RDP.

In Linux, the key pair content is used as an identity [SSH Authorization Key] to authenticate the root user connecting to the EC2 Instance.

In Windows, the key pair is used to generate and encrypt the initial random password for the EC2 Instance, such that the password is not preset in a public image. To fetch and decrypt the initial random password which is set in the EC2 instance, you have to use the Key-Pair. After fetching the password, you would be able to access the EC2 instance with UserName: Administrator and Password: <as fetched>

Create a key pair by following the below steps in AWS Management Console (If you have already created the key pair, you can ignore these steps)

NOTE: This key-pair file will be used for fetching password in a Windows instance & authenticating root user in Linux installation. Hence, this key-pair file needs to be preserved.

Section 3.2 - Launch StoreGrid EC2 instance

The AMI ID of Vembu StoreGrid Cloud AMI are as follows:

StoreGrid Cloud AMIs for the US Region
Vembu StoreGrid Cloud AMI ID for Windows [32 bit]: ami-21678a48
Vembu StoreGrid Cloud AMI ID for Windows [64 bit]: ami-eb678a82
Vembu StoreGrid Cloud AMI ID for Cent OS [32 bit]: ami-b16588d8
StoreGrid Cloud AMIs for the EU Region
Vembu StoreGrid Cloud AMI ID for Windows [32 bit]: ami-5fc3e82b
Vembu StoreGrid Cloud AMI ID for Windows [64 bit]: ami-c3c3e8b7
Vembu StoreGrid Cloud AMI ID for Cent OS [32 bit]: ami-e1cce795

Launch a StoreGrid EC2 instance by following the below steps :

Section 3.3 - Firewall Configuration

To allow the remote access to your EC2 instance, you need to grant permission to the 'default' security group which determines the ports opened up for communication in your EC2 instance.

Enable the firewall configurations by following the below steps:

The TCP port for which the permission should be granted is,

Section 3.4 - Logging to the instance

  1. For Linux instance, login to the Public DNS of the instance through SSH by executing the below command :

    ssh -i <key-pair file> root@<Public DNS Name of the instance>

    where <key-pair file> is the key-pair file to authenticate the SSH Connection with the EC2 Instance.

  2. For Windows instance, to get the instance password, please follow the below steps:

    • Click on 'Instances' link under 'Navigation' panel.

    • Right click on the particular instance and select 'Get Windows Password'.

    • Copy and paste your key pair file content in 'Private Key' field and click 'Decrypt Password'. You will get the decrypted RDP Password.

    Now, you can connect to the Windows instance using RDP (Remote Desktop or the Microsoft Terminal Services Client) with the UserName: Administrator and Password: <as fetched by the above steps>

    It is strongly recommended to change the password in the running instance after you first login to the instance.

    NOTE: Once the password is changed, you will not be able to login with the password fetched using 'Get Default Administrator Password'.

Section 3.5 - Mount EBS Volumes

To store the backup data of your clients in Amazon S3 and the metadata information of the StoreGrid Backup/Replication Server, you need to create EBS Volumes with a storage size as required. The EBS volume will be used by StoreGrid as a temporary cache location before uploading the clients' data to Amazon S3. After creating the EBS volumes, you can mount them to the EC2 instance that you have started. The attached EBS Volume also stores the MySQL Database data.

To do this, please follow the below steps:

Section 3.6 - Firewall Configurations for StoreGrid

To allow StoreGrid clients to connect to the StoreGrid Backup Server, you need to grant permission to the 'default' security group which determines the ports opened up for communication in your EC2 instance.

The TCP ports for which the permission should be granted are 32004, 32007. For accessing StoreGrid WebConsole remotely, you can open up 6060 and 6061 TCP Ports

Enable the firewall configurations by following the below steps:

The TCP port for which the permission should be granted is,

Port Number Description
32004 For default Backup Port
32007 For default SSL Backup Port
6060 For default HTTP Port of StoreGrid WebConsole
6061 For default HTTPS Port of StoreGrid WebConsole

Section 4 - StoreGrid Cloud AMI Configuration

To have a persistent configuration of the StoreGrid Server, the StoreGrid configuration values and Amazon access credentials can be stored in the EBS Volume mounted for the local backup cache. This will enable the AMI when restarted as a fresh instance to load the configuration values from the EBS Volume and start the StoreGrid with minimal manual intervention. Please follow the below steps to create the configuration as needed by StoreGrid to work with Amazon S3 as the backend storage.

  1. Create a folder 'SGCloudConf' under the attached volume in your instance.

  2. Place the your key-pair, EC2 certificate and EC2 private key files under '<Mounted_Location>/SGCloudConf' in the name of ec2-key-pair, ec2-cert.pem and ec2-priv-key.pem respectively [You can skip this step if 'EBSSnapShots' is disabled in SGConf.xml file].

  3. Create a file 'SGConf.xml' under '<Mounted_Location>/SGCloudConf'. This xml file stores the values of StoreGrid Configuration and EBS Volume snapshot schedule.

    For example, if you are going to run StoreGrid as a "Backup Server" in your EC2 instance and the unique StoreGrid ID of your EC2 instance is "backupserver@datamaniacs.com" and you would like to configure "weekly" snapshot for the volume "vol-123a4bc0" on every "Sunday at 2 AM", then your SGConf.xml should be as below:

    <StoreGrid>
    <Configuration ID="backupserver@datamaniacs.com" StartModule="601" />
    <EBSSnapShots Enabled="1">
    <Schedule Type="Weekly" Day="0" Hour="2" Mins="0" />
    <Volume ID="vol-123a4bc0" />
    </EBSSnapShots>
    </StoreGrid>
    Attribute Description
    ID The ID attribute under Configuration tag represents the unique StoreGrid ID of the StoreGrid installation.
    StartModule The StartModule attribute specifies that the StoreGrid instance to be started as 601 - Backup Server or 609 - Replication Server
    Enabled The Enabled attribute under EBSSnapShots tag represents the enable - 1 and disable - 0 of scheduled snapshots of the attached EBS Volume. It is recommended to have this value enabled, as this would make sure that there is a backup of the attached EBS Volume.
    Schedule The Schedule tag fills in the details for when to schedule the EBS Volume snapshot. For example, if you wish to do snapshot of the EBS volume daily on 2:10 AM, then you can configure as follows :
    <Schedule Type="Daily" Day="0" Hour="2" Mins="10" />
    Volume ID The Volume tag's ID attribute provides the EBS Volume's id value for taking the snapshot.

  4. Create a file 'S3Conf.xml' under '<Mounted_Location>/SGCloudConf' as follows - For example, if your access key is 'SDFD4SFDSG43', secret key is '4GSDGTSYRSY32' and the bucket name is 'datamaniacsbucket' then your S3Conf.xml file should be as follows:

    <StoreGrid>
    <S3Configuration AccessKey="SDFD4SFDSG43" SecretKey="4GSDGTSYRSY32" BucketName="datamaniacsbucket" />
    </StoreGrid>
    Attribute Description
    AccessKey AccessKey attribute stores the AWS access key.
    SecretKey SecretKey attribute stores the AWS secret key.
    BucketName Enter the name of the Amazon S3 bucket to store the backup data.

    You need to create the specified bucket (should be in lower case) in S3 for your S3 storage before starting StoreGrid. You can create the bucket by using the S3Fox (Firefox addon) or Bucket Explorer.

    The Amazon access credentials are encoded to a value understood by StoreGrid once after reading the values. The encoded values will be available in 'EncryptedS3Conf.xml' file.

  5. A log file can be located at the following location to debug the above steps,

    In Windows,
    "C:/Program Files/Vembu/StoreGrid/SGCloud/SGDaemon.log"

    In Linux,
    "/home/storegrid/Vembu/StoreGrid/SGCloud/SGDaemon.log"

    This will help identify the problem and resolve it.

  6. The logs of the Scheduled snapshots are located at,

    In Windows,
    "C:/Program Files/Vembu/StoreGrid/SGCloud/SnapShot.log"

    In Linux,
    "/home/storegrid/Vembu/StoreGrid/SGCloud/SnapShot.log"

    After StoreGrid is started, login to StoreGrid webconsole by entering http://<Public_DNS_name_of_the_instance>:6060 with the username of "admin" and password "storevembu".

Section 5 - Installing StoreGrid Client:

Download the StoreGrid v3.0 Client builds for installation in the machines you want to backup.

Supported OS Versions Size Download StoreGrid v3.0 Client Build
Windows 7 Compatibility
Windows® 7 / Vista / XP / 2000
Windows Server 2008/2003/2000, Windows SBS 2008/2003 & Windows EBS 2008
[Build No. : 3002009102219]		 20.85 MB
RedHat Linux 8.0 and later, RedHat Enterprise Linux, Fedora core 3 and later, CentOS 4.2 and later, SuSE Linux 9.x and Later
[Build No. : 3002009102219]
(Unzip the zip file and then execute the bin file) 		30.36 MB
Mandrake Linux 10.0 and Later
[Build No. : 3002009102219]
(Unzip the zip file and then execute the bin file) 		30.22 MB
Debian Linux 3.0 and later, Gentoo and Ubuntu 5.10 and later
[Build No. : 3002009102219]
(Unzip the zip file and then execute the bin file) 		30.59 MB
FREEBSD 5.4 and later
[Build No. : 3002009102219]
(Unzip the zip file and then execute the bin file) 		39.74 MB
Mac OS X 10.3.x (Panther) and 10.4.x (Tiger) for PowerPcs
Developer Tools not needed!
[Build No. : 3002009102219] 		45.31 MB
Mac OS X 10.4.5(Tiger), 10.5.x(Leopard) and 10.6.x (Snow Leopard) for Intel PCs
Developer Tools not needed!
[Build No. : 3002009102219] 		44.93 MB
Solaris (For Solaris10 - Sun and Intel)
Developer Tools not needed!
[Build No. : 3002009110216] 		32.82 MB
Installing StoreGrid Client in Amazon EC2 for Backups for mounted EBS volumes:

If you are looking at installing the StoreGrid Client build in your existing Amazon EC2 instances which you want to backup, then please do look at the instructions for installing StoreGrid Clients in an Amazon EC2 instance and configuring backups for the mounted EBS volumes.

If you have questions or run into any issues, please email us at storegrid-cloud@vembu.com

Section 6 - Seeding Backup Data:

When backup data is huge and the client cannot directly backup the data to the Backup Server running on Amazon EC2 due to limited network bandwidth in the client network, the backup data (client does a 'same machine' backup to an local external drive) can be taken to a datacenter/network which has higher upload bandwidth. The data can then be transferred (like for example through FTP) into the Backup Server machine running on EC2. The backups can then be attached to the backup server and the data migrated to the S3 through the Server Side Local to Remote Migration feature. For information on this, please refer: Server Side Local to Remote Migration

Section 7 - Migrating from older version to StoreGrid v3.0:

Existing partners, who are currently running StoreGrid v2.5 or v2.5.5 on EC2, cannot migrate to the latest v3.0. Migration from v2.x to v3.0 will be supported in the future.

Amazon Cloud
Pricing for StoreGrid Service Provider Edition
Call Me Back
© 2004-2010 Vembu Technologies Pvt. Ltd. All Rights Reserved. Trademarks | Privacy Policy