wiffu

Posts tagged data storage
Explore data storage on posterous

Posted

Amazon Simple Storage Service [S3]

What is S3??

Simple Storage Service: Is a web service that allows any developer to gain access to highly scalable, very reliable, inexpensive storage space. Your data is replicated to multiple servers at multiple data centers.

How to get started

Go to Amazon’s AWS page, then to the S3 page and sign up (check out the other services while you are there)

Pricing (From Amazon’s site…)

Pricing

…

New Pricing (effective June 1st, 2007)

Storage
$0.15 per GB-Month of storage used

Data Transfer
$0.10 per GB - all data uploaded

$0.18 per GB - first 10 TB / month data downloaded
$0.16 per GB - next 40 TB / month data downloaded
$0.13 per GB - data downloaded / month over 50 TB

Data transferred between Amazon S3 and Amazon EC2 is free of charge

Requests
$0.01 per 1,000 PUT or LIST requests
$0.01 per 10,000 GET and all other requests*
* No charge for delete requests

Storage and bandwidth size includes all file overhead

I looked around the web for similar services (hard to find someone that posts prices), and for 180 gig’s of reduntantly stored data, it was in the $200/month price range.

The same 180Gb on S3 would be

$27 to store for 30 days

$18 to xmit (the entire 180Gb) to S3

In addition to price, with S3, you are in full control of how and when you put and/or get your data.

Only pay for what you actually use

One of the niceties about S3 (and the other Amz web services) is that you pay just for what you use.

Didn’t use that service last month: Pay $0.

This allows you to ‘tinker’ all you want for mere pennies

Amazon S3 - Objects

What is an Object?

Object is the term we use in S3 for the ‘thing’ (file/data) you want to store.
Once an object is stored in S3, it contains the original data (contents of the file), plus a given amount of meta-data (name/value pairs).

You can add your own metadata but some of the standards are ‘Last-Modified‘ and ‘Content-Type‘

A given Object can be from 1byte to 5GBs

Amazon S3 - Buckets

Why Buckets?

Buckets provides a unique namespace for management of objects contained in the bucket

Bucket namespaces are Global across all of S3 (all users of S3. Similar concept as ‘domain names‘)

An S3 account is allowed 100 buckets

Amazon S3 - Keys

Key

A key is the unique identifier for an object within a bucket

Locating an object

Any Object can be located by its [bucket + key] using a RESTful formatted URL

http://s3.amazonaws.com/foo-products/2006/may/1845.prd

foo-products is the bucket & 2006/may/1845.prd is the Key

S3 - Authentication

Most requests to S3 require authentication, this ensures that you don’t get charged for operations you didn’t authorize, and that nobody else sees your private data.

You can grant various access models (acl) for an Object or an entire Bucket

private
public-read
public-read-write
authenticated-read

To set the ACL, when you PUT the Object to S3, you set a x-amz-acl header. For example…

x-amz-acl: public-read

ACl defaults to private if not set on the PUT

S3: Putting it all Together

How do you speak to S3?

At this point, all interaction is done with the HTTP protocol (the current exception is that you can retrieve objects using http or BitTorrent).

So, creating a program to interact with S3 is just a matter of creating HTTP requests and reading HTTP responses. Something PHP is quite capable of (especially with a little help from PEAR HTTP_Request and Crypt_HMAC)

First get an account and your Keys

Access Key ID: You add this to any requests to S3. Essentially it is your unique identifier that tells S3 a given request is targeted for your account.
Secret Access Key: For requests to S3 for Objects with acl’s that require authentication (i.e. private, authenticated-read), you ’sign’ your request with this secret key.

The following code examples are based on what is in the Amazon S3 developer docs

Create a Bucket

# create bucket requestPUT /[bucket-name] HTTP/1.0Date: Wed, 08 May 2007 08:45:09 GMTAuthorization: AWS [aws-access-key-id]:[header-signature]Host: s3.amazonaws.com# create bucket responseHTTP/1.1 200 OKx-amz-id-2: VjzdTviQorQtSjcgLshzCZSzN+7CnewvHA+6sNxR3VRcUPyO5fmSmo8bWnIS52qax-amz-request-id: 91A8CC60F9FC49E7Date: Wed, 08 Mar 2006 04:06:15 GMTLocation: /[bucket-name]Content-Length: 0Connection: keep-aliveServer: AmazonS3

Put Objects in your Bucket

# put object requestPUT /[bucket-name]/[key-name] HTTP/1.0Date: Wed, 08 Mar 2006 04:06:16 GMTAuthorization: AWS [aws-access-key-id]:[header-signature]Host: s3.amazonaws.comContent-Length: 14x-amz-meta-title: my titleContent-Type: text/plainthis is a test# put object responseHTTP/1.1 200 OKx-amz-id-2: wc15E1LUrjDZhNtT4QZtsbtadnOMKGjw5QTxkRDVO1owwbA6YoiqJJEuKShopufwx-amz-request-id: 7487CD42C5CA7524Date: Wed, 08 Mar 2006 04:06:16 GMTETag: "54b0c58c7ce9f2a8b551351102ee0938"Content-Length: 0Connection: keep-aliveServer: AmazonS3

Retrieve Objects from your bucket

# get object request

GET /[bucket-name]/[key-name] HTTP/1.0Date: Wed, 08 Mar 2006 04:06:18 GMTAuthorization: AWS [aws-access-key-id]:[header-signature]Host: s3.amazonaws.com

# get object response

HTTP/1.1 200 OKx-amz-id-2: FbGpiykb9oJEdJd0bcfwkL6S3lc06X0y7XSeA/GWyRdvlNEZ0irthljxKoeGFfB6x-amz-request-id: 9298531013923634Date: Wed, 08 Mar 2006 04:06:18 GMTLast-Modified: Wed, 08 Mar 2006 04:06:16 GMTETag: "54b0c58c7ce9f2a8b551351102ee0938"x-amz-meta-title: my titleContent-Type: text/plainContent-Length: 14Connection: keep-aliveServer: AmazonS3

this is a test

S3 - The PHP way

Implementing an API to S3 with PHP

Prerequisites

You’ll need the PEAR libraries Crypt_HMAC & HTTP_Request (at least things are much easier if you have these)

# sudo pear install Crypt_HMACpear.php.net" to updatedownloading Crypt_HMAC-1.0.1.tgz ...Starting to download Crypt_HMAC-1.0.1.tgz (2,149 bytes)....done: 2,149 bytesinstall ok: channel://pear.php.net/Crypt_HMAC-1.0.1

sam$ sudo pear install HTTP_Requestpear.php.net" to updatedownloading HTTP_Request-1.4.0.tgz ...Starting to download HTTP_Request-1.4.0.tgz (15,262 bytes).....done: 15,262 bytesdownloading Net_URL-1.0.14.tgz ...Starting to download Net_URL-1.0.14.tgz (5,173 bytes)...done: 5,173 bytesdownloading Net_Socket-1.0.7.tgz ...Starting to download Net_Socket-1.0.7.tgz (5,419 bytes)...done: 5,419 bytesinstall ok: channel://pear.php.net/Net_URL-1.0.14install ok: channel://pear.php.net/Net_Socket-1.0.7install ok: channel://pear.php.net/HTTP_Request-1.4.0

Creating the API

At this point all you really need to do is create a function for each needed interaction with S3 (or better yet, a PHP Object with a method for each). So something like…

createBucket()

putObject()

getObject()

getBucketListing()

...

These functions are going to be creating http requests and reading http responses. Sometimes this can be a bit tricky (one missing ‘\n’ and you’re screwed), so leverage what what other have done befor you. The Amazon web services site has some good examples but in particular, I would recommend you look at ‘Test Utility for Amazon S3 in PHP‘ which does a good job of demo’ing most of the S3 functionality using PHP.

I used this code as a starting point to develop a very simple ‘Rsync’ type application for Amazon S3.

Resources

Documentation from Amazon
Example PHP implementation
Who’s Using S3 here and here
My Rsync App (Proof of Concept at this point)

Tagged data storage

Posted

Find and Replace in Mysql

Simple but very useful MySQL trick... UPDATE `table` SET `field` = REPLACE ( `field` , '{string to find}', '{replacement string}' )

Tagged data storage

Posted

PHP Data Objects

DataObject is a design pattern that incapsualtes the database with a simple to use interface. This alows the Controller to interact with the database layer in an SQL'less way. To acomplish this you build a base DataObject abstract class that implements the base functionality and sets up the interface for a Data Object. Then for each of your 'Business Objects' (such as customer), you extend the DataObject class and implement the various methods provided by DataObject. (see attached code for details).

You'll in the posted PHP files below that the code gets more 'complex' as we move down the stack (Controller->BusinessObject->DataObject). This is the beauty of OOP since we have implemented the DataObject only once in our application (it may be used by many Business Objects). As long as we maintain its interface, we are free to refactore its implementation details and add functionallity. And there is only one file to change.

Retrival of object from the database

In the simple case of the controller needing to retrieve a customer. Traditionally this might involve building a query of something like...

This process involes the Controller having far too much knowelge of the Database layer. This tight coupling should be avoided.

The same process using a database object would go something like this...

Search the database for an Object

This is quite elegant with DO's. Just simply create a new DO, set what you do know about what you are seaching for then execute the object find method.

Saving and Updating Objects

Again this is very simple also. For a populated customer DO object, just issue its insert method and to update, use the update method.

Here is a Proof of Concept Controller PHP script

/** * This is just a Proof Of Concept, this is not Production code. * * This is the controller script, that in a very simple * way allows you to Insert/update and delete customers * using the Data Object Class. */ require("../includes/bootstrap.php"); require_once('CustomerDO.php'); $feedback = ""; // collect the id of the image to edit $customer_id = isset($_GET['customer_id']) ? hd($_GET['customer_id']) : null; /** * Create the CustomerDO object. The DB connection * is an argument of the constructor (Dependency Injection) */ $customer = new CustomerDO($db); /** * If there is a customer id provided, retrieve it from the DB */ if ($customer_id) { $customer->get($customer_id); } if (isset($_GET['submit'])) { $customer->first_name = hd($_GET['first_name']); $customer->last_name = hd($_GET['last_name']); if($customer->customer_id){ $customer->update(); $feedback = "Update Complete"; } else { $customer->insert(); $feedback = "Creation Complete"; } } else if(isset($_GET['delete']) && $customer->customer_id) { $customer->delete(); $feedback = "Delete Complete"; } // Set values for the template $first_name = $customer->first_name; $last_name = $customer->last_name; $customer_id = $customer->customer_id; // Just using simple include rather than template engine(i.e. Smarty) include("customer/index.tpl");

Here is the Customer Data Object that extends DataObject

require_once('DataObject.php'); /** * This is the Business Object class for Customer. * It extends the DataObject class and implemets * the needed interface methods of DataObject * */ class CustomerDO extends DataObject { /** * Specific C.R.U.D. queries for Customer * */ const QUERY_GET = " SELECT * FROM customer WHERE customer_id = ?"; const QUERY_INSERT = " INSERT INTO customer SET `customer_id` = ?, `first_name` = ?, `last_name` = ?, `create_date` = NOW(), `modified_date` = NOW()"; const QUERY_UPDATE = " UPDATE customer SET `first_name` = ?, `last_name` = ?, `modified_date` = NOW() WHERE customer_id = ?"; const QUERY_DELETE = " DELETE FROM customer WHERE customer_id = ?"; const QUERY_FIND = " SELECT * FROM customer"; /** * Allow for an optional Database handle param. This * Inversion Of Control allows for ease of unit testing. i.e. you * can supplay a mock database object * * @param Object $db DatabaseHandle */ public function __construct($db=null) { $this->init(); $this->db = $db; } /** * Set up the initial values for this objects * attributes. * */ private function init() { $this->attributes = array( 'customer_id' => '', 'first_name' => '', 'last_name' => '' ); } /** * Retrieve a specific object by its primary key * * @param mixed $data_object_id */ public function get($data_object_id) { parent::get($data_object_id, self::QUERY_GET); } /** * Persist a new object to the DB * */ public function insert() { $this->customer_id = $this->db->nextId(parent::DB_SEQUENCE_TABLE); $params = array( $this->customer_id, $this->first_name, $this->last_name ); $result = $this->db->query(self::QUERY_INSERT ,$params); db_check_error($result); } /** * Update an existing Object in the DB * */ public function update() { $params = array( $this->first_name, $this->last_name, $this->customer_id ); $result = $this->db->query(self::QUERY_UPDATE ,$params); db_check_error($result); } /** * Permanently remove an object from persistence * */ public function delete() { $params = array($this->customer_id); $row = $this->db->query(self::QUERY_DELETE ,$params); db_check_error($row); } /** * Query for all records matching the state of the current * Object. * * @param boolean $as_detached_recordset * @param string $order_by_field "{fieldname} {ASC | DESC}" * @param string $group_by_fields "{fieldname1, fieldname2...}" * @return ReadOnlyResultSet or array if $as_detached_recordset = true */ public function find($as_detached_recordset=false, $order_by_field=false, $group_by_fields=false, $return_structure=false) { return parent::find(self::QUERY_FIND, $as_detached_recordset, $order_by_field, $group_by_fields,$return_structure); } /** * Utility function to retrieve the number of objects in persistance * matching a given 'find' * * @return unknown */ public function count() { return count($this->find(true)); } }

And here is the Base Data Object Class

/** * This is the base DataObject Class. * * @package default * @author Sam **/ abstract class DataObject { protected $attributes; protected $db; // for debugging use public $sql; protected $attrib_to_dbfield_mapping = array(); /** * Default get implementation * * @param unknown_type $newsletter_id */ protected function get($do_id, $get_query) { $params = array($do_id); $row = $this->db->getRow($get_query ,$params,DB_FETCHMODE_ASSOC); db_check_error($row); if ($row) { foreach ($this->attributes as $class_attribute => $value) { if (key_exists($class_attribute,$row)) { $this->$class_attribute = $row[$class_attribute]; } } } } protected abstract function insert(); protected abstract function update(); protected abstract function delete(); protected abstract function count(); const DB_SEQUENCE_TABLE = 'sequence_ids'; /** * @todo implement OR, right now only AND is supported... */ const CONJUNCTION_AND = 'AND'; const CONJUNCTION_OR = 'OR'; public function __get ($attribute) { if (! key_exists($attribute,$this->attributes)) { throw new Exception("Attribute: [$attribute], not defined for ".__CLASS__); } return $this->attributes[$attribute]; } public function __set ($attribute, $value) { if (! key_exists($attribute,$this->attributes)) { throw new Exception("Attribute: [$attribute], not defined for ".__CLASS__); } $this->attributes[$attribute] = $value; } public function getAttributeList() { return array_keys($this->attributes); } /** * Used to build clauses for sql queries. * Assumptions: * - All date values are supplied as UNIX timestamps * * @param string $field_name * @param string $clause The expected format of the $clause param is * - '{ = |> } {value}' * - i.e. ">= 17645329" * - i.e. "!= Y" * - i.e. "Y" // = (equals) is inferred by default * a space is required to seperate the boolean clause and * the value. If no boolean clause is given, '=' is * assumed. */ protected function buildWhereClause($field_name, $clause) { $clause_boolean = '='; //default is equals '=' $clause_righthand_value = $clause; $hack_booleans = array('=','','=','!=',''); foreach ($hack_booleans as $boolean) { if (strpos($clause,$boolean)===0) { $clause_boolean = substr($clause,0,strlen($boolean)); $clause_righthand_value = trim(substr($clause,strlen($boolean)+1)); break; } } } /** * check date special case (righthand value should be a timestamp) * - be sure to use the attribute name if it exists (rather than the db field name). That way we can * rely on the '_date' naming convention. */ $date_attribute_name = array_search($field_name,$this->attrib_to_dbfield_mapping) ? array_search($field_name,$this->attrib_to_dbfield_mapping) : $field_name; if (strstr($date_attribute_name,'_date')) { if (! is_numeric($clause_righthand_value)) { throw new Exception("The righthand value for [{$field_name}]: not in correct format. Should be {boolean} {unix timestamp}. (" ."i.e. usage: 'expire_date' => '>= '.time())"); } $where = "`{$field_name}` {$clause_boolean} FROM_UNIXTIME('" . mysql_escape_string($clause_righthand_value) . "')"; // Not date, so put clause together normally } else { $where = "`{$field_name}` {$clause_boolean} '" . mysql_escape_string($clause_righthand_value) . "'"; } return $where; } /** * Maps DO attribute names to DB field names. If a mapping for * the given attribute does not exist, the attribute name is simply * returned * * @param string $attribute_name * @return string */ protected function mappedAttributeName($attribute_name) { return isset($this->attrib_to_dbfield_mapping[$attribute_name]) ? $this->attrib_to_dbfield_mapping[$attribute_name] : $attribute_name; } /** * Query for all records matching the state of the current * Object. * * @param strinh $find_query The sql query (w/out the where clause) * @param boolean $as_detached_recordset * @param string $order_by_field "{fieldname} {ASC | DESC}" * @param string $group_by_fields "{fieldname1, fieldname2...}" * @return ReadOnlyResultSet or array if $as_detached_recordset = true */ public function find($find_query, $as_detached_recordset=false, $order_by_field=false, $group_by_fields=false, $return_structure=false, $limit_terms=false) { if (! $as_detached_recordset && $return_structure) { throw new Exception('Usage: If you requests a $return_structure, you must specify $as_detached_recordset=true'); } $where_clauses = array(); // Loop through the properties build individual where clauses foreach ($this->attributes as $attrib_key=>$attrib_values) { /* * attribute values can have multi value so force to array & * deal only with the array case. * This allows you to define something like * WHERE * username != 'Anonymous' AND * username != '' */ $attrib_values = is_array($attrib_values) ? $attrib_values : array($attrib_values); foreach ($attrib_values as $attrib_value) { if (trim($attrib_value)!='') { $where_clauses[] = $this->buildWhereClause($this->mappedAttributeName($attrib_key), $attrib_value); } } } // If we have a where clause, build it $this->sql = $find_query; if (count($where_clauses)> 0){ $this->sql .= " WHERE " . implode(' '.DataObject::CONJUNCTION_AND.' ', $where_clauses); } if ($group_by_fields) { $group_by_fields = explode(',',$group_by_fields); $group_by_fields = array_map('trim',$group_by_fields); $this->sql .= " GROUP BY `".implode("`, `",$group_by_fields)."`"; } if ($order_by_field) { $order_by_field = explode(',',$order_by_field); foreach ($order_by_field as &$order_by) { $direction = stristr('DESC',$order_by) ? "DESC" : "ASC"; $order_by = str_ireplace(array("DESC","ASC"),'',$order_by); $order_by = "`".trim($order_by)."`"." {$direction} "; } $this->sql .= " ORDER BY ".implode("`, `",$order_by_field); } if ($limit_terms) { $this->sql .= " LIMIT ".$limit_terms; } $results = null; if ($as_detached_recordset) { $results = $this->db->getAll($this->sql,array(),DB_FETCHMODE_ASSOC); db_check_error($results); if($return_structure) { $results = $this->mapReturnStructure($results,$return_structure); } return $results; } else { $result_set = $this->db->query($this->sql); db_check_error($result_set); require_once('ReadOnlyResultSet.php'); return new ReadOnlyResultSet($result_set); } } /** * Takes the indexed array result of a query and re-structures it to * the supplied assocc array. * - ie * <code> * if $result_data is * [0]=>array('name'=>'foo','age'=>10, 'height'=>6.5, 'weight'=>100), * [1]=>array('name'=>'bar','age'=>13, 'height'=>6.5, 'weight'=>120) * and $return_structure is * 'name'=>array('age','weight') * then the final returned array would be... * 'foo'=>array('age'=>10,'weight'=>100), * 'bar'=>array('age'=>13,'weight'=>120) * * if there is only 1 array value in the return structure.. * i.e. 'name'=>'age' (or 'name'=>array('age') * then the final returned array would be...(no keys on value) * 'foo'=>10, * 'bar'=>13 * </code> * * @param array $result_data * @param array $return_structure */ private function mapReturnStructure($result_data, $return_structure) { $result = array(); foreach ($result_data as $datum) { $key = key($return_structure); $struct_values = is_array($return_structure[$key]) ? $return_structure[$key] : array($return_structure[$key]); foreach ($datum as $datum_key => $datum_value) { if (in_array($datum_key,$struct_values)) { if (count($struct_values)>1) { $result[$datum[$key]][$datum_key] = $datum_value; } else { $result[$datum[$key]] = $datum_value; } } } } return $result; } }

Tagged PHP data storage

About