Commit 735d8bae authored by Ciarán Ó Rourke's avatar Ciarán Ó Rourke
Browse files

Add Bucket Listing

Create object in backend for buckets to affirm their existence and to
store meta data. Support S3 bucket listing. Generalise bucket logic to
'container' where the protocol should not be known.

Change log
* Disk
    - append to bucket list instead of overwriting
    - parse bucket name from object path
    - store list of objects in each bucket in a unique file
    - store list of buckets in a file
    - get_object_list function
* use bucket + object path in unit tests
* test Fifo::get_bucket_object_list
* bucket listing in stream, storage and disk implementation
* list buckets using Phobos
* test for Phobos bucket listing
* function to format ListBuckets response
* bucket list and object list request identification
* handle bucket list request
* list buckets integration test
* https confict response if bucket already exists for create_container
* bucket list unit tests ignore buckets created by previous test runs
* function to reset Phobos descriptor
* function to reset Fifo descriptors and storage backend
* make sure to deallocate objid if it is allocated before reallocating
* function to check if object exists in the Phobos store
* function to check if object exists in Disk store
* test object_exists functions
* create container if object is PUT to a not yet existing one
* create container function in PutRequestHandler
* remove bucket logic from Disk
* remove bucket tests from Phobos_file and Fifo tests
* remove bucket logic from Stream and Storage as this is handled elsewhere
* handle bucket logic in Protocol Formatter
* rename bucket to container where appropriate
* support create bucket request
* test create bucket request
* function to check for container list request
* use sane bucket identifier
* function to extract container name
* use try-catch in create_container function
* test that creating an existing container fails
* update documentation
* flush log messages to logfile immediately
parent b11992ae
Pipeline #2679 passed with stages
in 22 minutes and 47 seconds
......@@ -14,8 +14,11 @@ Is inherited by [deimos::protocol::s3::S3_formatter][deimos-protocol-s3-S3_forma
| [Formatter](#deimos-protocol-Formatter-Formatter) | default constructor |
| [~Formatter](#deimos-protocol-Formatter-~Formatter) | default destructor |
| [list_objects_format_response](#deimos-protocol-Formatter-list_objects_format_response) | create a formatted response for the list_objects request |
| [add_custom_server_metadata](#deimos-protocol-Formatter-add_custom_server_metadata) | add metadata that the server stores to the object storage specific to a protocol |
| [list_containers_format_response](#deimos-protocol-Formatter-list_containers_format_response) | create a formatted response for the list_containers request |
| [add_custom_server_object_metadata](#deimos-protocol-Formatter-add_custom_server_object_metadata) | add metadata that the server stores to the object storage specific to a protocol, for objects |
| [add_custom_server_container_metadata](#deimos-protocol-Formatter-add_custom_server_container_metadata) | add metadata that the server stores to the object storage specific to a protocol, for containers |
| [get_object_listing_identifier](#deimos-protocol-Formatter-get_object_listing_identifier) | get the identifier to list objects according to a contraint |
| [get_container_listing_identifier](#deimos-protocol-Formatter-get_container_listing_identifier) | get the identifier to list containers |
......@@ -91,9 +94,44 @@ create a formatted response for the list_objects request
[Go to Top](#deimos-protocol-Formatter)
### <a name='deimos-protocol-Formatter-add_custom_server_metadata' /> public void deimos::protocol::Formatter::add_custom_server_metadata (const Headers &headers, std::map< std::string, std::string > &metadata) const =0
### <a name='deimos-protocol-Formatter-list_containers_format_response' /> public std::string deimos::protocol::Formatter::list_containers_format_response (const std::vector< std::map< std::string, std::string >> &container_list) const =0
add metadata that the server stores to the object storage specific to a protocol
create a formatted response for the list_containers request
#### Parameters:
| Type | Name | Description |
| ---- | ---- | ---- |
| const std::vector< std::map< std::string, std::string >> & | container_list | a vector of container names |
#### Returns:
| Type | Description |
| ---- | ---- |
| std::string | formatted response |
#### Qualifiers:
* const
* virtual
[Go to Top](#deimos-protocol-Formatter)
### <a name='deimos-protocol-Formatter-add_custom_server_object_metadata' /> public void deimos::protocol::Formatter::add_custom_server_object_metadata (const Headers &headers, std::map< std::string, std::string > &metadata) const =0
add metadata that the server stores to the object storage specific to a protocol, for objects
......@@ -115,6 +153,36 @@ add metadata that the server stores to the object storage specific to a protocol
#### Qualifiers:
* const
* virtual
[Go to Top](#deimos-protocol-Formatter)
### <a name='deimos-protocol-Formatter-add_custom_server_container_metadata' /> public void deimos::protocol::Formatter::add_custom_server_container_metadata (std::map< std::string, std::string > &metadata) const =0
add metadata that the server stores to the object storage specific to a protocol, for containers
#### Parameters:
| Type | Name | Description |
| ---- | ---- | ---- |
| std::map< std::string, std::string > & | metadata | the metadata to extend |
#### Qualifiers:
* const
* virtual
......@@ -137,7 +205,37 @@ get the identifier to list objects according to a contraint
#### Returns:
| Type | Description |
| ---- | ---- |
| std::string | the identifier to list objects (this will be part of the metadata added in [add_custom_server_metadata][deimos-protocol-Formatter-add_custom_server_metadata]) |
| std::string | the identifier to list objects (this will be part of the metadata added in [add_custom_server_object_metadata][deimos-protocol-Formatter-add_custom_server_object_metadata]) |
#### Qualifiers:
* const
* virtual
[Go to Top](#deimos-protocol-Formatter)
### <a name='deimos-protocol-Formatter-get_container_listing_identifier' /> public std::string deimos::protocol::Formatter::get_container_listing_identifier () const =0
get the identifier to list containers
#### Returns:
| Type | Description |
| ---- | ---- |
| std::string | the identifier to list containers |
......@@ -157,6 +255,6 @@ get the identifier to list objects according to a contraint
[Go to Top](#deimos-protocol-Formatter)
[deimos-protocol-Formatter-add_custom_server_metadata]:./Formatter.md#deimos-protocol-Formatter-add_custom_server_metadata
[deimos-protocol-Formatter-add_custom_server_object_metadata]:./Formatter.md#deimos-protocol-Formatter-add_custom_server_object_metadata
[deimos-protocol-Headers]:./Headers.md
[deimos-protocol-s3-S3_formatter]:./s3/S3_formatter.md
......@@ -17,6 +17,9 @@ Is inherited by [deimos::protocol::s3::S3_header][deimos-protocol-s3-S3_header].
| [log](#deimos-protocol-Headers-log) | log the contents of this object |
| [get_identifier](#deimos-protocol-Headers-get_identifier) | get the identifier of the request, the string right adjacent to PUT/GET/... |
| [is_object_request](#deimos-protocol-Headers-is_object_request) | check if the request is about an object or only metadata |
| [is_object_list_request](#deimos-protocol-Headers-is_object_list_request) | check if the request is to list objects |
| [is_container_list_request](#deimos-protocol-Headers-is_container_list_request) | check if the request is to list containers |
| [get_container](#deimos-protocol-Headers-get_container) | extract the name of the container |
| [get_body_length](#deimos-protocol-Headers-get_body_length) | gets the total length of all body chunks that is expected |
| [get_user_meta_data](#deimos-protocol-Headers-get_user_meta_data) | gets the user defined metadata stored in the HTTP headers |
| [get_header_by_name](#deimos-protocol-Headers-get_header_by_name) | extract one header value from the headers |
......@@ -161,6 +164,96 @@ check if the request is about an object or only metadata
#### Qualifiers:
* const
* virtual
[Go to Top](#deimos-protocol-Headers)
### <a name='deimos-protocol-Headers-is_object_list_request' /> public bool deimos::protocol::Headers::is_object_list_request () const =0
check if the request is to list objects
#### Returns:
| Type | Description |
| ---- | ---- |
| bool | true if the request is to list objects, false to get object or list containers types of requests |
#### Qualifiers:
* const
* virtual
[Go to Top](#deimos-protocol-Headers)
### <a name='deimos-protocol-Headers-is_container_list_request' /> public bool deimos::protocol::Headers::is_container_list_request () const =0
check if the request is to list containers
#### Returns:
| Type | Description |
| ---- | ---- |
| bool | true if the request is to list containers, false to get or list objects types of requests |
#### Qualifiers:
* const
* virtual
[Go to Top](#deimos-protocol-Headers)
### <a name='deimos-protocol-Headers-get_container' /> public std::string deimos::protocol::Headers::get_container () const =0
extract the name of the container
#### Returns:
| Type | Description |
| ---- | ---- |
| std::string | the container name as a string |
#### Qualifiers:
* const
* virtual
......
......@@ -25,7 +25,7 @@ namespace for http interface specific functionality
| Name | Description |
| ---- | ---- |
| [Authorisation_status](#deimos-protocol-Authorisation_status) | enum to store all the different states a [deimos::protocol::Authorisation][deimos-protocol-Authorisation] object can be in |
| [supported_apis](#deimos-protocol-supported_apis) | list of the HTTP APIs, this server supports. |
| [supported_apis](#deimos-protocol-supported_apis) | list of the HTTP APIs this server supports. |
......@@ -57,7 +57,7 @@ enum to store all the different states a [deimos::protocol::Authorisation][deimo
### <a name='deimos-protocol-supported_apis' /> public deimos::protocol::supported_apis
list of the HTTP APIs, this server supports.
list of the HTTP APIs this server supports.
......
......@@ -11,13 +11,52 @@ Inherits from [deimos::protocol::Formatter][deimos-protocol-Formatter].
## Private Functions
| Name | Description |
| ---- | ---- |
| [path_to_key](#deimos-protocol-s3-S3_formatter-path_to_key) | extract key from object path |
| [list_objects_format_response](#deimos-protocol-s3-S3_formatter-list_objects_format_response) | get an xml string for the response of an object listing request |
| [add_custom_server_metadata](#deimos-protocol-s3-S3_formatter-add_custom_server_metadata) | add metadata that the server stores to the object storage specific to a protocol |
| [list_containers_format_response](#deimos-protocol-s3-S3_formatter-list_containers_format_response) | get an xml string for the response of a bucket listing request |
| [add_custom_server_object_metadata](#deimos-protocol-s3-S3_formatter-add_custom_server_object_metadata) | add metadata that the server stores to the object storage specific to a protocol, for objects |
| [add_custom_server_container_metadata](#deimos-protocol-s3-S3_formatter-add_custom_server_container_metadata) | add metadata that the server stores to the object storage specific to a protocol, for containers |
| [get_object_listing_identifier](#deimos-protocol-s3-S3_formatter-get_object_listing_identifier) | get the identifier to list objects according to a contraint |
| [get_container_listing_identifier](#deimos-protocol-s3-S3_formatter-get_container_listing_identifier) | get the identifier to list containers |
## Private Functions
### <a name='deimos-protocol-s3-S3_formatter-path_to_key' /> private std::string deimos::protocol::s3::S3_formatter::path_to_key (const std::string &path) const
extract key from object path
#### Parameters:
| Type | Name | Description |
| ---- | ---- | ---- |
| const std::string & | path | the object path |
#### Returns:
| Type | Description |
| ---- | ---- |
| std::string | key string |
#### Qualifiers:
* const
* inline
[Go to Top](#deimos-protocol-s3-S3_formatter)
### <a name='deimos-protocol-s3-S3_formatter-list_objects_format_response' /> private std::string deimos::protocol::s3::S3_formatter::list_objects_format_response (const std::vector< std::map< std::string, std::string >> &object_list, const std::vector< std::string > &qualifiers) const override
get an xml string for the response of an object listing request
......@@ -55,9 +94,45 @@ get an xml string for the response of an object listing request
[Go to Top](#deimos-protocol-s3-S3_formatter)
### <a name='deimos-protocol-s3-S3_formatter-add_custom_server_metadata' /> private void deimos::protocol::s3::S3_formatter::add_custom_server_metadata (const Headers &headers, std::map< std::string, std::string > &metadata) const override
### <a name='deimos-protocol-s3-S3_formatter-list_containers_format_response' /> private std::string deimos::protocol::s3::S3_formatter::list_containers_format_response (const std::vector< std::map< std::string, std::string >> &bucket_list) const override
add metadata that the server stores to the object storage specific to a protocol
get an xml string for the response of a bucket listing request
#### Parameters:
| Type | Name | Description |
| ---- | ---- | ---- |
| const std::vector< std::map< std::string, std::string >> & | bucket_list | a vector of bucket meta data maps |
#### Returns:
| Type | Description |
| ---- | ---- |
| std::string | xml response string |
#### Qualifiers:
* const
* inline
* virtual
[Go to Top](#deimos-protocol-s3-S3_formatter)
### <a name='deimos-protocol-s3-S3_formatter-add_custom_server_object_metadata' /> private void deimos::protocol::s3::S3_formatter::add_custom_server_object_metadata (const Headers &headers, std::map< std::string, std::string > &metadata) const override
add metadata that the server stores to the object storage specific to a protocol, for objects
......@@ -79,6 +154,37 @@ add metadata that the server stores to the object storage specific to a protocol
#### Qualifiers:
* const
* inline
* virtual
[Go to Top](#deimos-protocol-s3-S3_formatter)
### <a name='deimos-protocol-s3-S3_formatter-add_custom_server_container_metadata' /> private void deimos::protocol::s3::S3_formatter::add_custom_server_container_metadata (std::map< std::string, std::string > &metadata) const override
add metadata that the server stores to the object storage specific to a protocol, for containers
#### Parameters:
| Type | Name | Description |
| ---- | ---- | ---- |
| std::map< std::string, std::string > & | metadata | the metadata to extend |
#### Qualifiers:
* const
* inline
......@@ -102,7 +208,38 @@ get the identifier to list objects according to a contraint
#### Returns:
| Type | Description |
| ---- | ---- |
| std::string | the identifier to list objects (this will be part of the metadata added in [add_custom_server_metadata][deimos-protocol-s3-S3_formatter-add_custom_server_metadata]) |
| std::string | the identifier to list objects (this will be part of the metadata added in [add_custom_server_object_metadata][deimos-protocol-s3-S3_formatter-add_custom_server_object_metadata]) |
#### Qualifiers:
* const
* inline
* virtual
[Go to Top](#deimos-protocol-s3-S3_formatter)
### <a name='deimos-protocol-s3-S3_formatter-get_container_listing_identifier' /> private std::string deimos::protocol::s3::S3_formatter::get_container_listing_identifier () const override
get the identifier to list containers
#### Returns:
| Type | Description |
| ---- | ---- |
| std::string | the identifier to list containers |
......@@ -125,4 +262,4 @@ get the identifier to list objects according to a contraint
[deimos-protocol-Formatter]:./../Formatter.md
[deimos-protocol-Headers]:./../Headers.md
[deimos-protocol-s3-S3_formatter-add_custom_server_metadata]:./S3_formatter.md#deimos-protocol-s3-S3_formatter-add_custom_server_metadata
[deimos-protocol-s3-S3_formatter-add_custom_server_object_metadata]:./S3_formatter.md#deimos-protocol-s3-S3_formatter-add_custom_server_object_metadata
......@@ -20,7 +20,9 @@ Inherits from [deimos::protocol::Headers][deimos-protocol-Headers].
| [set_headers](#deimos-protocol-s3-S3_header-set_headers) | sets the internal header instance to the given header object |
| [log](#deimos-protocol-s3-S3_header-log) | log the contents of this object |
| [is_object_request](#deimos-protocol-s3-S3_header-is_object_request) | check if the request is about an object or only metadata |
| [get_bucket](#deimos-protocol-s3-S3_header-get_bucket) | extracts the name of the S3 bucket requested in the HTTP message |
| [is_object_list_request](#deimos-protocol-s3-S3_header-is_object_list_request) | check if the request is to list objects |
| [is_container_list_request](#deimos-protocol-s3-S3_header-is_container_list_request) | check if the request is to list containers |
| [get_container](#deimos-protocol-s3-S3_header-get_container) | extract the name of the container |
| [get_key_without_bucket](#deimos-protocol-s3-S3_header-get_key_without_bucket) | extracts the name of the key id requested in the HTTP message |
| [get_identifier](#deimos-protocol-s3-S3_header-get_identifier) | get the identifier of the request, the string right adjacent to PUT/GET/... |
| [get_body_length](#deimos-protocol-s3-S3_header-get_body_length) | gets the total length of all body chunks that is expected |
......@@ -130,9 +132,75 @@ The way this is decided currently is by checking if there is a '/' in the path a
[Go to Top](#deimos-protocol-s3-S3_header)
### <a name='deimos-protocol-s3-S3_header-get_bucket' /> public std::string deimos::protocol::s3::S3_header::get_bucket () const
### <a name='deimos-protocol-s3-S3_header-is_object_list_request' /> public bool deimos::protocol::s3::S3_header::is_object_list_request () const override
extracts the name of the S3 bucket requested in the HTTP message
check if the request is to list objects
#### Returns:
| Type | Description |
| ---- | ---- |
| bool | true if the request is to list objects, false to get object or list containers types of requests |
For S3 a `false` return value means that this is an object request or a bucket list request.
#### Qualifiers:
* const
* inline
* virtual
[Go to Top](#deimos-protocol-s3-S3_header)
### <a name='deimos-protocol-s3-S3_header-is_container_list_request' /> public bool deimos::protocol::s3::S3_header::is_container_list_request () const override
check if the request is to list containers
#### Returns:
| Type | Description |
| ---- | ---- |
| bool | true if the request is to list containers, false to get or list objects types of requests |
For S3 a `false` return value means that this is an object request or an object list request.
#### Qualifiers:
* const
* inline
* virtual
[Go to Top](#deimos-protocol-s3-S3_header)
### <a name='deimos-protocol-s3-S3_header-get_container' /> public std::string deimos::protocol::s3::S3_header::get_container () const override
extract the name of the container
......@@ -156,6 +224,7 @@ extracts the name of the S3 bucket requested in the HTTP message
#### Qualifiers:
* const
* inline
* virtual
[Go to Top](#deimos-protocol-s3-S3_header)
......
......@@ -17,7 +17,7 @@ Is inherited by [deimos::storage::phobos::Phobos_file][deimos-storage-phobos-Pho
| ---- | ---- |
| [set_meta_data](#deimos-storage-Storage-set_meta_data) | set the metadata that an object that is added to the database should get |
| [get_meta_data](#deimos-storage-Storage-get_meta_data) | get the metadata associated to the current object as a map of key:value pairs |
| [get_object_list](#deimos-storage-Storage-get_object_list) | get the metadata associated to all objects of a particular bucket as a vector of maps of key:value pairs |
| [get_object_list](#deimos-storage-Storage-get_object_list) | get the metadata associated to all objects of a particular container as a vector of maps of key:value pairs |
## PUT functions
......@@ -39,6 +39,8 @@ Is inherited by [deimos::storage::phobos::Phobos_file][deimos-storage-phobos-Pho
| ---- | ---- |
| [Storage](#deimos-storage-Storage-Storage) | default constructor |
| [~Storage](#deimos-storage-Storage-~Storage) | default destructor |
| [reset](#deimos-storage-Storage-reset) | prepare for a new operation |
| [object_exists](#deimos-storage-Storage-object_exists) | check if an object exists |
......@@ -106,9 +108,9 @@ get the metadata associated to the current object as a map of key:value pairs
[Go to Top](#deimos-storage-Storage)
### <a name='deimos-storage-Storage-get_object_list' /> public std::vector< std::map< std::string, std::string > > deimos::storage::Storage::get_object_list (std::string bucket_name)=0
### <a name='deimos-storage-Storage-get_object_list' /> public std::vector< std::map< std::string, std::string > > deimos::storage::Storage::get_object_list (std::string container_name)=0
get the metadata associated to all objects of a particular bucket as a vector of maps of key:value pairs
get the metadata associated to all objects of a particular container as a vector of maps of key:value pairs
......@@ -116,7 +118,7 @@ get the metadata associated to all objects of a particular bucket as a vector of
#### Parameters:
| Type | Name | Description |
| ---- | ---- | ---- |
| std::string | bucket_name | The name of the requested bucket |
| std::string | container_name | The name of the requested container |
#### Returns:
| Type | Description |
......@@ -299,6 +301,50 @@ default destructor
* virtual
[Go to Top](#deimos-storage-Storage)
### <a name='deimos-storage-Storage-reset' /> public void deimos::storage::Storage::reset ()=0
prepare for a new operation
#### Qualifiers:
* virtual
[Go to Top](#deimos-storage-Storage)
### <a name='deimos-storage-Storage-object_exists' /> public bool deimos::storage::Storage::object_exists (const std::string &object_id)=0
check if an object exists
#### Parameters:
| Type | Name | Description |
| ---- | ---- | ---- |
| const std::string & | object_id | |
#### Returns:
| Type | Description |
| ---- | ---- |
| bool | |