Commit 1ba46322 authored by Sophie Wenzel-Teuber's avatar Sophie Wenzel-Teuber
Browse files

Add Deletion

Add support of the DELETE HTTP request and call the Phobos function phobos_delete

Changelog:
* Add new class DeleteRequestHandler and add it to the Factory
* Add phobos_delete_cpp to the Phobos wrapper, which calls phobos_delete
* Add a delete_object function to Stream/Fifo and Storage/Phobos
* Add tests to delete objects
parent fa808a8d
Pipeline #4038 passed with stages
in 19 minutes and 48 seconds
......@@ -75,8 +75,6 @@ option(
set(CMAKE_MODULE_PATH "${CMAKE_CURRENT_SOURCE_DIR}/cmake" ${CMAKE_MODULE_PATH})
include(target_link_system_libraries)
# set(Boost_DEBUG 1)
find_package(Boost REQUIRED COMPONENTS iostreams)
find_package(proxygen REQUIRED)
find_package(phobos REQUIRED)
......
# <a name='deimos-DeleteRequestHandler' /> public deimos::DeleteRequestHandler
proxygen class implementation for handling DELETE requests
## Inheritance:
Inherits from RequestHandler.
## Public Functions
| Name | Description |
| ---- | ---- |
| [onRequest](#deimos-DeleteRequestHandler-onRequest) | first function to be called when a new request comes in |
| [onBody](#deimos-DeleteRequestHandler-onBody) | function called on every body chunk belonging to this message |
| [onEOM](#deimos-DeleteRequestHandler-onEOM) | function called when the incoming message is finished |
| [onUpgrade](#deimos-DeleteRequestHandler-onUpgrade) | function called on upgrade |
| [requestComplete](#deimos-DeleteRequestHandler-requestComplete) | function ... |
| [onError](#deimos-DeleteRequestHandler-onError) | function called when an error occurred |
| [onEgressPaused](#deimos-DeleteRequestHandler-onEgressPaused) | called when the queue is full. |
| [onEgressResumed](#deimos-DeleteRequestHandler-onEgressResumed) | called when the queue is not longer full. |
| [DeleteRequestHandler](#deimos-DeleteRequestHandler-DeleteRequestHandler) | Constructor for stream class initialization. |
## Public Functions
### <a name='deimos-DeleteRequestHandler-onRequest' /> public void deimos::DeleteRequestHandler::onRequest (std::unique_ptr< proxygen::HTTPMessage > headers) noexcept override
first function to be called when a new request comes in
#### Parameters:
| Type | Name | Description |
| ---- | ---- | ---- |
| std::unique_ptr< proxygen::HTTPMessage > | headers | headers of the HTTP message this handler was created for |
[Go to Top](#deimos-DeleteRequestHandler)
### <a name='deimos-DeleteRequestHandler-onBody' /> public void deimos::DeleteRequestHandler::onBody (std::unique_ptr< folly::IOBuf > body) noexcept override
function called on every body chunk belonging to this message
#### Parameters:
| Type | Name | Description |
| ---- | ---- | ---- |
| std::unique_ptr< folly::IOBuf > | body | buffer for the body |
This is not used in this case because DELETE requests don't usually have a body
[Go to Top](#deimos-DeleteRequestHandler)
### <a name='deimos-DeleteRequestHandler-onEOM' /> public void deimos::DeleteRequestHandler::onEOM () noexcept override
function called when the incoming message is finished
[Go to Top](#deimos-DeleteRequestHandler)
### <a name='deimos-DeleteRequestHandler-onUpgrade' /> public void deimos::DeleteRequestHandler::onUpgrade (proxygen::UpgradeProtocol proto) noexcept override
function called on upgrade
#### Parameters:
| Type | Name | Description |
| ---- | ---- | ---- |
| proxygen::UpgradeProtocol | proto | the new protocol |
Not supported in our case!
[Go to Top](#deimos-DeleteRequestHandler)
### <a name='deimos-DeleteRequestHandler-requestComplete' /> public void deimos::DeleteRequestHandler::requestComplete () noexcept override
function ...
> **[Todo][todo]:** find out what this does?
[Go to Top](#deimos-DeleteRequestHandler)
### <a name='deimos-DeleteRequestHandler-onError' /> public void deimos::DeleteRequestHandler::onError (proxygen::ProxygenError err) noexcept override
function called when an error occurred
#### Parameters:
| Type | Name | Description |
| ---- | ---- | ---- |
| proxygen::ProxygenError | err | the error |
Not supported in our case!
[Go to Top](#deimos-DeleteRequestHandler)
### <a name='deimos-DeleteRequestHandler-onEgressPaused' /> public void deimos::DeleteRequestHandler::onEgressPaused () noexcept override
called when the queue is full.
Contents are copies from a proxygen example
#### Qualifiers:
* inline
[Go to Top](#deimos-DeleteRequestHandler)
### <a name='deimos-DeleteRequestHandler-onEgressResumed' /> public void deimos::DeleteRequestHandler::onEgressResumed () noexcept override
called when the queue is not longer full.
Contents are copies from a proxygen example
#### Qualifiers:
* inline
[Go to Top](#deimos-DeleteRequestHandler)
### <a name='deimos-DeleteRequestHandler-DeleteRequestHandler' /> public deimos::DeleteRequestHandler::DeleteRequestHandler (std::unique_ptr< protocol::Protocol_factory > protocol_factory, std::unique_ptr< stream::Stream > input_stream)
Constructor for stream class initialization.
#### Parameters:
| Type | Name | Description |
| ---- | ---- | ---- |
| std::unique_ptr< [protocol::Protocol_factory][deimos-protocol-Protocol_factory] > | protocol_factory | Factory to create the https interface specific objects |
| std::unique_ptr< [stream::Stream][deimos-stream-Stream] > | input_stream | [stream::Stream][deimos-stream-Stream] class instance to initialize the server |
#### Qualifiers:
* inline
[Go to Top](#deimos-DeleteRequestHandler)
[deimos-protocol-Protocol_factory]:./protocol/Protocol_factory.md
[deimos-stream-Stream]:./stream/Stream.md
[todo]:./../todo.md#todo
......@@ -7,6 +7,7 @@
| Name | Description |
| ---- | ---- |
| [DeimosException](./DeimosException.md) | exception class for all user defined exceptions in Deimos |
| [DeleteRequestHandler](./DeleteRequestHandler.md) | proxygen class implementation for handling DELETE requests |
| [GetmdRequestHandler](./GetmdRequestHandler.md) | proxygen class implementation for handling HEAD requests |
| [GetRequestHandler](./GetRequestHandler.md) | proxygen class implementation for handling GET requests |
| [HandlerFactory](./HandlerFactory.md) | factory to create the handlers for incoming HTTP requests |
......
......@@ -41,6 +41,7 @@ Is inherited by [deimos::storage::phobos::Phobos_file][deimos-storage-phobos-Pho
| [~Storage](#deimos-storage-Storage-~Storage) | default destructor |
| [reset](#deimos-storage-Storage-reset) | prepare for a new operation |
| [object_exists](#deimos-storage-Storage-object_exists) | check if an object exists |
| [delete_object](#deimos-storage-Storage-delete_object) | Delete a single object. |
......@@ -341,6 +342,35 @@ check if an object exists
#### Qualifiers:
* virtual
[Go to Top](#deimos-storage-Storage)
### <a name='deimos-storage-Storage-delete_object' /> public void deimos::storage::Storage::delete_object (std::string object_id)=0
Delete a single object.
#### Parameters:
| Type | Name | Description |
| ---- | ---- | ---- |
| std::string | object_id | the internal storage id from the data that should be deleted |
#### Qualifiers:
* virtual
......
......@@ -32,6 +32,7 @@ Inherits from [deimos::storage::Storage][deimos-storage-Storage].
| [set_meta_data](#deimos-storage-phobos-Phobos_file-set_meta_data) | set the metadata that an object that is added to the database should get |
| [get_meta_data](#deimos-storage-phobos-Phobos_file-get_meta_data) | get the metadata associated to the current object as a map of key:value pairs |
| [get_object_list](#deimos-storage-phobos-Phobos_file-get_object_list) | get the metadata associated to all objects of a particular container as a vector of maps of key:value pairs |
| [delete_object](#deimos-storage-phobos-Phobos_file-delete_object) | Delete a single object. |
| [db_put](#deimos-storage-phobos-Phobos_file-db_put) | Puts data to the databse. |
| [db_get](#deimos-storage-phobos-Phobos_file-db_get) | Gets data from the databse. |
| [prepare_put](#deimos-storage-phobos-Phobos_file-prepare_put) | Starts a put operation to the database. |
......@@ -301,6 +302,35 @@ get the metadata associated to all objects of a particular container as a vector
#### Qualifiers:
* virtual
[Go to Top](#deimos-storage-phobos-Phobos_file)
### <a name='deimos-storage-phobos-Phobos_file-delete_object' /> public void deimos::storage::phobos::Phobos_file::delete_object (std::string object_id) override
Delete a single object.
#### Parameters:
| Type | Name | Description |
| ---- | ---- | ---- |
| std::string | object_id | the internal storage id from the data that should be deleted |
#### Qualifiers:
* virtual
......
......@@ -14,6 +14,11 @@ Is inherited by [deimos::stream::fifo::Fifo][deimos-stream-fifo-Fifo].
| ---- | ---- |
| [start_put](#deimos-stream-Stream-start_put) | start a put operation |
| [put](#deimos-stream-Stream-put) | add a chunk of data to the object |
## GET functions
| Name | Description |
| ---- | ---- |
| [start_get](#deimos-stream-Stream-start_get) | start a gett operation |
| [get](#deimos-stream-Stream-get) | gets a chunk of data of the object |
| [finish_io](#deimos-stream-Stream-finish_io) | end an I/O operation |
......@@ -37,6 +42,7 @@ Is inherited by [deimos::stream::fifo::Fifo][deimos-stream-fifo-Fifo].
| [set_storage_meta_data](#deimos-stream-Stream-set_storage_meta_data) | setting the metadata that the created object should get |
| [get_meta_data](#deimos-stream-Stream-get_meta_data) | get the metadata associated to the current object as a map of key:value pairs |
| [get_container_object_list](#deimos-stream-Stream-get_container_object_list) | get the metadata associated to all objects of a particular container as a vector of maps of key:value pairs |
| [delete_object](#deimos-stream-Stream-delete_object) | Delete a single object. |
......@@ -107,6 +113,7 @@ This function can be called repeatedly until the server's message body is deplet
[Go to Top](#deimos-stream-Stream)
## GET functions
### <a name='deimos-stream-Stream-start_get' /> public void deimos::stream::Stream::start_get (std::string object_id)=0
start a gett operation
......@@ -429,6 +436,36 @@ get the metadata associated to all objects of a particular container as a vector
#### Qualifiers:
* const
* virtual
[Go to Top](#deimos-stream-Stream)
### <a name='deimos-stream-Stream-delete_object' /> public void deimos::stream::Stream::delete_object (std::string object_id) const =0
Delete a single object.
#### Parameters:
| Type | Name | Description |
| ---- | ---- | ---- |
| std::string | object_id | the internal storage id from the data that should be deleted |
#### Qualifiers:
* const
* virtual
......
......@@ -37,6 +37,7 @@ Inherits from [deimos::stream::Stream][deimos-stream-Stream].
| [set_storage_meta_data](#deimos-stream-fifo-Fifo-set_storage_meta_data) | setting the metadata that the created object should get |
| [get_meta_data](#deimos-stream-fifo-Fifo-get_meta_data) | get the metadata associated to the current object as a map of key:value pairs |
| [get_container_object_list](#deimos-stream-fifo-Fifo-get_container_object_list) | get the metadata associated to all objects of a particular container as a vector of maps of key:value pairs |
| [delete_object](#deimos-stream-fifo-Fifo-delete_object) | Delete a single object. |
| [start_put](#deimos-stream-fifo-Fifo-start_put) | start a put operation |
| [put](#deimos-stream-fifo-Fifo-put) | add a chunk of data to the object |
| [start_get](#deimos-stream-fifo-Fifo-start_get) | start a gett operation |
......@@ -352,6 +353,36 @@ get the metadata associated to all objects of a particular container as a vector
#### Qualifiers:
* const
* virtual
[Go to Top](#deimos-stream-fifo-Fifo)
### <a name='deimos-stream-fifo-Fifo-delete_object' /> public void deimos::stream::fifo::Fifo::delete_object (std::string object_id) const override
Delete a single object.
#### Parameters:
| Type | Name | Description |
| ---- | ---- | ---- |
| std::string | object_id | the internal storage id from the data that should be deleted |
#### Qualifiers:
* const
* virtual
......
......@@ -2,6 +2,9 @@
* Member
[deimos::DeleteRequestHandler::requestComplete][deimos-DeleteRequestHandler-requestComplete] () noexcept override
* find out what this does?
* Member
[deimos::GetRequestHandler::requestComplete][deimos-GetRequestHandler-requestComplete] () noexcept override
* find out what this does?
......@@ -16,6 +19,7 @@
[deimos-DeleteRequestHandler-requestComplete]:./deimos/DeleteRequestHandler.md#deimos-DeleteRequestHandler-requestComplete
[deimos-GetRequestHandler-requestComplete]:./deimos/GetRequestHandler.md#deimos-GetRequestHandler-requestComplete
[deimos-stream-Stream-m_db_result]:./deimos/stream/Stream.md#deimos-stream-Stream-m_db_result
[deimos-stream-fifo-Fifo-get_fifo_descriptors]:./deimos/stream/fifo/Fifo.md#deimos-stream-fifo-Fifo-get_fifo_descriptors
......@@ -225,7 +225,7 @@ bool S3_authorisation::has_signed_payload(const S3_header& headers) const
std::string payload_sha256 =
headers.get_header_by_name("x-amz-content-sha256");
if (content_length.empty()) {
if (content_length.empty() or content_length == "0") {
return false;
}
if (payload_sha256.empty() || payload_sha256 == "UNSIGNED-PAYLOAD") {
......
......@@ -48,6 +48,11 @@ class S3_header : public Headers {
spdlog::info("key: {}", get_key_without_bucket());
spdlog::debug("queries: {}", get_queries());
spdlog::debug("identifier: {}", get_identifier());
// m_headers->getHeaders().forEach([&](const std::string& header,
// const std::string& val) {
// spdlog::info("{}: {}", header, val);
// });
}
///
......
add_library(
server
delete_request_handler.cc
get_request_handler.cc
put_request_handler.cc
getmd_request_handler.cc
......
/*
* Copyright (c) 2020, Irish Centre for High End Computing (ICHEC), NUI Galway
* Authors:
* Ciarán O'Rourke <ciaran.orourke@ichec.ie>,
* Sophie Wenzel-Teuber <sophie.wenzel-teuber@ichec.ie>
*
* This source code is licensed under the MIT license found in the
* LICENSE file in the root directory of this source tree.
*/
#include "delete_request_handler.h"
#include "../deimos_exception.h"
#include <proxygen/httpserver/ResponseBuilder.h>
#include <spdlog/spdlog.h>
namespace deimos {
void DeleteRequestHandler::onRequest(
std::unique_ptr<proxygen::HTTPMessage> headers) noexcept
{
m_header->set_headers(std::move(headers));
m_header->log();
#ifndef DEIMOS_NO_AUTHORISATION
std::unique_ptr<protocol::Authorisation> auth =
m_protocol_factory->get_authorisation();
if (auth->authorise(*m_header) != protocol::Authorisation_status::valid) {
const protocol::Error_info* auth_error = auth->get_error();
proxygen::ResponseBuilder(downstream_)
.status(
auth_error->https_error_code,
auth_error->https_error_identifier)
.body(
auth_error->get_error_response_body(m_header->get_identifier()))
.sendWithEOM();
spdlog::warn(
"authorisation is invalid: {} ({})",
auth_error->https_error_identifier, auth_error->https_error_code);
return;
}
#endif
spdlog::debug("deleting object");
try {
m_stream->delete_object(m_header->get_identifier());
}
catch (const DeimosException& ex) {
spdlog::error("caught exception: {}", ex.what());
protocol::Error_info* error =
m_protocol_factory->error_from_return_code(ex.get_inner_error());
proxygen::ResponseBuilder(downstream_)
.status(error->https_error_code, error->https_error_identifier)
.body(error->get_error_response_body(m_header->get_identifier()))
.sendWithEOM();
return;
}
proxygen::ResponseBuilder response(downstream_);
response.status(200, "OK");
response.sendWithEOM();
spdlog::info("deletion complete");
}
void DeleteRequestHandler::onBody(
std::unique_ptr<folly::IOBuf> /*body*/) noexcept
{
/* ignore, only support HEAD */
}
void DeleteRequestHandler::onEOM() noexcept {}
void DeleteRequestHandler::onUpgrade(
proxygen::UpgradeProtocol /*protocol*/) noexcept
{
/* handler doesn't support upgrades */
}
void DeleteRequestHandler::requestComplete() noexcept
{
delete this;
}
void DeleteRequestHandler::onError(proxygen::ProxygenError /*err*/) noexcept
{
delete this;
}
} // namespace deimos
/*
* Copyright (c) 2020, Irish Centre for High End Computing (ICHEC), NUI Galway
* Authors:
* Ciarán O'Rourke <ciaran.orourke@ichec.ie>,
* Sophie Wenzel-Teuber <sophie.wenzel-teuber@ichec.ie>
*
* This source code is licensed under the MIT license found in the
* LICENSE file in the root directory of this source tree.
*/
#pragma once
#include "../protocol/protocol.h"
#include "../protocol/protocol_factory.h"
#include "../stream/stream.h"
#include <folly/Memory.h>