Changes to Phobos wrapper to reflect updated Phobos API.

Change log:
* make unit test a make target again
* remove deprecated and redundant target_link_directories call
* redesign phobos_cpp_wrapper for new Phobos version
* hardcode request of directory storage in Phobos for now

Sophie Wenzel-Teuber authored Jul 13, 2020 and Ciarán Ó Rourke committed Jul 21, 2020

Change log:
* Close file descriptors later in the process

Changelog:
* Add Find scripts to CMake to search for phobos and spdlog
* Add integrations tests to cmake by just copying the python code to the build directory
* Update Readme to reflect new SuperDeimos layout and add brief description

* set install directories through CMake
* print built subdirectories during build
* set up installation directories
* use development flags during debug build
* remove authorisation double negative cmake variable
* update README

get body length was throwing an exception when the Content-Length header was missing

Changelog:
* Add more error checks to s3 header class
* Add unit tests for s3 header class covering all typical use cases
* Add check for Markdown documentation to gitlab CI

Rename project from FiPhoboServer to Deimos

Change log:
* replace all occurances of variations of fiphoboserver with its deimos
equivalent

Use the spdlog library for project wide logging to a file. Path of
logfile is configurable as command line option.

Change log:
* use spdlog
* set global logging to file
* logfile as command line option
* update README command line option section with logfile option
* update README command line option section with default values for options
* control log level with CMake build type variable
* log received request
* get request logging
* put request logging
* getmd request logging
* fifo logging
* rename authorisation macro to comply with linter
* use rotating files with size limit

Changelog:
* in authorisation check if body is existent and signed
* add string `UNSIGNED_PAYLOAD` to the canonical request if requested
* check in put_request_handler if authorisation is waitig for the body or not

Changelog:
* Add `fiphobo:` prefix to all metadata saved by our server, that should not be sent back directly
* When sending the metadata, all entries starting with `fiphobo:` are ignored
* remove the bucket_name entry from all set metadata functions throughout the server and add the bucketname where all other metdata is set (put handler)

Changelog:
* Add content_length to phobos metadata
* Send the content_length from phobos metadata as header Content-Length
* Add all Phobos Metadata (formerly retrieved only by a HEAD request) as headers to the response to a GET request

Authorisation unit tests were being run when the project was built with authorisation disabled. Aurhorisation unit tests are now moved to a dedicated file that is built conditionally. Some changes are also made to the CI YAML file.

Change log:
* Move authorisation tests to dedicated file
* disable authorisation tests when building without authorisation
* default CI job runs unit tests and integration tests
* add CI job for build w/o authorisation
* rename Doxygen job to Documentation

This adds the CMake variable FIPHOBOSERVER_DISABLE_AUTHORISATION to disable the authentication

Changelog:
* Add option and variable to CMake
* Add ifndef pragmas to request handlers
* Update Readme

All different kinds of errors our server sends are in compliance with Amazon S3 errors. They are now stored in one place and should be accessed from anywhere.

Changelog:
* add the file s3_errors.h that contains const objects for all errors that might occur (most of them are not checked yet)
* change struct error_info into a class and add a function to create the XML from the object's variables
* Use these errors in the request handlers
* Add an error object to the s3 authorisation that will always contain "Access Denied" or another error if applicable
* Update tests to test the code generation and error return values

Sophie Wenzel-Teuber authored May 11, 2020 and Aaron Dees committed May 11, 2020

Changelog:
* Build documentation
* Add new flag to CMake to build Documentation with Doxygen, but without the Markdown converter
* Update CI for new CMake variable
* Update README to reflect changes

Outline usage profile of the fiphoboserver executable in the README

Change log:
* start server daemon subcommand
* move appropriate flags to start subcommand
* main function forks process given start command
* stop command to kill running server process
* integration tests: reflect subcommand invoking of server
* detail usage in README

Change log:
* src/utils directory for configuration, logging, etc
* Config class for handling configuration (command line parsing, etc)
* include Catch2 and CLI11 as system files so clang-tidy will ignore
  them
* support basic command line options:
      - hostname (positional, required)
      - number of threads
      - http_port
      - https_port
* add configuration to fiphoboserver executable main function
* return on exception for incompatible hostname
* return after parsing command line arguments if arguments are bad
  or --help is supplied
* don't pass command line arguments to folly
* Config returns struct of command line options
* accept configuration options from config file
* option to write current configuration to file
* integration tests: reflect changes to fiphoboserver executable

Add queries to authentication

See merge request oilgas/ltfs/fiphoboserver!31

S3 authorisation

See merge request oilgas/ltfs/fiphoboserver!29

The signature of amazon users is checked against a text file of known users and requests are accepted or rejected depending on the signature

Resolve "Python Linting"

See merge request oilgas/ltfs/fiphoboserver!28

Commit log:
* add yapf execution to formatting script
* run formatting script
* rename formatting and linting jobs
* install yapf in Format job

Resolve "Add Stress Test"

See merge request oilgas/ltfs/fiphoboserver!27

Change log:
* remove empty get file
* add function for PUT, GETMD GET with random key and get_file_name
* add stress test involving multiple parallel requests

General improvements

See merge request oilgas/ltfs/fiphoboserver!26

Change log:
* Dockerfile that inherits from an image with dependencies installed
* describe build process in README
* refer to superfiphoboserver in README
* rename m_file_closed to m_stream_completed
* formalise README

S3 error handling

See merge request oilgas/ltfs/fiphoboserver!23

Sophie Wenzel-Teuber authored Apr 08, 2020 and Aaron Dees committed Apr 08, 2020

All errors returned from phobos now have at least a little distinct error messages (as far as the return values permit it) and send an S3 readable XML along with the error number.

Resolve "Add Integration Tests"

See merge request oilgas/ltfs/fiphoboserver!25

* separate directories for unit tests and integration tests
* text files directory for testing server functions
* module for s3 utility functions
* module for file comparison utility functions
* module for generating random keys and meta data
* handle server executable within tests
* basic test for PUT and GET
* PUT and GET test for zero-byte file
* add integration testing to CI
* correct Clang Tidy job
* PUT command supports optional meta data
* test for GETMD
* test that GET with bad key fails
* test that PUT with used key fails

Resolve "CI"

See merge request oilgas/ltfs/fiphoboserver!22

* src/storage/phobos_file.hexplicit copy constructors to avoid double free of xd_objid
* tests: reflect meta data change to std::map from std::vector
* run Clang Format
* resolve Clang Tidy warnings
* add Gitlab-CI
* use fiphoboserver docker image
* update DockerHub image with Doxygen requirements
* CI: Clang Format job
* CI: Clang Tidy job
* CI: Doxygen build job
* CI: Unit Tests job

Cleanup codebase

See merge request oilgas/ltfs/fiphoboserver!20

Sophie Wenzel-Teuber authored Mar 31, 2020 and Aaron Dees committed Mar 31, 2020

Add a creation timestamp to the metadata and return only original Metadata in getmd (not the server related stuff)

Resolve "Add Unit Tests"

See merge request oilgas/ltfs/fiphoboserver!17

* utility functions for tests (utils.h, utils.cc)
* copy test text files to test binary directory during build
* tests for Phobos_file constructors
* tests for Phobos_file PUT and GET
* Phobos_file PUT and GET tests for empty file
* Phobos_file GETMD test
* in-memory (disk) storage solution for testing stream objects
* Fifo file descriptors must be copied explicitly as they are now an array
* Fifo tests for PUT and GET
* repeat Fifo tests for empty file
* Fifo tests for GETMD
* tests for Fifo object construction
* need to determine file size before db_get
* Phobos_file PUT  test for bad file descriptor exception
* Phobos_file PUT test for used object_id exception
* Phobos_file GETMD test for bad object_id
* Phobos_file GET test for bad object_id
* meta data is only written upon PUT operation in disk storage

Fix a documentation build issue: Now builds correctly without documentation enabled (Cmake variable CUSTOM_DOCU_PATH)

This reverts commit 1c7a83c1