1. 12 Aug, 2021 1 commit
  2. 23 Nov, 2020 1 commit
    • Ciarán Ó Rourke's avatar
      Add Bucket Listing · 735d8bae
      Ciarán Ó Rourke authored
      Create object in backend for buckets to affirm their existence and to
      store meta data. Support S3 bucket listing. Generalise bucket logic to
      'container' where the protocol should not be known.
      
      Change log
      * Disk
          - append to bucket list instead of overwriting
          - parse bucket name from object path
          - store list of objects in each bucket in a unique file
          - store list of buckets in a file
          - get_object_list function
      * use bucket + object path in unit tests
      * test Fifo::get_bucket_object_list
      * bucket listing in stream, storage and disk implementation
      * list buckets using Phobos
      * test for Phobos bucket listing
      * function to format ListBuckets response
      * bucket list and object list request identification
      * handle bucket list request
      * list buckets integration test
      * https confict response if bucket already exists for create_container
      * bucket list unit tests ignore buckets created by previous test runs
      * function to reset Phobos descriptor
      * function to reset Fifo descriptors and storage backend
      * make sure to deallocate objid if it is allocated before reallocating
      * function to check if object exists in the Phobos store
      * function to check if object exists in Disk store
      * test object_exists functions
      * create container if object is PUT to a not yet existing one
      * create container function in PutRequestHandler
      * remove bucket logic from Disk
      * remove bucket tests from Phobos_file and Fifo tests
      * remove bucket logic from Stream and Storage as this is handled elsewhere
      * handle bucket logic in Protocol Formatter
      * rename bucket to container where appropriate
      * support create bucket request
      * test create bucket request
      * function to check for container list request
      * use sane bucket identifier
      * function to extract container name
      * use try-catch in create_container function
      * test that creating an existing container fails
      * update documentation
      * flush log messages to logfile immediately
      735d8bae
  3. 27 Oct, 2020 1 commit
    • Sophie Wenzel-Teuber's avatar
      Modularise S3 protocol · b11992ae
      Sophie Wenzel-Teuber authored
      Add a protocol interface to abstract the s3 implementation away from the server. The server handlers now call generic protocol functions on objects created in a protocol factory. Only this factory knows, that the underlying objects are S3.
      
      ChangeLog:
      * add virtual classes for protocols
      * move s3 specific implementations to own folder/namespace and inherit generic protocol
      * add protocol_factory to generate s3 objects for the server
      * move fifo and phobos implementations to own namespace to mirror the same layout as for the protocol
      b11992ae
  4. 28 Sep, 2020 1 commit
    • Ciarán Ó Rourke's avatar
      Object Listing · 2c31093f
      Ciarán Ó Rourke authored
      Support the list objects functionality of s3. Bucket only GET requests
      get a response with a list of the objects in the bucket
      
      Change log:
      * add list object functions to the phobos_cpp_wrapper
      * add listing objects to storage functionality
      * add bucket object listing wrapper to stream
      * empty list objects function is disk storage backend to satisfy storage class
      * string_to_map function to aid returning object_list in storage
      * remove Catch2 header file
      * oid_to_key function to parse away bucket from object name
      * s3_utilities: function to create xml response for list objects
      * bucket only GET request calls list objects
      * get_request_handler: respond with object_list for bucket only requests
      * add object listing to unit tests
      * integration test for object listing
      * unit test for empty bucket
      * integration test for empty bucket
      * Use bucket and key in unit tests to reflect actual usage
      * generate random bucket name for testing so number of objects in
          returned list can be measured
      * replace assertTrue with assertEqual in integration tests where appropriate
      * number integration tests to ensure order (alphanumerical) of execution
      * instruct integration tests to stop on first error
      * free object meta data struct
      * fix development flags not being used during debug build
      * use debug build for CI
      * use release build for CentOS7
      * appease development flags
      * disable LeakSanitization for CI (does not work on Gitlab-CI)
      * source relevant env list in CI before_script
      * export LD_LIBRARY_PATH in CI configure step
      * pass bucket name to xml parser to avoid attempting to access an object of an empty list
      * update documentation
      2c31093f
  5. 02 Sep, 2020 1 commit
  6. 21 Aug, 2020 1 commit
    • Ciarán Ó Rourke's avatar
      Get CI running on CentOS8 · 4557e4d4
      Ciarán Ó Rourke authored
      Change log:
      * get unit tests and integration tests working for default job
      * get format job working
      * satisfy clang-format
      * get Documentation job working
      * use custom target_link_system_libraries command to force system
      includes
      * statements inside braces to satisfy clang-tidy
      * remove clang-tidy macro checks
      4557e4d4
  7. 18 Aug, 2020 2 commits
    • Ciarán Ó Rourke's avatar
      Support CentOS8 · 3847d8d1
      Ciarán Ó Rourke authored
      Various adjustments to build and pass tests on CentOS8
      
      Change log:
      * move setting of installation directories to src/CMakeLists.txt (needs to be where target is created)
      * explicitly find Boost::iosteams
      * buffer for storing empty file causes exception so add 1 to size
      3847d8d1
    • Ciarán Ó Rourke's avatar
      Support newer Phobos version · 4f72c208
      Ciarán Ó Rourke authored
      Changes to Phobos wrapper to reflect updated Phobos API.
      
      Change log:
      * make unit test a make target again
      * remove deprecated and redundant target_link_directories call
      * redesign phobos_cpp_wrapper for new Phobos version
      * hardcode request of directory storage in Phobos for now
      4f72c208
  8. 21 Jul, 2020 1 commit
  9. 07 Jul, 2020 1 commit
    • Sophie Wenzel-Teuber's avatar
      Update Deimos to new superbuild · bc23380f
      Sophie Wenzel-Teuber authored
      Changelog:
      * Add Find scripts to CMake to search for phobos and spdlog
      * Add integrations tests to cmake by just copying the python code to the build directory
      * Update Readme to reflect new SuperDeimos layout and add brief description
      bc23380f
  10. 03 Jul, 2020 1 commit
  11. 02 Jul, 2020 1 commit
    • Ciarán Ó Rourke's avatar
      * project version · ac507741
      Ciarán Ó Rourke authored
      * set install directories through CMake
      * print built subdirectories during build
      * set up installation directories
      * use development flags during debug build
      * remove authorisation double negative cmake variable
      * update README
      ac507741
  12. 22 Jun, 2020 1 commit
    • Sophie Wenzel-Teuber's avatar
      Fix a bug in S3 header · 5f97c454
      Sophie Wenzel-Teuber authored
      get body length was throwing an exception when the Content-Length header was missing
      
      Changelog:
      * Add more error checks to s3 header class
      * Add unit tests for s3 header class covering all typical use cases
      * Add check for Markdown documentation to gitlab CI
      5f97c454
  13. 18 Jun, 2020 2 commits
    • Ciarán Ó Rourke's avatar
      Rename Project · 2238f58c
      Ciarán Ó Rourke authored
      Rename project from FiPhoboServer to Deimos
      
      Change log:
      * replace all occurances of variations of fiphoboserver with its deimos
      equivalent
      2238f58c
    • Ciarán Ó Rourke's avatar
      Implement logging · 99ca98cf
      Ciarán Ó Rourke authored
      Use the spdlog library for project wide logging to a file. Path of
      logfile is configurable as command line option.
      
      Change log:
      * use spdlog
      * set global logging to file
      * logfile as command line option
      * update README command line option section with logfile option
      * update README command line option section with default values for options
      * control log level with CMake build type variable
      * log received request
      * get request logging
      * put request logging
      * getmd request logging
      * fifo logging
      * rename authorisation macro to comply with linter
      * use rotating files with size limit
      99ca98cf
  14. 15 Jun, 2020 1 commit
  15. 11 Jun, 2020 1 commit
    • Sophie Wenzel-Teuber's avatar
      Update the Metadata generation · ac693d84
      Sophie Wenzel-Teuber authored
      Changelog:
      * Add `fiphobo:` prefix to all metadata saved by our server, that should not be sent back directly
      * When sending the metadata, all entries starting with `fiphobo:` are ignored
      * remove the bucket_name entry from all set metadata functions throughout the server and add the bucketname where all other metdata is set (put handler)
      ac693d84
  16. 10 Jun, 2020 1 commit
    • Sophie Wenzel-Teuber's avatar
      Update the server to work with Estuary copytool · dc31d3c1
      Sophie Wenzel-Teuber authored
      Changelog:
      * Add content_length to phobos metadata
      * Send the content_length from phobos metadata as header Content-Length
      * Add all Phobos Metadata (formerly retrieved only by a HEAD request) as headers to the response to a GET request
      dc31d3c1
  17. 26 May, 2020 1 commit
  18. 13 May, 2020 1 commit
    • Sophie Wenzel-Teuber's avatar
      More flexible error creation · 499566af
      Sophie Wenzel-Teuber authored
      All different kinds of errors our server sends are in compliance with Amazon S3 errors. They are now stored in one place and should be accessed from anywhere.
      
      Changelog:
      * add the file s3_errors.h that contains const objects for all errors that might occur (most of them are not checked yet)
      * change struct error_info into a class and add a function to create the XML from the object's variables
      * Use these errors in the request handlers
      * Add an error object to the s3 authorisation that will always contain "Access Denied" or another error if applicable
      * Update tests to test the code generation and error return values
      499566af
  19. 07 May, 2020 1 commit
    • Ciarán Ó Rourke's avatar
      Make server a daemon process. Support for start and stop subcommands. · 1f7100b0
      Ciarán Ó Rourke authored
      Outline usage profile of the fiphoboserver executable in the README
      
      Change log:
      * start server daemon subcommand
      * move appropriate flags to start subcommand
      * main function forks process given start command
      * stop command to kill running server process
      * integration tests: reflect subcommand invoking of server
      * detail usage in README
      1f7100b0
  20. 05 May, 2020 1 commit
    • Ciarán Ó Rourke's avatar
      Support command line parsing and configuration files · ba82dd36
      Ciarán Ó Rourke authored
      Change log:
      * src/utils directory for configuration, logging, etc
      * Config class for handling configuration (command line parsing, etc)
      * include Catch2 and CLI11 as system files so clang-tidy will ignore
        them
      * support basic command line options:
            - hostname (positional, required)
            - number of threads
            - http_port
            - https_port
      * add configuration to fiphoboserver executable main function
      * return on exception for incompatible hostname
      * return after parsing command line arguments if arguments are bad
        or --help is supplied
      * don't pass command line arguments to folly
      * Config returns struct of command line options
      * accept configuration options from config file
      * option to write current configuration to file
      * integration tests: reflect changes to fiphoboserver executable
      ba82dd36
  21. 24 Apr, 2020 1 commit
  22. 15 Apr, 2020 1 commit
    • Sophie Wenzel-Teuber's avatar
      Add Amazon Authorisation · 18b7825f
      Sophie Wenzel-Teuber authored
      The signature of amazon users is checked against a text file of known users and requests are accepted or rejected depending on the signature
      18b7825f
  23. 09 Apr, 2020 1 commit
  24. 08 Apr, 2020 1 commit
    • Sophie Wenzel-Teuber's avatar
      Add S3 error handling · b45eb717
      Sophie Wenzel-Teuber authored and Aaron Dees's avatar Aaron Dees committed
      All errors returned from phobos now have at least a little distinct error messages (as far as the return values permit it) and send an S3 readable XML along with the error number.
      b45eb717
  25. 02 Apr, 2020 1 commit
    • Ciarán Ó Rourke's avatar
      * src/storage/phobos_file.cc: need to set xd_objid to NULL to avoid double free · 1392c1f9
      Ciarán Ó Rourke authored
      * src/storage/phobos_file.hexplicit copy constructors to avoid double free of xd_objid
      * tests: reflect meta data change to std::map from std::vector
      * run Clang Format
      * resolve Clang Tidy warnings
      * add Gitlab-CI
      * use fiphoboserver docker image
      * update DockerHub image with Doxygen requirements
      * CI: Clang Format job
      * CI: Clang Tidy job
      * CI: Doxygen build job
      * CI: Unit Tests job
      1392c1f9
  26. 31 Mar, 2020 1 commit
  27. 30 Mar, 2020 1 commit
    • Ciarán Ó Rourke's avatar
      Squashed commit of the following: · 755f0d1a
      Ciarán Ó Rourke authored
      * utility functions for tests (utils.h, utils.cc)
      * copy test text files to test binary directory during build
      * tests for Phobos_file constructors
      * tests for Phobos_file PUT and GET
      * Phobos_file PUT and GET tests for empty file
      * Phobos_file GETMD test
      * in-memory (disk) storage solution for testing stream objects
      * Fifo file descriptors must be copied explicitly as they are now an array
      * Fifo tests for PUT and GET
      * repeat Fifo tests for empty file
      * Fifo tests for GETMD
      * tests for Fifo object construction
      * need to determine file size before db_get
      * Phobos_file PUT  test for bad file descriptor exception
      * Phobos_file PUT test for used object_id exception
      * Phobos_file GETMD test for bad object_id
      * Phobos_file GET test for bad object_id
      * meta data is only written upon PUT operation in disk storage
      755f0d1a
  28. 24 Mar, 2020 1 commit
  29. 16 Mar, 2020 2 commits
  30. 13 Mar, 2020 1 commit
  31. 12 Mar, 2020 1 commit
    • Ciarán Ó Rourke's avatar
      Implement Clang Tidy · daa9c80e
      Ciarán Ó Rourke authored
      Change log:
      * use std::move to return from get_meta_data
      * Fifo::put: replace Folly object argument with void pointer and size_t as arguments and return ssize_t
      * Fifo::put considers rc < count to be an exception
      * run Clang Tidy
      * rename i_ps to IPs in main.cc
      daa9c80e
  32. 05 Mar, 2020 1 commit
  33. 27 Feb, 2020 1 commit
    • Ciarán Ó Rourke's avatar
      Support for getmd feature · 3269d433
      Ciarán Ó Rourke authored
      Change log:
      * Send bad message upon entering unsupported request handler
      * add c++ wrapper for pho_attrs_foreach
      * Phobos_file::get_meta_data returns vector of key-value pairs
      * Phobos_file::get_meta_data functions like:
      * phobos_getmd_cpp fills the descriptor with attributes for the requested object
      * phobos_attrs_foreach_cpp loops over the attrs and returns them to the stream
      * Fifo::get_meta_data simply passes on the result of the storage call
      * Fifo::get_meta_data returns vector of key-value pairs
      * add Getmd request handler
      * add getmd_request_handler.h to headers
      * request handler has a meta_data member variable
      * fill meta_data variable with Phobos attributes
      * use repeated calls to ResponseBuilder.header to fill header with Phobos attributes
      * send entire message at once
      * add support for testing getmd
      3269d433
  34. 24 Feb, 2020 1 commit
  35. 20 Feb, 2020 3 commits