ExSeisDat issueshttps://git.ichec.ie/oilgas/ExSeisDat/-/issues2019-01-07T11:42:16Zhttps://git.ichec.ie/oilgas/ExSeisDat/-/issues/106Clean up Python tests and bindings2019-01-07T11:42:16ZChristopher WernerClean up Python tests and bindingshttps://git.ichec.ie/oilgas/ExSeisDat/-/issues/105Integrate Python bindings with cmake2019-01-07T11:40:55ZChristopher WernerIntegrate Python bindings with cmakeFollowing the initial bindings of python to C++, ensure it can be integrated with cmakeFollowing the initial bindings of python to C++, ensure it can be integrated with cmakehttps://git.ichec.ie/oilgas/ExSeisDat/-/issues/90Python API2019-01-07T15:40:54ZChristopher WernerPython APIS-01443
As a Tullow Geophysicist, I want to be able to call PIOL features from Python because I don't know or necessarily have experience with the language the PIOL is written in.
**Pre-Dev Notes:**
A number of operations are done in ...S-01443
As a Tullow Geophysicist, I want to be able to call PIOL features from Python because I don't know or necessarily have experience with the language the PIOL is written in.
**Pre-Dev Notes:**
A number of operations are done in ExSeisDat during class destruction. This doesn't play well with Python lifetime management. Implicit operations in the destructor should be made available explicitly, either as a method, or done at the start of other write functions when the file metadata has been changes on the object.
SWIG can be used to automatically generate bindings from C++ to a number of languages, i.e. Python.
The SWIG based wrapping needs to be tested. The `wraptests` should cover this.
The C and C++ API names have become out-of-sync. These need to be updated.
Ideally, some mechanism could be put in place to confirm the coverage and naming schemes of the C wrapper, but this could prove difficult.
Ensure all destructors of exposed classes can also be called explicitly. This is needed to match the memory model of Python. Currently, some classes perform actions such as writing to file, or executing a pipeline, during destruction. Since Python is garbage collected, when this action is completed is a lot less clear than it is in C++.
**Implementation Notes:**
{- ADDME -}
**System Test Changes:**
{- ADDME -}
**Bug Fixes:**
{- ADDME -}
**C++ API Changes:**
{- ADDME -}
**C API Changes:**
{- ADDME -}
**Success Criteria:**
* The full API is available through python calls
* The Python API is documented
* The user guide is updated to reflect python usage
* At least one python API example is created
**Tasks**
* [ ] Ensure all destructors of exposed classes can also be called explicitly
* [ ] Refactor: Re-sync the C and C++ APIs
* [ ] Add Python bindings
* [ ] Test Python bindings
CREATED ON - 20/04/2016March 2019 Roadmap2019-01-01https://git.ichec.ie/oilgas/ExSeisDat/-/issues/9Test Python bindings2018-11-06T14:08:15ZPádraig Ó ConbhuíTest Python bindingsThe SWIG based wrapping needs to be tested. The `wraptests` should cover this.The SWIG based wrapping needs to be tested. The `wraptests` should cover this.March 2019 Roadmap2019-01-01https://git.ichec.ie/oilgas/ExSeisDat/-/issues/8Add Python bindings2018-12-17T15:37:30ZPádraig Ó ConbhuíAdd Python bindingsSWIG can be used to automatically generate bindings from C++ to a number of languages, i.e. Python.SWIG can be used to automatically generate bindings from C++ to a number of languages, i.e. Python.March 2019 Roadmap2019-01-01https://git.ichec.ie/oilgas/ExSeisDat/-/issues/7Refactor: Re-sync the C and C++ APIs2018-11-06T14:07:36ZPádraig Ó ConbhuíRefactor: Re-sync the C and C++ APIsThe C and C++ API names have become out-of-sync. These need to be updated.
Ideally, some mechanism could be put in place to confirm the coverage and naming schemes of the C wrapper, but this could prove difficult.The C and C++ API names have become out-of-sync. These need to be updated.
Ideally, some mechanism could be put in place to confirm the coverage and naming schemes of the C wrapper, but this could prove difficult.March 2019 Roadmap2019-01-01https://git.ichec.ie/oilgas/ExSeisDat/-/issues/111Seis Format2019-03-04T21:21:44ZPádraig Ó ConbhuíSeis FormatAdd support for reading / writing in Shearwater's `Seis` format.Add support for reading / writing in Shearwater's `Seis` format.https://git.ichec.ie/oilgas/ExSeisDat/-/issues/103SEIS to SEGY (serial)2018-11-06T14:07:10ZChristopher WernerSEIS to SEGY (serial)S-01719
As a ExSeisDat Developer I want to write a SEIS to SEGY utility with a serial implantation for reading seis based implementation to read SEIS and ExSeisDat to write the SEGY file because I want to better understand the I/O patte...S-01719
As a ExSeisDat Developer I want to write a SEIS to SEGY utility with a serial implantation for reading seis based implementation to read SEIS and ExSeisDat to write the SEGY file because I want to better understand the I/O patterns of SEIS while also providing an initial version of the SEIS to SEGY converter.
**PreDev Notes:**
Description of the compression algorithm:
Seis traces can be stored as 1 byte (int16), 2 byte (int32), or 4 byte (double) elements. When the traces are compressed from doubles, they are sectioned into groups (packets) that each get a scalar, equal to the largest number able to be stored in the compressed integer divided by the largest (magnitude) number in the packet. Every other element in the packet is multiple by that scalar.
**Implementation Notes:**
The JSON file is parsed when the Seis reader is initialised and the header descriptors within the JSON file are also parsed at this time. Since the Seis header entries are self describing, they do not require the complex rule structure that SEGY did. Rather, a PIOL_META value can be mapped to the default seis header name and then simply read in. It reads each header parameter individually (i.e. it does not read in the entire file and then parse it). The traces are read in, using the parsed JSON header to determine the compression type and packet size.
Further improvements to reader:
* Currently the seis reader does not read in files with multiple .hdr/.tr subfiles.
* Currently the seis header reads in all entries for a particular metadata entry; it cannot get a header value from a single trace.
* Currently the seis header does not read in the entire .hdr and then parse the data, it only reads in the values used.
* Needs to write a checksum "test" to show that it works
* Needs to write utility SEIS to SEGY
**System Test Changes:**
{- ADDME -}
**Bug Fixes:**
{- ADDME -}
**C++ API Changes:**
{- ADDME -}
**C API Changes:**
{- ADDME -}
**Success Criteria:**
* Reader should read in the type of SEIS files used by Tullow
* Checksums for original SEGY and SEIS to SEGY should be the same
**Tasks:**
* [ ] Review
* [x] Read Json file
* [x] Read tr file
* [ ] Read HDR file (in prog)
* [ ] SEIS to SEGY util (in prog)
CREATED ON - 01/05/2018March 2019 Roadmap2019-03-01https://git.ichec.ie/oilgas/ExSeisDat/-/issues/102SEGY to SEIS (serial)2018-11-06T14:07:02ZChristopher WernerSEGY to SEIS (serial)S-01713
As a ExSeisDat Developer I want to write a SEIS to SEGY utility using MPI IO to read because I want to better understand the I/O patterns of SEIS while also providing an initial version of the SEGY to SEIS converter.
**PreDev ...S-01713
As a ExSeisDat Developer I want to write a SEIS to SEGY utility using MPI IO to read because I want to better understand the I/O patterns of SEIS while also providing an initial version of the SEGY to SEIS converter.
**PreDev Notes:**
{- ADDME -}
**Implementation Notes:**
{- ADDME -}
**System Test Changes:**
{- ADDME -}
**Bug Fixes:**
{- ADDME -}
**C++ API Changes:**
{- ADDME -}
**C API Changes:**
{- ADDME -}
**Success Criteria:**
* SEIS file written should be format that is usable by Tullow
* SEIS files that are tested should be able to be read by Reveal and converted into SEG-Y; the SEG-Y file should be the equivalent to the original.
**Done:**
Currently, we can write traces into the tr files serially and headers into hdr files serially. Writing data into the seis file is partially complete, and data needs to be stored into classes.
**Tasks:**
* [ ] Review
* [x] Write Traces into .tr files
* [x] Write Headers into .hd file
* [x] Write Metadata into .seis metadata file
* [x] Write particular headers into .db files
* [ ] Write complete SEGY to SEIS utility
CREATED ON - 03/04/2018March 2019 Roadmap2019-03-01https://git.ichec.ie/oilgas/ExSeisDat/-/issues/97SEIS Rule Structure (P1)2018-11-06T14:06:54ZChristopher WernerSEIS Rule Structure (P1)As a ExSeisDat Developer I want a SEIS specific Rule structure for reading / writing SEIS file headers.
**Pre-Dev Notes:**
The current Rule structure can be renamed to SEGY_Rule. An interface Rule class can replace SEGY_Rule, with SEGY...As a ExSeisDat Developer I want a SEIS specific Rule structure for reading / writing SEIS file headers.
**Pre-Dev Notes:**
The current Rule structure can be renamed to SEGY_Rule. An interface Rule class can replace SEGY_Rule, with SEGY_Rule inheriting from it. And a SEGY_Rule instance should be attached to the SEGY reader / writer, and a SEIS_Rule instance attached to the SEIS reader / writer.
We will only worry about standard headers for the moment.
**Implementation Notes:**
The hard-coded SEGY header byte locations have been removed from the existing rule structure. Now the SEG-Y rules get the byte offsets from a separate file in a similar fashion to the SEIS rules.
**System Test Changes:**
{- ADDME -}
**Bug Fixes:**
Coordinate Headers for SEGY (source X, Source Y, Receiver X, Receiver Y) was being read in as a float, while it was being stored in the SEGY file as a double. They are now read in as doubles to prevent precision loss.
**C++ API Changes:**
{- ADDME -}
**C API Changes:**
{- ADDME -}
**Success Criteria:**
{- ADDME -}
CREATED ON 06/06/2018 March 2019 Roadmaphttps://git.ichec.ie/oilgas/ExSeisDat/-/issues/13Add Seis reader & writer2018-11-06T14:06:33ZPádraig Ó ConbhuíAdd Seis reader & writerS-01722
As a ExSeisDat Developer I want to write a SEGY to SEIS utility in parallel based implementation to read SEIS and ExSeisDat to write the SEGY file because I want to better understand the I/O patterns of SEIS while also providing...S-01722
As a ExSeisDat Developer I want to write a SEGY to SEIS utility in parallel based implementation to read SEIS and ExSeisDat to write the SEGY file because I want to better understand the I/O patterns of SEIS while also providing an initial version of the SEIS to SEGY converter
**PreDev Notes:**
{- ADDME -}
**Implementation Notes:**
{- ADDME -}
**System Test Changes:**
{- ADDME -}
**Bug Fixes:**
{- ADDME -}
**C++ API Changes:**
{- ADDME -}
**C API Changes:**
{- ADDME -}
**Success Criteria:**
Reader should read in the type of SEIS files used by Tullow
**Tasks**
* [x] Review of Parallel Write
* [ ] Review of Parallel Read
CREATED ON 01/05/2018March 2019 Roadmap2019-03-01https://git.ichec.ie/oilgas/ExSeisDat/-/issues/12Refactor: Generalize file reader and writer tests for arbitrary file formats2018-11-06T14:06:25ZPádraig Ó ConbhuíRefactor: Generalize file reader and writer tests for arbitrary file formatsThe file reader writer tests can use a mock I/O class, eliminating the need for actual I/O during correctness tests. Mock I/O classes can also check the reader and writer classes are reading / writing the correct bytes to the correct par...The file reader writer tests can use a mock I/O class, eliminating the need for actual I/O during correctness tests. Mock I/O classes can also check the reader and writer classes are reading / writing the correct bytes to the correct part of the file.
Another approach is to use a distributed in-memory file class, and check what has been written to the file, or initialize the file in-memory.March 2019 Roadmap2019-03-01https://git.ichec.ie/oilgas/ExSeisDat/-/issues/11Refactor: Clean up the Reader and Writer interfaces for clearer specialization2019-03-04T21:24:23ZPádraig Ó ConbhuíRefactor: Clean up the Reader and Writer interfaces for clearer specializationAlso re-design or just remove the "model" reader files.Also re-design or just remove the "model" reader files.March 2019 Roadmap2019-03-01https://git.ichec.ie/oilgas/ExSeisDat/-/issues/107Rewrite tests using Catch22019-03-07T13:13:35ZPádraig Ó ConbhuíRewrite tests using Catch2As a user of the library, I would like some proof that the library does what it says it does. I'd also like some advanced examples of how to use the library.
The current test suite suggests it does, but its spaghetti-layout makes that i...As a user of the library, I would like some proof that the library does what it says it does. I'd also like some advanced examples of how to use the library.
The current test suite suggests it does, but its spaghetti-layout makes that impossible to verify by eye.
Rewriting the tests for the core features needed to implement `ReadSEGY` and `WriteSEGY` with a framework like Catch2, with clarity of test cases in mind, should fulfill these needs.Aaron DeesAaron Deeshttps://git.ichec.ie/oilgas/ExSeisDat/-/issues/95Auto build and deploy documentation2018-11-06T13:57:24ZChristopher WernerAuto build and deploy documentationS-01724
As a ExSeisDat Developer I want the Doxygen documentation automatically built and deployed on GitHub so the documentation is available and up to date.
**PreDev Notes:**
Using Travis CI integration with GitHub, it should be pos...S-01724
As a ExSeisDat Developer I want the Doxygen documentation automatically built and deployed on GitHub so the documentation is available and up to date.
**PreDev Notes:**
Using Travis CI integration with GitHub, it should be possible to add a script to build the docs, then add them to a GitHub Pages repo for ExSeisDat, likely at ichec.github.io/exseisdat, whenever a push is made to the devel branch.
**Implementation Notes:**
{- ADDME -}
**System Test Changes:**
{- ADDME -}
**Bug Fixes:**
{- ADDME -}
**C++ API Changes:**
{- ADDME -}
**C API Changes:**
{- ADDME -}
**Success Criteria:**
Pushing code to the devel branch automatically builds and updates the online documentation for the head of the devel branch.
CREATED ON 08/05/2018https://git.ichec.ie/oilgas/ExSeisDat/-/issues/94Tooling for documenting tests and cross-referencing them with the code they test2018-11-06T13:57:00ZChristopher WernerTooling for documenting tests and cross-referencing them with the code they testS-01743
As a ExSeisDat Developer I want the tests to be documentable, with a cross-reference between functions and their unit tests.
**PreDev Notes:**
Doxygen has capabilities for cross-referencing. It looks like there's a \test comma...S-01743
As a ExSeisDat Developer I want the tests to be documentable, with a cross-reference between functions and their unit tests.
**PreDev Notes:**
Doxygen has capabilities for cross-referencing. It looks like there's a \test command, but it's not clear if it does the cross-referencing.
**Implementation Notes:**
{- ADDME -}
**System Test Changes:**
{- ADDME -}
**Bug Fixes:**
{- ADDME -}
**C++ API Changes:**
{- ADDME -}
**C API Changes:**
{- ADDME -}
**Success Criteria:**
The test documentation, and the test to function cross reference should reside with the test code.
When looking at a function in the documentation, a link to the relevant test or tests should be available.
CREATED ON 03/08/2018https://git.ichec.ie/oilgas/ExSeisDat/-/issues/91Create C and C++ examples2018-11-06T14:05:37ZChristopher WernerCreate C and C++ examplesS-01704
As a Tullow Geophysicist I want examples of how to use ExSeisDat in C and C++ environments, as I want to use the library and may not have experience in the language
**PreDev Notes:**
Make sure examples have both C and C++ vers...S-01704
As a Tullow Geophysicist I want examples of how to use ExSeisDat in C and C++ environments, as I want to use the library and may not have experience in the language
**PreDev Notes:**
Make sure examples have both C and C++ versions.
The files produced should be the same (use checksum to check them).
**Implementation Notes:**
{- ADDME -}
**System Test Changes:**
{- ADDME -}
**Bug Fixes:**
{- ADDME -}
**C++ API Changes:**
{- ADDME -}
**C API Changes:**
{- ADDME -}
**Success Criteria:**
{- ADDME -}
CREATED ON - 09/01/2018March 2019 Roadmap2019-03-01https://git.ichec.ie/oilgas/ExSeisDat/-/issues/15Metadata to CSV Converter2018-11-06T13:56:42ZPádraig Ó ConbhuíMetadata to CSV ConverterAs a ExSeisDat Developer I want to create a utility to print and save every x number of values for a metadata parameter as a csv/txt because I need to use it for QC and bringing the values into other visualisation programs.
## Pre-Dev N...As a ExSeisDat Developer I want to create a utility to print and save every x number of values for a metadata parameter as a csv/txt because I need to use it for QC and bringing the values into other visualisation programs.
## Pre-Dev Notes
An end-user should be able to pick a tracer header parameter or parameters (ie Source X, Inline, etc).
## Utility command CLI:
list_headers -i input_file -o output_file -p parameter1[,parameter2[,parameter3[...]]] -n output_every_n_trace
### Example invocation:
list_headers -i example_file.segy -o example_file_metadata.csv -p TRACE_NUMBER,SOURCE_X,SOURCE_Y -n 5
File could look like (e.g.):
~~~
# TRACE_NUMBER, SOURCE_X, SOURCE_Y
0, ..., ...
5, ..., ...
10, ..., ...
15, ..., ...
...
~~~
## Success Criteria:
User inputs a SEGY file and the program outputs a CSV file with the requested parameters and stride.Pádraig Ó ConbhuíPádraig Ó Conbhuí2018-11-07https://git.ichec.ie/oilgas/ExSeisDat/-/issues/5Finalise tutorial material and API documentation2018-11-06T13:58:13ZPádraig Ó ConbhuíFinalise tutorial material and API documentationS-01708
As a prospective end-user I want more information about the ExSeisDat project and the ExSeisPIOL and the ExSeisFlow so that I can use the libraries.
**PreDev Notes:**
An approach could be as follows: Add a doxygen command, e.g...S-01708
As a prospective end-user I want more information about the ExSeisDat project and the ExSeisPIOL and the ExSeisFlow so that I can use the libraries.
**PreDev Notes:**
An approach could be as follows: Add a doxygen command, e.g. `@docs_reviewed`, and add it to each reviewed piece of documentation. Ideally, there may be some mechanism for generating an error if a doc _doesn't_ have the command. Then, remove the command.
**Implementation Notes:**
{- ADDME -}
**System Test Changes:**
{- ADDME -}
**Bug Fixes:**
{- ADDME -}
**C++ API Changes:**
{- ADDME -}
**C API Changes:**
{- ADDME -}
**Success Criteria:**
* [ ] Ensure the API documentation is complete, correct, and useful.
* [ ] Documentation Review process is updated
CREATED ON - 05/02/2018
March 2019 Roadmap2019-01-01https://git.ichec.ie/oilgas/ExSeisDat/-/issues/4Finish auto-upload of documentation2018-11-06T13:58:01ZPádraig Ó ConbhuíFinish auto-upload of documentationFinalize where the public repository will be, and the automatic uploading of documentation.Finalize where the public repository will be, and the automatic uploading of documentation.March 2019 Roadmap2019-01-01