ExSeisDat issueshttps://git.ichec.ie/oilgas/ExSeisDat/-/issues2018-11-06T13:56:42Zhttps://git.ichec.ie/oilgas/ExSeisDat/-/issues/15Metadata to CSV Converter2018-11-06T13:56:42ZPádraig Ó ConbhuíMetadata to CSV ConverterAs a ExSeisDat Developer I want to create a utility to print and save every x number of values for a metadata parameter as a csv/txt because I need to use it for QC and bringing the values into other visualisation programs.
## Pre-Dev N...As a ExSeisDat Developer I want to create a utility to print and save every x number of values for a metadata parameter as a csv/txt because I need to use it for QC and bringing the values into other visualisation programs.
## Pre-Dev Notes
An end-user should be able to pick a tracer header parameter or parameters (ie Source X, Inline, etc).
## Utility command CLI:
list_headers -i input_file -o output_file -p parameter1[,parameter2[,parameter3[...]]] -n output_every_n_trace
### Example invocation:
list_headers -i example_file.segy -o example_file_metadata.csv -p TRACE_NUMBER,SOURCE_X,SOURCE_Y -n 5
File could look like (e.g.):
~~~
# TRACE_NUMBER, SOURCE_X, SOURCE_Y
0, ..., ...
5, ..., ...
10, ..., ...
15, ..., ...
...
~~~
## Success Criteria:
User inputs a SEGY file and the program outputs a CSV file with the requested parameters and stride.Pádraig Ó ConbhuíPádraig Ó Conbhuí2018-11-07https://git.ichec.ie/oilgas/ExSeisDat/-/issues/90Python API2019-01-07T15:40:54ZChristopher WernerPython APIS-01443
As a Tullow Geophysicist, I want to be able to call PIOL features from Python because I don't know or necessarily have experience with the language the PIOL is written in.
**Pre-Dev Notes:**
A number of operations are done in ...S-01443
As a Tullow Geophysicist, I want to be able to call PIOL features from Python because I don't know or necessarily have experience with the language the PIOL is written in.
**Pre-Dev Notes:**
A number of operations are done in ExSeisDat during class destruction. This doesn't play well with Python lifetime management. Implicit operations in the destructor should be made available explicitly, either as a method, or done at the start of other write functions when the file metadata has been changes on the object.
SWIG can be used to automatically generate bindings from C++ to a number of languages, i.e. Python.
The SWIG based wrapping needs to be tested. The `wraptests` should cover this.
The C and C++ API names have become out-of-sync. These need to be updated.
Ideally, some mechanism could be put in place to confirm the coverage and naming schemes of the C wrapper, but this could prove difficult.
Ensure all destructors of exposed classes can also be called explicitly. This is needed to match the memory model of Python. Currently, some classes perform actions such as writing to file, or executing a pipeline, during destruction. Since Python is garbage collected, when this action is completed is a lot less clear than it is in C++.
**Implementation Notes:**
{- ADDME -}
**System Test Changes:**
{- ADDME -}
**Bug Fixes:**
{- ADDME -}
**C++ API Changes:**
{- ADDME -}
**C API Changes:**
{- ADDME -}
**Success Criteria:**
* The full API is available through python calls
* The Python API is documented
* The user guide is updated to reflect python usage
* At least one python API example is created
**Tasks**
* [ ] Ensure all destructors of exposed classes can also be called explicitly
* [ ] Refactor: Re-sync the C and C++ APIs
* [ ] Add Python bindings
* [ ] Test Python bindings
CREATED ON - 20/04/2016March 2019 Roadmap2019-01-01https://git.ichec.ie/oilgas/ExSeisDat/-/issues/9Test Python bindings2018-11-06T14:08:15ZPádraig Ó ConbhuíTest Python bindingsThe SWIG based wrapping needs to be tested. The `wraptests` should cover this.The SWIG based wrapping needs to be tested. The `wraptests` should cover this.March 2019 Roadmap2019-01-01https://git.ichec.ie/oilgas/ExSeisDat/-/issues/8Add Python bindings2018-12-17T15:37:30ZPádraig Ó ConbhuíAdd Python bindingsSWIG can be used to automatically generate bindings from C++ to a number of languages, i.e. Python.SWIG can be used to automatically generate bindings from C++ to a number of languages, i.e. Python.March 2019 Roadmap2019-01-01https://git.ichec.ie/oilgas/ExSeisDat/-/issues/7Refactor: Re-sync the C and C++ APIs2018-11-06T14:07:36ZPádraig Ó ConbhuíRefactor: Re-sync the C and C++ APIsThe C and C++ API names have become out-of-sync. These need to be updated.
Ideally, some mechanism could be put in place to confirm the coverage and naming schemes of the C wrapper, but this could prove difficult.The C and C++ API names have become out-of-sync. These need to be updated.
Ideally, some mechanism could be put in place to confirm the coverage and naming schemes of the C wrapper, but this could prove difficult.March 2019 Roadmap2019-01-01https://git.ichec.ie/oilgas/ExSeisDat/-/issues/5Finalise tutorial material and API documentation2018-11-06T13:58:13ZPádraig Ó ConbhuíFinalise tutorial material and API documentationS-01708
As a prospective end-user I want more information about the ExSeisDat project and the ExSeisPIOL and the ExSeisFlow so that I can use the libraries.
**PreDev Notes:**
An approach could be as follows: Add a doxygen command, e.g...S-01708
As a prospective end-user I want more information about the ExSeisDat project and the ExSeisPIOL and the ExSeisFlow so that I can use the libraries.
**PreDev Notes:**
An approach could be as follows: Add a doxygen command, e.g. `@docs_reviewed`, and add it to each reviewed piece of documentation. Ideally, there may be some mechanism for generating an error if a doc _doesn't_ have the command. Then, remove the command.
**Implementation Notes:**
{- ADDME -}
**System Test Changes:**
{- ADDME -}
**Bug Fixes:**
{- ADDME -}
**C++ API Changes:**
{- ADDME -}
**C API Changes:**
{- ADDME -}
**Success Criteria:**
* [ ] Ensure the API documentation is complete, correct, and useful.
* [ ] Documentation Review process is updated
CREATED ON - 05/02/2018
March 2019 Roadmap2019-01-01https://git.ichec.ie/oilgas/ExSeisDat/-/issues/4Finish auto-upload of documentation2018-11-06T13:58:01ZPádraig Ó ConbhuíFinish auto-upload of documentationFinalize where the public repository will be, and the automatic uploading of documentation.Finalize where the public repository will be, and the automatic uploading of documentation.March 2019 Roadmap2019-01-01https://git.ichec.ie/oilgas/ExSeisDat/-/issues/3Add Tutorial2019-02-05T16:17:24ZPádraig Ó ConbhuíAdd TutorialAdd a tutorial in the form of a number of documented examples, forming a narrative.Add a tutorial in the form of a number of documented examples, forming a narrative.March 2019 Roadmap2019-01-01https://git.ichec.ie/oilgas/ExSeisDat/-/issues/2Document Examples2018-11-06T13:57:48ZPádraig Ó ConbhuíDocument ExamplesS-01659
As a prospective end-user I want more information about the ExSeisDat project and the ExSeisPIOL and the ExSeisFlow so that I can use the libraries.
**PreDev Notes:**
{- ADDME -}
**Implementation Notes:**
{- ADDME -}
**Syst...S-01659
As a prospective end-user I want more information about the ExSeisDat project and the ExSeisPIOL and the ExSeisFlow so that I can use the libraries.
**PreDev Notes:**
{- ADDME -}
**Implementation Notes:**
{- ADDME -}
**System Test Changes:**
{- ADDME -}
**Bug Fixes:**
{- ADDME -}
**C++ API Changes:**
{- ADDME -}
**C API Changes:**
{- ADDME -}
**Success Criteria:**
* [ ] Examples are fully documented & commented
* [ ] Documentation review process is updated
**Tasks:**
* [ ] Comment the examplesMarch 2019 RoadmapPádraig Ó ConbhuíPádraig Ó Conbhuí2019-01-01https://git.ichec.ie/oilgas/ExSeisDat/-/issues/103SEIS to SEGY (serial)2018-11-06T14:07:10ZChristopher WernerSEIS to SEGY (serial)S-01719
As a ExSeisDat Developer I want to write a SEIS to SEGY utility with a serial implantation for reading seis based implementation to read SEIS and ExSeisDat to write the SEGY file because I want to better understand the I/O patte...S-01719
As a ExSeisDat Developer I want to write a SEIS to SEGY utility with a serial implantation for reading seis based implementation to read SEIS and ExSeisDat to write the SEGY file because I want to better understand the I/O patterns of SEIS while also providing an initial version of the SEIS to SEGY converter.
**PreDev Notes:**
Description of the compression algorithm:
Seis traces can be stored as 1 byte (int16), 2 byte (int32), or 4 byte (double) elements. When the traces are compressed from doubles, they are sectioned into groups (packets) that each get a scalar, equal to the largest number able to be stored in the compressed integer divided by the largest (magnitude) number in the packet. Every other element in the packet is multiple by that scalar.
**Implementation Notes:**
The JSON file is parsed when the Seis reader is initialised and the header descriptors within the JSON file are also parsed at this time. Since the Seis header entries are self describing, they do not require the complex rule structure that SEGY did. Rather, a PIOL_META value can be mapped to the default seis header name and then simply read in. It reads each header parameter individually (i.e. it does not read in the entire file and then parse it). The traces are read in, using the parsed JSON header to determine the compression type and packet size.
Further improvements to reader:
* Currently the seis reader does not read in files with multiple .hdr/.tr subfiles.
* Currently the seis header reads in all entries for a particular metadata entry; it cannot get a header value from a single trace.
* Currently the seis header does not read in the entire .hdr and then parse the data, it only reads in the values used.
* Needs to write a checksum "test" to show that it works
* Needs to write utility SEIS to SEGY
**System Test Changes:**
{- ADDME -}
**Bug Fixes:**
{- ADDME -}
**C++ API Changes:**
{- ADDME -}
**C API Changes:**
{- ADDME -}
**Success Criteria:**
* Reader should read in the type of SEIS files used by Tullow
* Checksums for original SEGY and SEIS to SEGY should be the same
**Tasks:**
* [ ] Review
* [x] Read Json file
* [x] Read tr file
* [ ] Read HDR file (in prog)
* [ ] SEIS to SEGY util (in prog)
CREATED ON - 01/05/2018March 2019 Roadmap2019-03-01https://git.ichec.ie/oilgas/ExSeisDat/-/issues/102SEGY to SEIS (serial)2018-11-06T14:07:02ZChristopher WernerSEGY to SEIS (serial)S-01713
As a ExSeisDat Developer I want to write a SEIS to SEGY utility using MPI IO to read because I want to better understand the I/O patterns of SEIS while also providing an initial version of the SEGY to SEIS converter.
**PreDev ...S-01713
As a ExSeisDat Developer I want to write a SEIS to SEGY utility using MPI IO to read because I want to better understand the I/O patterns of SEIS while also providing an initial version of the SEGY to SEIS converter.
**PreDev Notes:**
{- ADDME -}
**Implementation Notes:**
{- ADDME -}
**System Test Changes:**
{- ADDME -}
**Bug Fixes:**
{- ADDME -}
**C++ API Changes:**
{- ADDME -}
**C API Changes:**
{- ADDME -}
**Success Criteria:**
* SEIS file written should be format that is usable by Tullow
* SEIS files that are tested should be able to be read by Reveal and converted into SEG-Y; the SEG-Y file should be the equivalent to the original.
**Done:**
Currently, we can write traces into the tr files serially and headers into hdr files serially. Writing data into the seis file is partially complete, and data needs to be stored into classes.
**Tasks:**
* [ ] Review
* [x] Write Traces into .tr files
* [x] Write Headers into .hd file
* [x] Write Metadata into .seis metadata file
* [x] Write particular headers into .db files
* [ ] Write complete SEGY to SEIS utility
CREATED ON - 03/04/2018March 2019 Roadmap2019-03-01https://git.ichec.ie/oilgas/ExSeisDat/-/issues/91Create C and C++ examples2018-11-06T14:05:37ZChristopher WernerCreate C and C++ examplesS-01704
As a Tullow Geophysicist I want examples of how to use ExSeisDat in C and C++ environments, as I want to use the library and may not have experience in the language
**PreDev Notes:**
Make sure examples have both C and C++ vers...S-01704
As a Tullow Geophysicist I want examples of how to use ExSeisDat in C and C++ environments, as I want to use the library and may not have experience in the language
**PreDev Notes:**
Make sure examples have both C and C++ versions.
The files produced should be the same (use checksum to check them).
**Implementation Notes:**
{- ADDME -}
**System Test Changes:**
{- ADDME -}
**Bug Fixes:**
{- ADDME -}
**C++ API Changes:**
{- ADDME -}
**C API Changes:**
{- ADDME -}
**Success Criteria:**
{- ADDME -}
CREATED ON - 09/01/2018March 2019 Roadmap2019-03-01https://git.ichec.ie/oilgas/ExSeisDat/-/issues/13Add Seis reader & writer2018-11-06T14:06:33ZPádraig Ó ConbhuíAdd Seis reader & writerS-01722
As a ExSeisDat Developer I want to write a SEGY to SEIS utility in parallel based implementation to read SEIS and ExSeisDat to write the SEGY file because I want to better understand the I/O patterns of SEIS while also providing...S-01722
As a ExSeisDat Developer I want to write a SEGY to SEIS utility in parallel based implementation to read SEIS and ExSeisDat to write the SEGY file because I want to better understand the I/O patterns of SEIS while also providing an initial version of the SEIS to SEGY converter
**PreDev Notes:**
{- ADDME -}
**Implementation Notes:**
{- ADDME -}
**System Test Changes:**
{- ADDME -}
**Bug Fixes:**
{- ADDME -}
**C++ API Changes:**
{- ADDME -}
**C API Changes:**
{- ADDME -}
**Success Criteria:**
Reader should read in the type of SEIS files used by Tullow
**Tasks**
* [x] Review of Parallel Write
* [ ] Review of Parallel Read
CREATED ON 01/05/2018March 2019 Roadmap2019-03-01https://git.ichec.ie/oilgas/ExSeisDat/-/issues/12Refactor: Generalize file reader and writer tests for arbitrary file formats2018-11-06T14:06:25ZPádraig Ó ConbhuíRefactor: Generalize file reader and writer tests for arbitrary file formatsThe file reader writer tests can use a mock I/O class, eliminating the need for actual I/O during correctness tests. Mock I/O classes can also check the reader and writer classes are reading / writing the correct bytes to the correct par...The file reader writer tests can use a mock I/O class, eliminating the need for actual I/O during correctness tests. Mock I/O classes can also check the reader and writer classes are reading / writing the correct bytes to the correct part of the file.
Another approach is to use a distributed in-memory file class, and check what has been written to the file, or initialize the file in-memory.March 2019 Roadmap2019-03-01https://git.ichec.ie/oilgas/ExSeisDat/-/issues/11Refactor: Clean up the Reader and Writer interfaces for clearer specialization2019-03-04T21:24:23ZPádraig Ó ConbhuíRefactor: Clean up the Reader and Writer interfaces for clearer specializationAlso re-design or just remove the "model" reader files.Also re-design or just remove the "model" reader files.March 2019 Roadmap2019-03-01https://git.ichec.ie/oilgas/ExSeisDat/-/issues/14Final ExSeisDat project reports2018-11-06T13:57:34ZPádraig Ó ConbhuíFinal ExSeisDat project reports* [ ] D 2.0: Report on the development and implementation of PIOL
* [ ] D 2.2: Beta release of PIOL
* [ ] D 3.3: Report on the development of the VFS (aka ExSeisFlow)
* [ ] D 3.4: Beta release of VFS (ExSeisFlow)* [ ] D 2.0: Report on the development and implementation of PIOL
* [ ] D 2.2: Beta release of PIOL
* [ ] D 3.3: Report on the development of the VFS (aka ExSeisFlow)
* [ ] D 3.4: Beta release of VFS (ExSeisFlow)March 2019 Roadmap2019-03-31https://git.ichec.ie/oilgas/ExSeisDat/-/issues/119Add unit test coverage2020-05-28T23:20:24ZPádraig Ó ConbhuíAdd unit test coverageAdd coverage for unit test files and code coverage for individual unit test execution.Add coverage for unit test files and code coverage for individual unit test execution.ExSeisDat Cleanuphttps://git.ichec.ie/oilgas/ExSeisDat/-/issues/116Identify repeated patterns in the project that should be abstracted into util...2019-07-16T15:47:29ZPádraig Ó ConbhuíIdentify repeated patterns in the project that should be abstracted into utilities.e.g. block decomposing data for use with collective routines.e.g. block decomposing data for use with collective routines.https://git.ichec.ie/oilgas/ExSeisDat/-/issues/112Generated .gitlab-ci.yml files for matrix builds.2019-05-01T14:34:55ZPádraig Ó ConbhuíGenerated .gitlab-ci.yml files for matrix builds.To cleanly test combinations of compilers, flags, and libraries, a "matrix build" would typically be used.
Gitlab CI doesn't support matrix builds, instead requiring each case be written explicitly.
A script should be written to define...To cleanly test combinations of compilers, flags, and libraries, a "matrix build" would typically be used.
Gitlab CI doesn't support matrix builds, instead requiring each case be written explicitly.
A script should be written to define all the combinations needed and generate the appropriate .gitlab-ci.yml file. A pipeline test should also be added to test the current .gitlab-ci.yml file is equivalent to the file generated by the generator, to ensure it's up-to-date.https://git.ichec.ie/oilgas/ExSeisDat/-/issues/111Seis Format2019-03-04T21:21:44ZPádraig Ó ConbhuíSeis FormatAdd support for reading / writing in Shearwater's `Seis` format.Add support for reading / writing in Shearwater's `Seis` format.