ExSeisDat issueshttps://git.ichec.ie/oilgas/ExSeisDat/-/issues2018-11-06T12:04:44Zhttps://git.ichec.ie/oilgas/ExSeisDat/-/issues/47Improve Unit Tests for MPI2018-11-06T12:04:44ZChristopher WernerImprove Unit Tests for MPIS-01510
Add / edit spec tests to explicitly test cases where the data passed in is different for different processes.
**PreDev Notes:**
Currently, the spectests mostly call MPI-based functions with the same value on every process.
**...S-01510
Add / edit spec tests to explicitly test cases where the data passed in is different for different processes.
**PreDev Notes:**
Currently, the spectests mostly call MPI-based functions with the same value on every process.
**Implementation Notes:**
{- ADDME -}
**System Test Changes:**
{- ADDME -}
**Bug Fixes:**
{- ADDME -}
**C++ API Changes:**
{- ADDME -}
**C API Changes:**
{- ADDME -}
**Success Criteria:**
{- ADDME -}
CREATED ON - 16/09/2016https://git.ichec.ie/oilgas/ExSeisDat/-/issues/46Survey/Line Coordinate transformations2018-11-06T12:04:29ZChristopher WernerSurvey/Line Coordinate transformationsS-01644
As a Tullow Geophysicist I want to have an API to do transformations from Inline,Crossline coordinates to survey coordinates (source/receiver coordinates) and vice versa so that I don't have to work out the transformation manual...S-01644
As a Tullow Geophysicist I want to have an API to do transformations from Inline,Crossline coordinates to survey coordinates (source/receiver coordinates) and vice versa so that I don't have to work out the transformation manually from the file or previous knowledge
**Pre-Dev Notes:**
Surveys are conducted in real world coordinates (x/y, usually metres) and other files such as velocity models are in line coordinates. To simplify usage I want an API that does the transformation so I don't have to.
In a gridded input file the il/xl & x,y coordinates of the cdp of each trace is specified. These can be used to compute the transformation variables needed.
*From Reveal Documentation*
For 3D data, the transformation between grid and world coordinates is:
![6697](/uploads/0ec4f171e1688fdcdfe29a4ee8b4abe2/6697.png)
where;
![6699](/uploads/6883b4a9ffea1f8ec6b1a6419d8a7d03/6699.png)
Angle = angle of an inline in degrees counter-clockwise from the X axis)
h = handedness of the coordinate system (1 if left-handed, -1 otherwise)
(XLINE_0, ILINE_0) = (X origin, Y origin)
For a 2D project, the transformation is:
![6702](/uploads/2d422c5322488e39c828bc1e3fc2bafb/6702.jpeg)
where
![6703](/uploads/19a994ad5a8ba8cff0c925cbcc7f0cbe/6703.png)
Angle = angle of an inline in degrees counter-clockwise from the X axis
CMP_0 = (X origin, Y origin)
**Implementation Notes:**
{- ADDME -}
**System Test Changes:**
{- ADDME -}
**Bug Fixes:**
{- ADDME -}
**C++ API Changes:**
{- ADDME -}
**C API Changes:**
{- ADDME -}
**Success Criteria:**
{- ADDME -}
CREATED ON - 17/05/2017https://git.ichec.ie/oilgas/ExSeisDat/-/issues/454D binning for multi-file surveys2018-11-06T12:04:15ZChristopher Werner4D binning for multi-file surveysS-01646
As a Tullow Geophysicist I want use the 4d bin on a site that has multi-file surveys because it is common for a survey to have more than 1 SEG-Y file
**Pre-Dev Notes:**
Currently the 4D binning is a utility that compares the c...S-01646
As a Tullow Geophysicist I want use the 4d bin on a site that has multi-file surveys because it is common for a survey to have more than 1 SEG-Y file
**Pre-Dev Notes:**
Currently the 4D binning is a utility that compares the coordinates in two SEG-Y files. It produces an output file for each input file that contains the traces that are contained in the bin. To encorporate this into ExSeisFlow, there should be a mechanism to compare two different sets (ie a layer higher than the set layer)
Previously: Integrate the 4d bin into the set layer. This should provide the functionality. Fourdbin would need to be implemented as an operation between sets.
**Implementation Notes:**
{- ADDME -}
**System Test Changes:**
{- ADDME -}
**Bug Fixes:**
{- ADDME -}
**C++ API Changes:**
{- ADDME -}
**C API Changes:**
{- ADDME -}
**Success Criteria**
* [ ] 4D binning is integrated into ExSeisFlow
CREATED ON - 31/05/2017https://git.ichec.ie/oilgas/ExSeisDat/-/issues/44Save a flow with ExSeisFlow to a .hist compatible OpenCPS file2018-11-06T12:04:01ZChristopher WernerSave a flow with ExSeisFlow to a .hist compatible OpenCPS fileS-01662
As a Tullow Geophysicist I would like that a workflow with ExSeisFlow to be output with a .hist compatible OpenCPS file so that the history of processing is recorded for accountability and QC.
**Pre-Dev Notes:**
What does Tu...S-01662
As a Tullow Geophysicist I would like that a workflow with ExSeisFlow to be output with a .hist compatible OpenCPS file so that the history of processing is recorded for accountability and QC.
**Pre-Dev Notes:**
What does Tullow want to do with this file?
Does it need interface with Reveal or simply a file log of operations applied to traces?
**Implementation Notes:**
{- ADDME -}
**System Test Changes:**
{- ADDME -}
**Bug Fixes:**
{- ADDME -}
**C++ API Changes:**
{- ADDME -}
**C API Changes:**
{- ADDME -}
**Success Criteria:**
{- ADDME -}
CREATED ON 13/07/2017https://git.ichec.ie/oilgas/ExSeisDat/-/issues/43Progress indicator inside PIOL2018-11-06T12:03:42ZChristopher WernerProgress indicator inside PIOLS-01672
As a Tullow Geophysicist I want percentages to be logged (class log) and immediately processed when especially heavy I/O functions etc are encountered.
i.e operations that will plausible take several hours in particular workload...S-01672
As a Tullow Geophysicist I want percentages to be logged (class log) and immediately processed when especially heavy I/O functions etc are encountered.
i.e operations that will plausible take several hours in particular workloads.
**PreDev Notes:**
{- ADDME -}
**Implementation Notes:**
{- ADDME -}
**System Test Changes:**
{- ADDME -}
**Bug Fixes:**
{- ADDME -}
**C++ API Changes:**
{- ADDME -}
**C API Changes:**
{- ADDME -}
**Success Criteria:**
{- ADDME -}
CREARED ON 30/08/2017https://git.ichec.ie/oilgas/ExSeisDat/-/issues/42Adding custom operations in a a workflow2018-11-06T12:02:29ZChristopher WernerAdding custom operations in a a workflowS-01702
As a Tullow Geophysicist I want to be able create and pass custom operations to Flow.
**PreDev Notes:**
Example:
Want to modify offsets so they are binned, thus apply the following operation to all headers:
int offbin = 50
...S-01702
As a Tullow Geophysicist I want to be able create and pass custom operations to Flow.
**PreDev Notes:**
Example:
Want to modify offsets so they are binned, thus apply the following operation to all headers:
int offbin = 50
offset = floor( getPrm(i, Meta::offset, prm) / offbin ) * offbin
**Implementation Notes:**
{- ADDME -}
**System Test Changes:**
{- ADDME -}
**Bug Fixes:**
{- ADDME -}
**C++ API Changes:**
{- ADDME -}
**C API Changes:**
{- ADDME -}
**Success Criteria:**
* [ ] Can create arbitrary Flow::Set operations.
CREATED ON 19/12/2017https://git.ichec.ie/oilgas/ExSeisDat/-/issues/41Enable external GUI for selection, plotting2018-11-06T12:02:07ZChristopher WernerEnable external GUI for selection, plottingS-01438
As a Tullow Geophysicist, I want a GUI for graphically selecting traces to include in a gather or to visualise. I want this because ??.
**Pre Dev Notes:**
This is a huge task to get nice.
How well does Reveal deal with really...S-01438
As a Tullow Geophysicist, I want a GUI for graphically selecting traces to include in a gather or to visualise. I want this because ??.
**Pre Dev Notes:**
This is a huge task to get nice.
How well does Reveal deal with really large files? If well use python interface to use reveal gui
Do Tullow have a specific graphical program in mind that they want us to interface with? If we had a specific one we could look into it
*Hook this into flow*
**Implementation Notes:**
{- ADDME -}
**System Test Changes:**
{- ADDME -}
**Bug Fixes:**
{- ADDME -}
**C++ API Changes:**
{- ADDME -}
**C API Changes:**
{- ADDME -}
**Success Criteria:**
{- ADDME -}
CREATED ON - 20/04/2016https://git.ichec.ie/oilgas/ExSeisDat/-/issues/40Wiggle plot2018-11-06T12:01:47ZChristopher WernerWiggle plotS-01440
As a Tullow Geophysicists I want an API to calculate Wiggle (inline or crossline vs time, see below) plots in a parallel aware way and output them to memory in a standard image format. I want this because it helps me do quality ...S-01440
As a Tullow Geophysicists I want an API to calculate Wiggle (inline or crossline vs time, see below) plots in a parallel aware way and output them to memory in a standard image format. I want this because it helps me do quality control on what my code is doing.
**PreDev Notes:**
![6688](/uploads/63a96ae2b1a78a0a77bb431e2a006e7b/6688.png)
This may be tricky to do in parallel. Convert output into intermediary binary format which can then be visualised with some set of python/octave calls.
**Implementation Notes:**
{- ADDME -}
**System Test Changes:**
{- ADDME -}
**Bug Fixes:**
{- ADDME -}
**C++ API Changes:**
{- ADDME -}
**C API Changes:**
{- ADDME -}
**Success Criteria:**
{- ADDME -}
CREATED ON - 20/04/2016https://git.ichec.ie/oilgas/ExSeisDat/-/issues/39I/O distribution/reordering for more Contiguous Reads and Writes2018-11-06T12:01:33ZChristopher WernerI/O distribution/reordering for more Contiguous Reads and WritesS-01548
When sorting a file, for example, processes will request seemingly random list of traces during I/O. However, the whole application is requesting a contiguous chunk of the data file. By sorting and distributing the requests acro...S-01548
When sorting a file, for example, processes will request seemingly random list of traces during I/O. However, the whole application is requesting a contiguous chunk of the data file. By sorting and distributing the requests across the processes, it may be possible to recognize these as contiguous I/O calls, perform a contiguous I/O call, and then perform the more random communication between the processes (using e.g. MPI_Scatterv) instead of to/from the disk.
MPI / Lustre *should* be doing this under the hood, but there's really no guarantee that it is.
**Old description:**
Perform a contigious read and a contigous write of sorted traces using communication to transfer the traces and parameters.
CREATED ON - 18/11/2016https://git.ichec.ie/oilgas/ExSeisDat/-/issues/38Coordinate plot2018-11-06T12:01:19ZChristopher WernerCoordinate plotS-01439
As a Tullow Geophysicists I want an API call to perform a coordinate plot (plot of receivers and sources) in a parallel aware way and output them to memory in a standard image format. I want this because it helps me do quality c...S-01439
As a Tullow Geophysicists I want an API call to perform a coordinate plot (plot of receivers and sources) in a parallel aware way and output them to memory in a standard image format. I want this because it helps me do quality control on what my code is doing.
**PreDev Notes:**
This may be tricky to do in parallel. Convert output into intermediary binary format which can then be visualised with some set of python/octave calls.
**Implementation Notes:**
{- ADDME -}
**System Test Changes:**
{- ADDME -}
**Bug Fixes:**
{- ADDME -}
**C++ API Changes:**
{- ADDME -}
**C API Changes:**
{- ADDME -}
**Success Criteria:**
{- ADDME -}
CREATED ON - 20/04/2016https://git.ichec.ie/oilgas/ExSeisDat/-/issues/37Matlab API2018-11-06T12:00:59ZChristopher WernerMatlab APIS-01684
As a Tullow Geophysicist, I want to be able to call PIOL features from Matlab because I don't know or necessarily have experience with the language the PIOL is written in.
**Pre-Dev Notes:**
* Library function should be able c...S-01684
As a Tullow Geophysicist, I want to be able to call PIOL features from Matlab because I don't know or necessarily have experience with the language the PIOL is written in.
**Pre-Dev Notes:**
* Library function should be able called nativity
* Library should run in serial and parallel - need to investigate MATLAB's parallel programming library
**Implementation Notes:**
{- ADDME -}
**System Test Changes:**
{- ADDME -}
**Bug Fixes:**
{- ADDME -}
**C++ API Changes:**
{- ADDME -}
**C API Changes:**
{- ADDME -}
**Success Criteria:**
{- ADDME -}
CREATED ON - 02/11/2017https://git.ichec.ie/oilgas/ExSeisDat/-/issues/36Investigate IME client commands2018-11-06T12:00:21ZChristopher WernerInvestigate IME client commandsS-01528
No material added.
**PreDev Notes:**
{- ADDME -}
**Implementation Notes:**
{- ADDME -}
**System Test Changes:**
{- ADDME -}
**Bug Fixes:**
{- ADDME -}
**C++ API Changes:**
{- ADDME -}
**C API Changes:**
{- ADDME -}
...S-01528
No material added.
**PreDev Notes:**
{- ADDME -}
**Implementation Notes:**
{- ADDME -}
**System Test Changes:**
{- ADDME -}
**Bug Fixes:**
{- ADDME -}
**C++ API Changes:**
{- ADDME -}
**C API Changes:**
{- ADDME -}
**Success Criteria:**
{- ADDME -}
CREATED ON - 21/10/2016https://git.ichec.ie/oilgas/ExSeisDat/-/issues/35Trace Writing Re-Ordering2018-11-06T11:59:39ZChristopher WernerTrace Writing Re-OrderingS-01456
As a Tullow Geophysicist I want to be able to reorganise my traces so that each SEG-Y "trace" contains a time slice. I want this because sometimes I do many time slices and this should improve performance.
For a set of traces, ...S-01456
As a Tullow Geophysicist I want to be able to reorganise my traces so that each SEG-Y "trace" contains a time slice. I want this because sometimes I do many time slices and this should improve performance.
For a set of traces, say, where the index in the trace data represents depth, and the traces are sorted by in-line coordinate: these would be written to disk in order of depth, then in-line coordinate. This story would allow traces to be written in order of in-line coordinate, and then depth.
In other words, if the set of traces represents a "square" of data, and the order this square is written to disk as well defined, we would like to be able to rotate this square before writing.
**PreDev Notes:**
{- ADDME -}
**Implementation Notes:**
{- ADDME -}
**System Test Changes:**
{- ADDME -}
**Bug Fixes:**
{- ADDME -}
**C++ API Changes:**
{- ADDME -}
**C API Changes:**
{- ADDME -}
**Success Criteria:**
{- ADDME -}
CREATED ON - 09/05/2016https://git.ichec.ie/oilgas/ExSeisDat/-/issues/34Use Trace Sorting Code2018-11-06T11:59:07ZChristopher WernerUse Trace Sorting CodeS-01461
Use the trace sorting code (bytes 3229–3230) the binary header to determine the order of traces prior to a read. Use this to decrease overhead in sorting/ordering algorithms.
**Pre-Dev Notes:**
Trace Sort Codes from the SEG-Y...S-01461
Use the trace sorting code (bytes 3229–3230) the binary header to determine the order of traces prior to a read. Use this to decrease overhead in sorting/ordering algorithms.
**Pre-Dev Notes:**
Trace Sort Codes from the SEG-Y standard:
* 0 = Unknown
* 1 = As recorded (no sorting)
* 2 = CDP ensemble
* 3 = Single fold continuous profile
* 4 = Horizontally stacked
* 5 = Common source point
* 6 = Common receiver point
* 7 = Common offset point
* 8 = Common mid-point
* 9 = Common conversion point
This should be implemented as part of the sorting algorithm, checking the current order before sorting and updating the sort code after sorting.
This needs a nice way of adding arbitrary binary header metadata. Currently, the binary header metadata supported is fixed.
**Implementation Notes:**
{- ADDME -}
**System Test Changes:**
{- ADDME -}
**Bug Fixes:**
{- ADDME -}
**C++ API Changes:**
{- ADDME -}
**C API Changes:**
{- ADDME -}
**Success Criteria:**
{- ADDME -}
CREATED ON - 10/05/2016https://git.ichec.ie/oilgas/ExSeisDat/-/issues/33Verify Trace Sorting for a Velocity Model File2018-11-06T11:58:50ZChristopher WernerVerify Trace Sorting for a Velocity Model FileS-01472
As a Tullow Geophysicist I want to be able to check if a velocity model meets the assumptions I make about trace ordering in my codes. I want this because I sometimes suspect my traces are not in the order I thought they were an...S-01472
As a Tullow Geophysicist I want to be able to check if a velocity model meets the assumptions I make about trace ordering in my codes. I want this because I sometimes suspect my traces are not in the order I thought they were and I can't manually check hundreds of thousands of traces.
This should be implemented as a utility program. Effectively, just checking the file is sorted according to the expected sorting criteria for a velocity model.
**PreDev Notes:**
{- ADDME -}
**Implementation Notes:**
{- ADDME -}
**System Test Changes:**
{- ADDME -}
**Bug Fixes:**
{- ADDME -}
**C++ API Changes:**
{- ADDME -}
**C API Changes:**
{- ADDME -}
**Success Criteria:**
{- ADDME -}
CREATED ON - 16/06/2016https://git.ichec.ie/oilgas/ExSeisDat/-/issues/32Cross Gather Automatic Gain Control2018-11-06T11:58:28ZChristopher WernerCross Gather Automatic Gain ControlS-01643
As a Tullow geophysicist, I wish to be able to perform automatic gain control, where the same gain correction is applied to each trace in the gather. I want this for visualisation purposes to correct for possible scaling shadow....S-01643
As a Tullow geophysicist, I wish to be able to perform automatic gain control, where the same gain correction is applied to each trace in the gather. I want this for visualisation purposes to correct for possible scaling shadow.
**PreDev Notes:**
NB: OpenCPS allows for gather agc for RMS and Mean Absolute Value (does not allow for RMS with Triangle Window and Median).
The current AGC implementation uses a 1-d window over an individual trace. This would be a square window over a set of traces and is described on page 121 of the OpenCPS manual.
**Implementation Notes:**
{- ADDME -}
**System Test Changes:**
{- ADDME -}
**Bug Fixes:**
{- ADDME -}
**C++ API Changes:**
{- ADDME -}
**C API Changes:**
{- ADDME -}
**Success Criteria:**
{- ADDME -}
CREATED ON - 17/05/2017https://git.ichec.ie/oilgas/ExSeisDat/-/issues/31Describe temporary rules which have been placed on the trace header in the te...2018-11-06T11:58:07ZChristopher WernerDescribe temporary rules which have been placed on the trace header in the text headerS-01531
As a Tullow Geophysicist, when I set a temporary new rule to store data in a location other than specified by the SEG-Y standard, I want the PIOL to store a description of the rule in the SEG-Y text header. I want this because s...S-01531
As a Tullow Geophysicist, when I set a temporary new rule to store data in a location other than specified by the SEG-Y standard, I want the PIOL to store a description of the rule in the SEG-Y text header. I want this because sometimes I send my SEG-Y files to other people, and they need to know what I stored and where.
**PreDev Notes:**
{- ADDME -}
**Implementation Notes:**
{- ADDME -}
**System Test Changes:**
{- ADDME -}
**Bug Fixes:**
{- ADDME -}
**C++ API Changes:**
{- ADDME -}
**C API Changes:**
{- ADDME -}
**Success Criteria:**
{- ADDME -}
CREATED ON - 27/10/2016https://git.ichec.ie/oilgas/ExSeisDat/-/issues/30Out of core 4d bin2018-11-06T11:57:48ZChristopher WernerOut of core 4d binS-01532
As a Tullow geophysicist I want to run 4d binning on very large datasets but there are limits on how much memory is available, and how many nodes I can use. Modify the 4d bin algorithm to support out-of-core calculations.
**Pre...S-01532
As a Tullow geophysicist I want to run 4d binning on very large datasets but there are limits on how much memory is available, and how many nodes I can use. Modify the 4d bin algorithm to support out-of-core calculations.
**PreDev Notes:**
Currently, the 4d bin expects the whole SEGY file can be distributed across the nodes and read into RAM at once.
Investigate a 4d bin operation that doesn't rely on the whole file being read into RAM.
**Implementation Notes:**
{- ADDME -}
**System Test Changes:**
{- ADDME -}
**Bug Fixes:**
{- ADDME -}
**C++ API Changes:**
{- ADDME -}
**C API Changes:**
{- ADDME -}
**Success Criteria:**
{- ADDME -}
Created on 27/10/2016https://git.ichec.ie/oilgas/ExSeisDat/-/issues/29Out of core sort2018-11-06T11:57:31ZChristopher WernerOut of core sortS-01544
Out-of-core, or external memory algorithms, refer to algorithms that are designed to work with data sets too large to fit in RAM.
**PreDev Notes**
Currently, the sort operation expects the whole SEGY file can be read at once, ...S-01544
Out-of-core, or external memory algorithms, refer to algorithms that are designed to work with data sets too large to fit in RAM.
**PreDev Notes**
Currently, the sort operation expects the whole SEGY file can be read at once, distributed across whatever nodes are being used. For particularly large files, this won't be possible.
Investigate sorting for large files that don't fit the total memory available.
e.g. If, say, half the file can be read into the total available memory, could both halves be sorted separately, and written to disk, then a merge-sort be used to combine them?
**Implementation Notes:**
{- ADDME -}
**System Test Changes:**
{- ADDME -}
**Bug Fixes:**
{- ADDME -}
**C++ API Changes:**
{- ADDME -}
**C API Changes:**
{- ADDME -}
**Success Criteria:**
{- ADDME -}
CREATED ON - 10/11/2016https://git.ichec.ie/oilgas/ExSeisDat/-/issues/28Add parser for 6-byte SEG-Y numbers2018-11-06T11:57:11ZChristopher WernerAdd parser for 6-byte SEG-Y numbersS-01546
Support locations 225-230 and 205-210 in the SEG-Y trace header.
**Pre-Dev Notes**
From the SEG-Y standard: The constant is encoded as a four-byte, two's complement integer (bytes 205–208 or bytes 225-228) which is the mantiss...S-01546
Support locations 225-230 and 205-210 in the SEG-Y trace header.
**Pre-Dev Notes**
From the SEG-Y standard: The constant is encoded as a four-byte, two's complement integer (bytes 205–208 or bytes 225-228) which is the mantissa and a two-byte, two's complement integer (bytes 209–210 or bytes 229-230) which is the power of ten exponent (i.e. Bytes 205–208 * 10**Bytes 209–210).
This will require creating a new rule for a 6 byte header.
This can be handled when the RuleEntry class is changed to specifying byte positions and parsing values.
A parser for converting this 6-byte number to a native number will also be needed.
Once that's in place, maybe add a default metadata key for the given trace metadatas?
**Implementation Notes:**
{- ADDME -}
**System Test Changes:**
{- ADDME -}
**Bug Fixes:**
{- ADDME -}
**C++ API Changes:**
{- ADDME -}
**C API Changes:**
{- ADDME -}
**Success Criteria:**
{- ADDME -}
CREATED ON - 18/11/2016