Each DataBuffer instance represents some in memory data. More...
#include <databuffer.h>
Public Types | |
| enum | flags_t { C_ORDERING =1, CONTIGUOUS =2, POSITIVE_STRIDE =4, DEGENERATE =8 } |
Public Member Functions | |
| DataBuffer (const DataBuffer &)=delete | |
| DataBuffer & | operator= (const DataBuffer &)=delete |
| virtual std::string | toString () const =0 |
| virtual std::shared_ptr< void > | voidData ()=0 |
| virtual std::shared_ptr< const void > | voidData () const =0 |
| virtual bool | contiguous () const =0 |
| virtual std::int64_t | allocsize () const =0 |
| virtual std::int64_t | totalsize () const =0 |
| virtual std::int64_t | itemsize () const =0 |
| virtual const std::int64_t * | sizeptr () const =0 |
| virtual const std::int64_t * | strideptr () const =0 |
| virtual std::array< std::int64_t, 3 > | size3d () const =0 |
| virtual std::array< std::int64_t, 3 > | stride3d () const =0 |
| virtual bool | ownsdata () const =0 |
| virtual bool | isScalar () const =0 |
| virtual bool | isAllSame (const std::int64_t *used_in) const =0 |
| virtual double | scalarAsDouble () const =0 |
| virtual RawDataType | datatype () const =0 |
| virtual void | fill (double value)=0 |
| virtual void | clear ()=0 |
| virtual std::pair< double, double > | range () const =0 |
| virtual void | copyFrom (const DataBuffer *src, const std::int64_t *srcorig, const std::int64_t *dstorig, const std::int64_t *cpyorig, const std::int64_t *cpysize)=0 |
| virtual std::shared_ptr< DataBuffer > | clone () const =0 |
| virtual std::shared_ptr< DataBuffer > | scaleToFloat (const std::array< double, 2 > &)=0 |
| virtual std::shared_ptr< DataBuffer > | scaleToStorage (const std::array< double, 2 > &, RawDataType)=0 |
| virtual std::uint32_t | layout () const =0 |
| virtual bool | is_cstride () const =0 |
| virtual void | check_cstride () const =0 |
| virtual std::shared_ptr< DataBuffer > | slice1 (int dim, std::int64_t start, std::int64_t size) const =0 |
Static Public Member Functions | |
| static std::shared_ptr< DataBuffer > | makeDataBuffer3d (void *raw, std::int64_t nbytes, const std::array< std::int64_t, 3 > &size, RawDataType dtype) |
Detailed Description
Each DataBuffer instance represents some in memory data.
Warning, creeping features. The class starter out as a trivial pointer + 3d size struct but is growing dangerously close to emulating a Python numpy.ndarray.
This data type represents an unsafe buffer pointer, plus:
- (template parameter) value type
- (template parameter) number of dimensions
- Size in each dimension and implied total buffer size.
- Stride in each dimension to convert 1d/Nd.
- Option to flag the entire buffer as constant-value and store just the constant. The buffer pointer is then null. Note, I could also allocate a single T and set buffer to point to that. But this is just asking for buffer overrun bugs.
There is no knowledge of where in the survey this data belongs.
There is no support for views covering just part of a buffer, and strides must all be positive. It is possible to support those scenarios but let's not put in bells and whistles before we see they are needed.
DataBuffer is a non-templated base class to allow passing buffers around without needing to have the calling code templated as well.
copyFrom() is somewhat unsafe. It can verify (using dynamic_cast) that "src" is of the same type as "this". But srcorig etc. are passed as dumb pointers so the code cannot know whether they are of the correct size. Maybe I am being too general here; maybe the number of dimensions should always be 3?
The actual bulk data is stored as a std::shared_ptr<T> which means it can be passed around independantly of the DataBuffer instance. The voidData() method returns the actual smart pointer as a std::shared_ptr<void> which can be downcast to the correct type. the data() method in the templated leaf type returns the raw pointer. It is possible to also have methods returning void* and std::shared_ptr<T> but don't add those unless actually used.
Constructors that expect an external buffer are required to provide it as a smart pointer. The DataBuffer will share ownership with the caller. There is nothing that prevents the callers from passing a shared_ptr with a no-op deleter, effectively removing reference counting. But that will of course void their warranty. And make the callers 100% responsible for the data being valid long enough.
Unsafe external buffers are currently used for read() and write() requests from the application code and for delivery from the file back end. Reference counted external buffers are used when the buffer was built from decompressed data.
Each of those scenarios should be ok since no buffers should be held onto after the functions return. TODO-Worry: Could async read requests get delivered after the call to read() got aborted via an exception? TODO-Worry: Might somebody decide to hold on to a buffer in order to implement delayed write? In 99% of cases the application's data buffer needs to be copied into a properly reference counted buffer anyway. Maybe make that 100% by not allowing short cuts when the application writes exactly one brick. Another problem is if the writer decides to copy out brick at a time into a reusable one brick buffer. This could also lead to the user's buffer being held on to longer. Maybe make sure all bricks are copied up front before any writing starts. This might also simplify parallelized compression.
As an alternative to the above, I have considered a scheme that makes less use of smart pointers so I can reduce the overhead they cause. Beware of premature optimization though. And more things that could go wrong with buffers freed early. The changes would be:
- Inside DataBuffer there will be an unsafe "T *_data" pointer instead of a smart pointer.
- If the DataBuffer owns its storage it will need code in the destructor to free it. No reference counting though.
- Callers that want to use an external buffer has no choice but to pass an unsafe pointer. Which means the caller is always responsible for keeping the data alive long enough. Note that I suspect this will happen anyway (see "voiding the warranty", above). So this might not be a major point.
- It will still be possible to obtain a smart pointer to the raw data if the DataBuffer instance itself is reference counted. Use the aliasing constructor of std::shared_ptr. Use enable_shared_from_this, or (preferably) change the signature of voidData() to provide the instance to alias. See ZgyInternalBulk::_writeOneNormalBrick where the code needs a smart pointer in preparation to queue up a write.
- Beware that the DataBuffer::_data pointer must not be deleted (e.g. due to a realloc) before the instance itself goes out of scope. That rule applies in any case, but without reference counting we are talking about accessing garbage data instead of just stale data.
- Copy/assign would need a deep copy instead of aliasing the data buffer. However, shallow copying (e.g. to create a slice view) is not supported yet. So this is a future worry.
Other changes I have considered, orthogonal to the above:
- Move all data members to the parent. _data will be declared as std::shared_ptr<void>, and data() will need to do a cast. Which is safe with respect to alignmemt as long as the cast is to the concrete type that the buffer initially had. Several of the methods in the DataBuffer base class can now be made non virtual if desired.
- Have the destructor and/or clear() check the reference count of _data and do a deep copy if there are external references and if _data uses fake refcounts with an empty destructor. Unfortunately the latter is difficult to check for. Also, this fix only helps for asynchronous writes, not reads.
- TODO-Low: Remove the template on number of dimensions. This would make the API simpler. See e.g. sizeptr() vs. size3d(). Current usage in OpenZGY is always NDim=3. If the Alpha tile feature is resurrected then there will be a need for 2d arrays, but can probably be handled as 2d arrays with nk=1. Not quite as elegant as the current template but might still be a good idea overall.
TODO-Low decide on support for strided data.
It is easy enough for this class to allow arbitrary strides. Even strides that don't make sense. The problem is that code accessing the data would like to make assumptions about the stride so the code becomes simpler and also simpler to test. The current situation is unclear. Some code is flexible but it might not be possible to use that flexibility due to other code making assumptions.
- Fixed order InlineCrosslineSlice with no padding. Stride is simply (nj*nk, nk, 1).
- Fixed order InlineCrosslineSlice with padding. As above, but the nj and nk numbers might be larger to account for unused space at the end in the j and k direction. Unused space in the slowest varying direction will not be detected. And is not usually relevant to the consumer.
- As above but allow applying an initial offset. This might be done by adjusting the data pointer, so there is no extra calculation to be done. Caller may still assume the data starts at (0,0,0).
- As above but any of the 6 possible orderings may be chosen.
- No restrictions. Strides may be negative, or they might not make any sense.
The more support there is for handling strides, the more opportunities exist to use views on a buffer instead of copying them.
Member Function Documentation
◆ clone()
|
pure virtual |
Deep copy of a DataBuffer
Implemented in InternalZGY::DataBufferNd< T, NDim >.
◆ copyFrom()
|
pure virtual |
Corresponds to openzgy.impl._partialCopy(). To make templates work better the C++ version is an instance method with 'this' as destination.
Implemented in InternalZGY::DataBufferNd< T, NDim >.
◆ makeDataBuffer3d()
|
static |
Convert voidptr + nbytes + size3d to a DataBuffer.
To avoid too much copy/paste, this method this able to create several different buffer types by calling different DataBufferNd constructors.
If voidptr is null this creates an uninitialized 3d DataBuffer with the given size and type and isScalar() == false. nbytes should be 0.
If voidptr is not null then this creates a scalar DataBuffer if nbytes indicates there is room for just one sample. If there are enough bytes it creates a regular DataBuffer pointing to the voidptr that was passed in.
The valid values for nbytes are:
0 - If and only if voidptr is null. sizeof(T) - (T*)voidptr contains the scalar to use. sizeof(double) - (double*)voidptr contains the scalar to use, size[0]*size[1]*size[2]*sizeof(T) - (T*)voidptr is used as-is.
Any other value will raise an exception.
Technically the scheme is ambiguous if sizeof(T) is 8 bytes and size indicates less than 8 bytes. Cannot know for sure whether the buffer points to a double or to 8/4/2/1 element of T. This should never happen in practice. If worried, remove the "double" feature or introduce a new is_double flag.
The scheme also cannot distinguish between a scaler size 1x1x1 and a regular buffer size 1x1x1. But those two are practically equivalent.
Caveat: If the data pointer is not correctly aligned to the type of data then the returned DataBuffer will also have a misaligned pointer.
Caveat: Since the user can pass in a void* buffer, this will not be reference counted and could go out of scope beore the databuffer does. This might change. To mitigate this somehow, if the code knows the buffer is no longer valid it can call databuffer.clear().
◆ scaleToFloat()
|
pure virtual |
Convert DataBufferNd<T,N> to DataBufferNd<float,N>, hiding leaf types.
Implemented in InternalZGY::DataBufferNd< T, NDim >.
◆ scaleToStorage()
|
pure virtual |
Convert DataBufferNd<float,N> to DataBufferNd<T,N>, hiding leaf types.
Implemented in InternalZGY::DataBufferNd< T, NDim >.
The documentation for this class was generated from the following files:
- impl/databuffer.h
- impl/databuffer.cpp
