Memory Segment Connection and Disconnection
int rsm_memseg_import_connect(rsmapi_controller_handle_t controller, rsm_node_id_t node_id, rsm_memseg_id_t segment_id, rsm_permission_t perm, rsm_memseg_import_handle_t *im_memseg);
This function connects to segment segment_id on remote node node_id using the specified permission perm and returns a segment handle.
The argument perm specifies the access mode requested by the importer for this connection. To establish the connection, the access permissions specified by the exporter are compared to the access mode, user ID, and group ID used by the importer. If the request mode is not valid, the connection request is denied. The perm argument is limited to the following octal values:
| 0400 | Read mode |
| 0200 | Write mode |
| 0600 | Read/write mode |
The specified controller must have a physical connection to the controller used in the export of the segment.
Return Values: Returns 0 if successful; returns an error value otherwise.
| RSMERR_BAD_CTLR_HNDL | Invalid controller handle |
| RSMERR_CTLR_NOT_PRESENT | Controller not present |
| RSMERR_BAD_SEG_HNDL | Invalid segment handle |
| RSMERR_PERM_DENIED | Permission denied |
| RSMERR_SEG_NOT_PUBLISHED_TO_NODE | Segment not published to node |
| RSMERR_SEG_NOT_PUBLISHED | No such segment published |
| RSMERR_REMOTE_NODE_UNREACHABLE | Remote node not reachable |
| RSMERR_INTERRUPTED | Connection interrupted |
| RSMERR_INSUFFICIENT_MEM | Insufficient memory |
| RSMERR_INSUFFICIENT_RESOURCES | Insufficient resources |
| RSMERR_BAD_ADDR | Bad address |
int rsm_memseg_import_disconnect(rsm_memseg_import_handle_t im_memseg);
This function disconnects a segment and frees its resources. All existing mappings are removed and the handle im_memseg is freed.
Return Values: Returns 0 if successful; returns an error value otherwise.
| RSMERR_BAD_SEG_HNDL | Invalid segment handle |
| RSMERR_SEG_STILL_MAPPED | Segment still mapped |
| RSMERR_POLLFD_IN_USE | pollfd in use |
Memory Access Primitives
The following interfaces provide a mechanism for transferring small amounts of data (between 8 and 64 bits). The get interfaces use a repeat count (rep_cnt) to indicate the number of data items of a given size the process will read from successive locations beginning at byte offset offset in the imported segment write to successive locations beginning at datap. The put interfaces use a repeat count (rep_cnt) to indicate the number of data items of a given size the process will read from successive locations beginning at datap and write to the imported segment at successive locations beginning at the byte offset specified by the argument offset.
These interfaces also provide byte swapping in case the source and destination have incompatible endian characteristics.
Function Prototypes:
int rsm_memseg_import_get8(rsm_memseg_import_handle_t im_memseg, off_t offset, uint8_t *datap, ulong_t rep_cnt);
int rsm_memseg_import_get16(rsm_memseg_import_handle_t im_memseg, off_t offset, uint16_t *datap, ulong_t rep_cnt);
int rsm_memseg_import_get32(rsm_memseg_import_handle_t im_memseg, off_t offset, uint32_t *datap, ulong_t rep_cnt);
int rsm_memseg_import_get64(rsm_memseg_import_handle_t im_memseg, off_t offset, uint64_t *datap, ulong_t rep_cnt);
int rsm_memseg_import_put8(rsm_memseg_import_handle_t im_memseg, off_t offset, uint8_t *datap, ulong_t rep_cnt);
int rsm_memseg_import_put16(rsm_memseg_import_handle_t im_memseg, off_t offset, uint16_t *datap, ulong_t rep_cnt);
int rsm_memseg_import_put32(rsm_memseg_import_handle_t im_memseg, off_t offset, uint32_t *datap, ulong_t rep_cnt);
int rsm_memseg_import_put64(rsm_memseg_import_handle_t im_memseg, off_t offset, uint64_t *datap, ulong_t rep_cnt);
The following interfaces are intended for data transfers larger than the ones supported by the segment access operations.
int rsm_memseg_import_put(rsm_memseg_import_handle_t im_memseg, off_t offset, void *src_addr, size_t length);
This function copies data from local memory, specified by the src_addr and length, to the corresponding imported segment locations specified by the handle and offset.
int rsm_memseg_import_get(rsm_memseg_import_handle_t im_memseg, off_t offset, void *dst_addr, size_t length);
This function is similar to rsm_memseg_import_put(), but data flows from the imported segment into local regions defined by the dest_vec argument
All the put and get routines write or read the specified quantity of data from the byte offset location specified by the argument offset from the base of the segment. The offset must align at the appropriate boundary. For example, rsm_memseg_import_get64() requires that offset and datap align at a double-word boundary, while rsm_memseg_import_put32() requires an offset aligned at a word boundary.
By default, the barrier mode attribute of a segment is implicit, which means the caller assumes the data transfer has completed or failed upon return from the operation. Because the default barrier mode is implicit, the application must initialize the barrier by using the rsm_memseg_import_init_barrier() function before calling put or get routines when using the default mode. To use the explicit operation mode, the caller must use a barrier operation to force the completion of a transfer and determine if any errors have occurred as a result.
Note - If an import segment is partially mapped by passing an offset in the rsm_memseg_import_map() routine, the offset argument in the put or get routines is from the base of the segment, not the offset at which the segment has been partially mapped. The user must make sure that the correct byte offset is passed to put and get routines.
Return Values: Returns 0 if successful; returns an error value otherwise.
| RSMERR_BAD_SEG_HNDL | Invalid segment handle |
| RSMERR_BAD_ADDR | Bad address |
| RSMERR_BAD_MEM_ALIGNMENT | Invalid memory alignment |
| RSMERR_BAD_OFFSET | Invalid offset |
| RSMERR_BAD_LENGTH | Invalid length |
| RSMERR_PERM_DENIED | Permission denied |
| RSMERR_BARRIER_UNINITIALIZED | Barrier not initialized |
| RSMERR_BARRIER_FAILURE | I/O completion error |
| RSMERR_CONN_ABORTED | Connection aborted |
| RSMERR_INSUFFICIENT_RESOURCES | Insufficient resources |
Scatter-Gather Access
The rsm_memseg_import_putv() and rsm_memseg_import_getv() functions allow the use of a list of I/O requests instead of a single source and destination address.
Function Prototypes:
int rsm_memseg_import_putv(rsm_scat_gath_t *sg_io);
int rsm_memseg_import_getv(rsm_scat_gath_t *sg_io);
The I/O vector component of the scatter-gather list (sg_io) enables the specification of local virtual addresses or local_memory_handles. Handles are an efficient way to repeatedly use a local address range because allocated system resources (such as locked down local memory) are maintained until the handle is freed. The supporting functions for handles are rsm_create_localmemory_handle() and rsm_free_localmemory_handle().
You can gather virtual addresses or handles into the vector in order to write to a single remote segment, or scatter the results of reading from a single remote segment to the vector of virtual addresses or handles.
I/O for the entire vector is initiated before returning. The barrier mode attribute of the import segment determines whether the I/O has completed before the function returns. Setting the barrier mode attribute to implicit guarantees that data transfer is completed in the order entered in the vector. An implicit barrier open and close surrounds each list entry. If an error is detected, I/O for the vector is terminated and the function returns immediately. The residual count indicates the number of entries for which the I/O either did not complete or was not initiated.
To specify that a notification event must be sent to the target segment if the putv or getv operation completes successfully, specify the value RSM_IMPLICIT_SIGPOST in the flags entry in the rsm_scat_gath_t structure. The flags entry may also contain the value RSM_SIGPOST_NO_ACCUMULATE, which will be passed on to the signal post operation when RSM_IMPLICIT_SIGPOST is set.
Return Values: Returns 0 if successful; returns an error value otherwise.