onload extensions api

Posted 2022-02-05 raindayinrain

概述

扩展api&关联库

源码

随onload分发版本提供

通用组件

1. #include <onload/extensions.h>
2. libonload_ext.a, libonload_ext.so
此类api不依赖onload

onload_is_present

// If the application is linked with libonload_ext, 
// but not running with Onload this will return 0. 
// If the application is running with Onload this will return 1.
int onload_is_present (void)

onload_fd_stat

struct onload_stat

int32_t stack_id;
char* stack_name;
int32_t endpoint_id;
int32_t endpoint_state;
;

// Retrieves internal details about an accelerated socket.
//0 socket is not accelerated
//1 socket is accelerated
//‐ENOMEM when memory cannot be allocated
//内部会掉malloc分配stack_name指向内存，所以后续需要对stack_name执行free
extern int onload_fd_stat(int fd, struct onload_stat* stat);

onload_fd_check_feature

// Used to check whether the Onload file descriptor supports a feature or not.
//0 if the feature is supported but not on this fd
//>0 if the feature is supported both by onload and this fd
//<0 if the feature is not supported:
//‐ENOSYS if onload_fd_check_feature() is not supported.
//‐ ENOTSUPP if the feature is not supported by onload.
int onload_fd_check_feature (int fd, enum onload_fd_feature feature);
enum onload_fd_feature 
	/* Check whether this fd supports ONLOAD_MSG_WARM or not */
	ONLOAD_FD_FEAT_MSG_WARM,
	/* see Notes for details */
	ONLOAD_FD_FEAT_UDP_TX_TS_HDR
;

onload_thread_set_spin

//For a thread calling this function, 
//onload_thread_set_spin() sets the per‐thread spinning actions, 
//it is not per‐stack and not per‐socket.
//0 on success
//‐EINVAL if unsupported type is specified.
int onload_thread_set_spin(enum onload_spin_type type, unsigned spin)

参数解释

type
	Which operation to change the spin status of. 
	The type must be one of the following:
	enum onload_spin_type 
	
		ONLOAD_SPIN_ALL, /* enable or disable all spin options */
		ONLOAD_SPIN_UDP_RECV,
		ONLOAD_SPIN_UDP_SEND,
		ONLOAD_SPIN_TCP_RECV,
		ONLOAD_SPIN_TCP_SEND,
		ONLOAD_SPIN_TCP_ACCEPT,
		ONLOAD_SPIN_PIPE_RECV,
		ONLOAD_SPIN_PIPE_SEND,
		ONLOAD_SPIN_SELECT,
		ONLOAD_SPIN_POLL,
		ONLOAD_SPIN_PKT_WAIT,
		ONLOAD_SPIN_EPOLL_WAIT,
		ONLOAD_SPIN_STACK_LOCK,
		ONLOAD_SPIN_SOCK_LOCK,
		ONLOAD_SPIN_SO_BUSY_POLL,
		ONLOAD_SPIN_TCP_CONNECT,
		ONLOAD_SPIN_MIMIC_EF_POLL, 
		/* thread spin configuration which mimics spin settings in EF_POLL_USEC. 
		Note that this has no effect on the usec‐setting part of EF_POLL_USEC.
		This needs to be set separately*/
		ONLOAD_SPIN_MAX /* special value to mark largest valid input */
	;
spin
	A boolean which indicates whether the operation should spin or not.

额外说明

Notes
Spin time (for all threads) is set using the EF_SPIN_USEC parameter.
Examples
The onload_thread_set_spin API can be used to control spinning on a per‐thread or per‐API basis. 
The existing spin‐related configuration options set the default behavior for threads, 
and the onload_thread_set_spin API overrides the default for the thread calling this function.
例子1：Disable all sorts of spinning:
onload_thread_set_spin(ONLOAD_SPIN_ALL, 0);
例子2：Enable all sorts of spinning:
onload_thread_set_spin(ONLOAD_SPIN_ALL, 1);
例子3：Enable spinning only for certain threads:
1 Set the spin timeout by setting EF_SPIN_USEC, and disable spinning by default
by setting EF_POLL_USEC=0.
2 In each thread that should spin, invoke onload_thread_set_spin().
例子4：Disable spinning only in certain threads:
1 Enable spinning by setting EF_POLL_USEC=<timeout>.
2 In each thread that should not spin, invoke onload_thread_set_spin().
WARNING: If a thread is set to NOT spin and then blocks this may invoke an interrupt for the whole stack. Interrupts occurring on moderately busy threads may cause unintended and undesirable consequences.
例子5：Enable spinning for UDP traffic, but not TCP traffic:
1 Set the spin timeout by setting EF_SPIN_USEC, and disable spinning by default
by setting EF_POLL_USEC=0.
2In each thread that should spin (UDP only), do:
onload_thread_set_spin(ONLOAD_SPIN_UDP_RECV, 1)
onload_thread_set_spin(ONLOAD_SPIN_UDP_SEND, 1)
例子6：Enable spinning for TCP traffic, but not UDP traffic:
1 Set the spin timeout by setting EF_SPIN_USEC, and disable spinning by default
by setting EF_POLL_USEC=0.
2 In each thread that should spin (TCP only), do:
onload_thread_set_spin(ONLOAD_SPIN_TCP_RECV, 1)
onload_thread_set_spin(ONLOAD_SPIN_TCP_SEND, 1)
onload_thread_set_spin(ONLOAD_SPIN_TCP_ACCEPT, 1)

Spinning and sockets:
When a thread calls onload_thread_set_spin() it sets the spinning actions applied when the thread accesses any socket ‐ irrespective of whether the socket is created by this thread.
If a socket is created by thread‐A and is accessed by thread‐B, calling onload_thread_set_spin(ONLOAD_SPIN_ALL, 1) only from thread‐B will enable spinning for thread‐B, 
but not for thread‐A. 
In the same scenario, if onload_thread_set_spin(ONLOAD_SPIN_ALL, 1) is called only from thread‐A, 
then spinning is enabled only for thread‐A, but not for thread‐B.
The onload_thread_set_spin() function sets the per‐thread spinning action.

onload_thread_get_spin

//For the current thread, identify which operations should spin.
int onload_thread_get_spin(unsigned *state)

参数解释

state
	Location at which to write the spin status as a bitmask. 
	Bit n of the mask is set if spinning has been enabled for spin type n (see onload_thread_set_spin on page 285).

额外说明

0 on success
Notes
	Spin time (for all threads) is set using the EF_SPIN_USEC parameter.
Examples
	Determine if spinning is enabled for UDP receive:
	unsigned state; onload_thread_get_spin(&state);
	if (state & (1 << ONLOAD_SPIN_UDP_RECV)) 
	
		// spinning is enabled for UDP receive
	

onload_socket_nonaccel

// Create a socket which is not accelerated by Onload. 
// This function is useful 
// when attempting to reserve a port for an ephemeral ef_vi instance without installing Onload filters. 
// It is also possible to use the stackname API to disable acceleration for specific socket(s).
// This function takes arguments and returns values 
// that correspond exactly to the standard socket() function call.
// Return the file descriptor that refers to the created endpoint.
// ‐1 with errno ENOSYS if the Onload extensions library is not in use.
int onload_socket_nonaccel(int domain, int type, int protocol)

onload_socket_unicast_nonaccel

// Create a socket that will only accelerate multicast traffic. 
// If this socket is not able to receive multicast, for example, 
// because it is bound to a unicast local address, 
// or it is a TCP socket, 
// then it will be handed over to the kernel.
// This function is useful for cases 
// where a socket will be used solely for multicast traffic 
// to avoid consuming limited filter table resource. 
// This does not prevent unicast traffic from arriving at the socket, 
// and if appropriate traffic is received, 
// it will still be delivered via the un‐accelerated path. 
// It is most useful for sockets that are bound to INADDR_ANY, 
// because for these Onload must install a filter per IP address 
// that is configured on an accelerated interface, 
// on each accelerated hardware port.
// If a socket is bound to a multicast local address, 
// then no unicast filters will be installed, so there is no need for this function.
// Return the file descriptor that refers to the created endpoint.
// ‐1 with errno ENOSYS if the Onload extensions library is not in use.
int onload_socket_unicast_nonaccel(int domain, int type, int protocol)

解释

This function takes arguments and returns values that correspond 
exactly to the standard socket() function call.

Stacks API

Using the Onload Extensions API 
an application can bind selected sockets to specific Onload stacks 
and in this way ensure that time‐critical sockets are not starved of resources by other non‐critical sockets. 
The API allows an application to select sockets 
which are to be accelerated thus reserving Onload resources for performance critical paths. 
This also prevents non‐critical paths from creating jitter for critical paths.

onload_set_stackname

Select the Onload stack that new sockets are placed in. 
A socket can exist only in a single stack. 
A socket can be moved to a different stack ‐ see onload_move_fd() below.
Moving a socket to a different stack does not create a copy of the socket 
in originator and target stacks.
//0 on success
//‐1 with errno set to ENAMETOOLONG if the name exceeds permitted length
/‐1 with errno set to EINVAL if other parameters are invalid.
int onload_set_stackname(int who, int scope, const char *name)

参数解释

who
	Must be one of the following:
	‐ ONLOAD_THIS_THREAD ‐ to modify the stack name in which all subsequent sockets are created by this thread.
	‐ ONLOAD_ALL_THREADS ‐ to modify the stack name in which all subsequent sockets are created by all threads in the current process. ONLOAD_THIS_THREAD takes precedence over ONLOAD_ALL_THREADS.
scope
	Must be one of the following:
	‐ ONLOAD_SCOPE_THREAD ‐ name is scoped with current thread
	‐ ONLOAD_SCOPE_PROCESS ‐ name is scoped with current process
	‐ ONLOAD_SCOPE_USER ‐ name is scoped with current user
	‐ ONLOAD_SCOPE_GLOBAL ‐ name is global across all threads, users and processes.
	‐ ONLOAD_SCOPE_NOCHANGE ‐ undo effect of a previous call to onload_set_stackname(ONLOAD_THIS_THREAD, …), see Notes on page 290.
name
	One of the following:
	‐ the stack name up to 8 characters.
	‐ an empty string to set no stackname
	‐ the special value ONLOAD_DONT_ACCELERATE to prevent sockets created in this thread, user, process from being accelerated.
	Sockets identified by the options above will belong to the Onload stack until a subsequent call using onload_set_stackname identifies a different stack or the ONLOAD_SCOPE_NOCHANGE option is used.

解释说明

注意1：
This applies for stacks selected for sockets created by socket() and for pipe(), 
it has no effect on accept(). 
Passively opened sockets created via accept() will always be in the same stack 
as the listening socket that they are linked to, 
this means that the following are functionally identical i.e.
onload_set_stackname(foo)
socket
listen
onload_set_stackname(bar)
accept

and:
onload_set_stackname(foo)
socket
listen
accept
onload_set_stackname(bar)
In both cases the listening socket and the accepted socket will be in stack foo.
注意2：
Scope defines the namespace in which a stack belongs. 
A stackname of foo in scope user is not the same as a stackname of foo in scope thread. 
Scope restricts the visibility of a stack 
to either the current thread, current process, current user or is unrestricted (global). 
This has the property that with, for example, process based scoping, 
two processes can have the same stackname without sharing a stack 
‐ as the stack for each process has a different namespace.
注意3：
Scoping can be thought of as adding a suffix to the supplied name e.g.
ONLOAD_SCOPE_THREAD: <stackname>‐t<thread_id>
ONLOAD_SCOPE_PROCESS: <stackname>‐p<process_id>
ONLOAD_SCOPE_USER: <stackname>‐u<user_id>
ONLOAD_SCOPE_GLOBAL: <stackname>
This is an example only and the implementation is free to do something different such as maintaining different lists for different scopes.
注意4：
ONLOAD_SCOPE_NOCHANGE will undo the effect of a previous call to onload_set_stackname(ONLOAD_THIS_THREAD, …).
If you have previously used onload_set_stackname(ONLOAD_THIS_THREAD, …) and want to revert to the behavior of threads that are using the ONLOAD_ALL_THREADS configuration, 
without changing that configuration, 
you can do the following:
onload_set_stackname(ONLOAD_ALL_THREADS, ONLOAD_SCOPE_NOCHANGE, "");

Related environment variables
Related environment variables are:
EF_DONT_ACCELERATE
Default: 0
Minimum: 0
Maximum: 1
Scope: Per‐process
If this environment variable is set then acceleration for ALL sockets is disabled 
and handed off to the kernel stack until the application overrides this state with a call to onload_set_stackname().

EF_STACK_PER_THREAD
Default: 0
Minimum: 0
Maximum: 1
Scope: Per‐process
If this environment variable is set each socket created by the application will be placed in a stack depending on the thread in which it is created. Stacks could, for example, be named using the thread ID of the thread that creates the stack, but this should not be relied upon.
A call to onload_set_stackname overrides this variable. EF_DONT_ACCELERATE takes precedence over this variable.

EF_NAME
Default: none
Minimum: 0 chars
Maximum: 8 chars
Scope: per‐stack
The environment variable EF_NAME will be honored to control Onload stack sharing. However, a call to onload_set_stackname overrides this variable and, EF_DONT_ACCELERATE and EF_STACK_PER_THREAD both take precedence over EF_NAME.

onload_move_fd

// Move the file descriptor to the current stack. 
// The target stack can be specified with onload_set_stackname(),
// then use onload_move_fd() to put the socket into the target stack.
// A socket can exist only in a single stack. 
// Moving a socket to a different stack does not create a copy of the socket 
// in originator and target stacks. 
// Limited to TCP closed or accepted sockets only.
// fd ‐ the file descriptor to be moved to the current stack.
// 0 on success
// non‐zero otherwise.
int onload_move_fd (int fd)

注意点

Notes
•Useful to move fds obtained by accept() 
to a different Onload stack from the listening socket.
•Cannot be used on actively opened connections, 
although it is possible to use onload_set_stackname() 
before calling connect() to achieve the same result.
•The socket must have empty send and retransmit queues 
(i.e. send not called on this socket)
•The socket must have a simple receive queue 
(no loss, reordering, etc)
•The fd is not yet in an epoll set.
•The onload_move_fd function should not be used 
if SO_TIMESTAMPING is set to a non‐zero value for the originating socket.
•Should not be used simultaneously with other I/O multiplex actions 
i.e. poll(), select(), recv() etc on the file descriptor.
•This function is not async‐safe and should never be called 
from any process function handling signals.
•This function cannot be used to hand sockets over to the kernel. 
It is not possible to use onload_set_stackname (ONLOAD_DONT_ACCELERATE) and then onload_move_fd().

NOTE: The onload_move_fd function does not check whether a destination stack has either RX or TX timestamping enabled.

onload_stackname_save

Save the state of the current onload stack identified by the previous call to onload_set_stackname()
// 0 on success
// ‐ENOMEM when memory cannot be allocated.
int onload_stackname_save (void)

onload_stackname_restore

Restore stack state saved with a previous call to onload_stackname_save(). All updates/changes to state of the current stack will be deleted and all state previously saved will be restored. To avoid unexpected results, the stack should be restored in the same thread as used to call onload_stackname_save().
// 0 on success
// non‐zero if an error occurs.
int onload_stackname_restore (void)

注意点

The API stackname save and restore functions provide flexibility when binding sockets to an Onload stack.
Using a combination of onload_set_stackname(), onload_stackname_save() and onload_stackname_restore(), the user is able to create default stack settings which apply to one or more sockets, save this state and then create changed stack settings which are applied to other sockets. The original default settings can then be restored to apply to subsequent sockets.

Stacks API Usage

Using a combination of the EF_DONT_ACCELERATE environment variable and the function onload_set_stackname(), the user is able to control/select sockets which are to be accelerated and isolate these performance critical sockets and threads from the rest of the system.

onload_stack_opt_set_int

Set/modify per stack options that all subsequently created stacks will use instead of using the existing global stack options.
// name
// Stack option to modify
// value
// New value for the stack option.
// 0 on success
// errno set to EINVAL if the requested option is not found or ENOMEM.
int onload_stack_opt_set_int(const char* name, int64_t value)

onload_stack_opt_set_int(“EF_SCALABLE_FILTERS_ENABLE”, 1);

注意

Cannot be used to modify options on existing stacks ‐ only for new stacks.
Cannot be used to modify process options ‐ only stack options.
Modified options will be used for all newly created stacks until onload_stack_opt_reset() is called.

onload_stack_opt_reset

Revert to using global stack options for newly created stacks.
// 0 always
int onload_stack_opt_reset(void)

注意

Should be called following a call to onload_stack_opt_set_int() to revert to using global stack options for all newly created stacks.

Stacks API ‐ Examples

•This thread will use stack foo, other threads in the stack will continue as before.
onload_set_stackname(ONLOAD_THIS_THREAD, ONLOAD_SCOPE_GLOBAL, "foo")
•All threads in this process will get their own stack called foo. This is equivalent to the EF_STACK_PER_THREAD environment variable.
onload_set_stackname(ONLOAD_ALL_THREADS, ONLOAD_SCOPE_THREAD, "foo")
•All threads in this process will share a stack called foo. If another process did the same function call it will get its own stack.
onload_set_stackname(ONLOAD_ALL_THREADS, ONLOAD_SCOPE_PROCESS, "foo")
•All threads in this process will share a stack called foo. If another process run by the same user did the same, it would share the same stack as the first process. If another process run by a different user did the same it would get is own stack.
onload_set_stackname(ONLOAD_ALL_THREADS, ONLOAD_SCOPE_USER, "foo")

•Equivalent to EF_NAME. All threads will use a stack called foo which is shared by any other process which does the same.
onload_set_stackname(ONLOAD_ALL_THREADS, ONLOAD_SCOPE_GLOBAL, "foo")
•Equivalent to EF_DONT_ACCELERATE. New sockets/pipes will not be accelerated until another call to onload_set_stackname().
onload_set_stackname(ONLOAD_ALL_THREADS, ONLOAD_SCOPE_GLOBAL, ONLOAD_DONT_ACCELERATE)

onload_ordered_epoll_wait