linotp.lib.resources module

ResourceScheduler - handle iteration on resources list with blocking

similar to a list of circuit breakers.

the ResourceSchedule keeps track to the resouces, which might have been blocked for a certain time. After the blocking time has been expired, the access to the resource will be scheduled again

exception linotp.lib.resources.AllResourcesUnavailable

Bases: Exception

to be thrown when all services are unavailable.

class linotp.lib.resources.DictResourceRegistry

Bases: ResourceRegistry

the dict resource registry is a module global dict, which keeps an entry per resource. The advantage of the dict is that the operations on the dict are threadsafe. Other resource registries might use different strategies

see: http://effbot.org/

pyfaq/what-kinds-of-global-value-mutation-are-thread-safe.htm

remark: we use the tread safe dictionary method ‘setdefault’ which is an atomic version of:

>>>    if key not in dict:
>>>        dict[key] = value
>>>    return dic[key]
registry = {}
classmethod store(resource, value)

store the resource along with a new value

Parameters

  • resource – the resource

  • value – the value which should be associated with the resource

classmethod store_or_retrieve(resource, value)

if a resource already exists in the registry, it is returned. If it does not exist, it is preserved and then retrieved to be returned.

Parameters

  • resource – the resource

  • value – the fallback value, if the resource does not exist

Returns

the value associated with the resource

class linotp.lib.resources.ResourceRegistry

Bases: object

the resource registry is a global registry, which keeps an entry per resource. The resource is identified by a unique identifier, eg. URI. the value could be specific eg. the block expiration time only

classmethod store(resource, value)

abstraction to store the resource along with a new value

Parameters

  • resource – a resource identifier

  • value – the value which should be associated with the resource

classmethod store_or_retrieve(resource, value)

atomic operation - get or set a value from/into the registry. The set operation is only made, when there is no value for the resource in the registry

class linotp.lib.resources.ResourceScheduler(uri_list=None, tries=1, resource_registry_class=<class 'linotp.lib.resources.DictResourceRegistry'>)

Bases: object

Class to manage the list of resources (uris) in a global register, while keeping track of the connect-ability

example usage:

>>>    rs = ResourceScheduler(
>>>                retry=3, uri_list=['h://1', 'h://2', 'h://3', 'h://4'))
>>>
>>>    for uri in rs.next():
>>>        try:
>>>            return conn(uri)
>>>        except TimeoutException:
>>>            rs.block(uri, delay=2)
>>>
>>>    print "all resouces unavailable!"

about:blocking

the blocking with a fixed delay could lead to the problem, that when a server is offline, the server will be retried after a short delay. Looking at the relevant functions next (S) and block (b) we have the following event sequence with a constant delay between the retries.

———————————————————-> timeline

S S b S b S b S b S S

  • the sequence ( S -> S ) indicate that the server has been reachable in the first request.

What required is, is that we remember if the previous state already has been blocked - the block_indicator

———————————————————-> timeline

S S b S b S b S b S S

in 0 0 0 1 0 1 0 1 0 1 0 out 0 0 1 0 1 0 1 0 1 0 0

as we can see, that if the next():S gets the input 1, it knows, that the former request was blocked.

But this helps nothing as the next():S always resets the block status to give a chance for a new request. The idea is now to just remember the former state and this could as well be used to aggregate the number of b->S sequences. Therefore we use a second state, the block_counter. Thus we have the tuple

(block_indicator, block_counter)

with the following algorithm

  • the func block():b always only sets the block_indicator (x,n) -> (1,n)

  • the func next():S takes the block_indicator - if set, and adds it to the block_counter (1,n) -> (0, n+1) - if not set, the block_counter will be reseted (0,n) -> (0,0)

the upper sequence would have the following counters:

———————————————————-> timeline

s S b S b S b S b S S

in 0,0 0,0 0,0 1,0 0,1 1,1 0,2 1,2 0,3 1,4 0,5 out 0,0 0,0 1,0 0,1 1,1 0,2 1,2 0,3 1,4 0,5 0,0

Now we can dynamically adjust the delay by the doubling the time based on the counter: delay + delay * 2**n assuming a delay of 30 seconds, this will result in a sequence of doubling delays

0 -> 30 = 0,5 Min 1 -> 30 + 30 * 2^1 = 90 = 1,5 Min 2 -> 30 + 30 * 2^2 = 150 = 2,5 Min 3 -> 30 + 30 * 2^3 = 270 = 4,5 Min 4 -> 30 + 30 * 2^4 = 510 = 8,5 Min 5 -> 30 + 30 * 2^5 = 990 = 16,5 Min 6 -> 30 + 30 * 2^6 = 1950 = 32,5 Min 7 -> 30 + 30 * 2^7 = 3970 = 64,5 Min 8 -> 30 + 30 * 2^8 = 7710 = 128,5 Min

the max delay time should be limited to 8 which is ~2 hours

block(resource, delay=30, immediately=False)

mark the given resource as blocked for a delay of seconds

  • the blocking is only possible if all retries are made. this is controlled by the _retry_complete, that is controlled in the retry iterator

Parameters

  • resource – the resource that should be blocked

  • delay – optional - specify the delay, till a request should be re-triggered

  • immediately – should be locked immediately

linotp.lib.resources.string_to_list(string_list, sep=',')

tiny helper to create a list from a string with separators

Parameters

  • string_list – single string which should be split into a list

  • sep – the item separator