Callback for initial messages on incoming connections. Handles encryption, utility messages, and association of the connection with a Node.
        Once this initial setup is done, the relevant connected callback is executed, and further messages are deferred to the onMessageReceived callback.

        :param conn: connection object
        :type conn: TcpConnection
        :param message: received message
        :type message: any


Check whether this node should initiate a connection to another node

        :param node: the other node
        :type node: Node


Connect to a node if necessary.

        :param node: node to connect to
        :type node: Node


Callback for receiving a message on a new outgoing connection. Used only if encryption is enabled to exchange the random keys.
        Once the key exchange is done, this triggers the onNodeConnected callback, and further messages are deferred to the onMessageReceived callback.

        :param conn: connection object
        :type conn: TcpConnection
        :param message: received message
        :type message: any


Callback for when a connection is terminated or considered dead. Initiates a reconnect if necessary.

        :param conn: connection object
        :type conn: TcpConnection


Send a message to a node. Returns False if the connection appears to be dead either before or after actually trying to send the message.

        :param node: target node
        :type node: Node
        :param message: message
        :param message: any
        :returns success
        :rtype bool


Destroy this transport


Put an item into the queue.
        True - if item placed in queue.
        False - if queue is full and item can not be placed.


Put an item into the queue. Items should be comparable, eg. tuples.
        True - if item placed in queue.
        False - if queue is full and item can not be placed.


Extract the smallest item from queue.
        Return default if queue is empty.


Attempt to acquire lock.

        :param lockID: unique lock identifier.
        :type lockID: str
        :param sync: True - to wait until lock is acquired or failed to acquire.
        :type sync: bool
        :param callback: if sync is False - callback will be called with operation result.
        :type callback: func(opResult, error)
        :param timeout: max operation time (default - unlimited)
        :type timeout: float
        :return True if acquired, False - somebody else already acquired lock


Check if lock is acquired by ourselves.

        :param lockID: unique lock identifier.
        :type lockID: str
        :return True if lock is acquired by ourselves.


Decorator which wraps checks and returns an error response on failure.


Decorator which ensures that one of the WATCHMAN_TOKENS is provided if set.

    WATCHMAN_TOKEN_NAME can also be set if the token GET parameter must be
    customized.


Establish a connection to the chat server.

        Returns when an error has occurred, or :func:`disconnect` has been
        called.


Return ``request_header`` for use when constructing requests.

        Returns:
            Populated request header.


Set this client as active.

        While a client is active, no other clients will raise notifications.
        Call this method whenever there is an indication the user is
        interacting with this client. This method may be called very
        frequently, and it will only make a request when necessary.


Parse the image upload response to obtain status.

        Args:
            res: http_utils.FetchResponse instance, the upload response

        Returns:
            dict, sessionStatus of the response

        Raises:
            hangups.NetworkError: If the upload request failed.


Parse channel array and call the appropriate events.


Add services to the channel.

        The services we add to the channel determine what kind of data we will
        receive on it.

        The "babel" service includes what we need for Hangouts. If this fails
        for some reason, hangups will never receive any events. The
        "babel_presence_last_seen" service is also required to receive presence
        notifications.

        This needs to be re-called whenever we open a new channel (when there's
        a new SID and client_id.


Send a Protocol Buffer formatted chat API request.

        Args:
            endpoint (str): The chat API endpoint to use.
            request_pb: The request body as a Protocol Buffer message.
            response_pb: The response body as a Protocol Buffer message.

        Raises:
            NetworkError: If the request fails.


Send a generic authenticated POST request.

        Args:
            url (str): URL of request.
            content_type (str): Request content type.
            response_type (str): The desired response format. Valid options
                are: 'json' (JSON), 'protojson' (pblite), and 'proto' (binary
                Protocol Buffer). 'proto' requires manually setting an extra
                header 'X-Goog-Encode-Response-If-Executable: base64'.
            data (str): Request body data.

        Returns:
            FetchResponse: Response containing HTTP code, cookies, and body.

        Raises:
            NetworkError: If the request fails.


Invite users to join an existing group conversation.


Create a new conversation.


Return conversation info and recent events.


Return one or more user entities.

        Searching by phone number only finds entities when their phone number
        is in your contacts (and not always even then), and can't be used to
        find Google Voice contacts.


Return info about the current user.


Return presence status for a list of users.


Remove a participant from a group conversation.


Rename a conversation.

        Both group and one-to-one conversations may be renamed, but the
        official Hangouts clients have mixed support for one-to-one
        conversations with custom names.


Enable or disable message history in a conversation.


Set the notification level of a conversation.


Set focus to a conversation.


Set whether group link sharing is enabled for a conversation.


Set the typing status of a conversation.


Return info on recent conversations and their events.


Convert a microsecond timestamp to a UTC datetime instance.


Convert UserID to hangouts_pb2.ParticipantId.


Return WatermarkNotification from hangouts_pb2.WatermarkNotification.


Return authorization headers for API request.


Make an HTTP request.

        Automatically uses configured HTTP proxy, and adds Google authorization
        header and cookies.

        Failures will be retried MAX_RETRIES times before raising NetworkError.

        Args:
            method (str): Request method.
            url (str): Request URL.
            params (dict): (optional) Request query string parameters.
            headers (dict): (optional) Request headers.
            data: (str): (optional) Request body data.

        Returns:
            FetchResponse: Response data.

        Raises:
            NetworkError: If the request fails.


Search for entities by phone number, email, or gaia_id.


Return EntityLookupSpec from phone number, email address, or gaia ID.


Return a readable name for a conversation.

    If the conversation has a custom name, use the custom name. Otherwise, for
    one-to-one conversations, the name is the full name of the other user. For
    group conversations, the name is a comma-separated list of first names. If
    the group conversation is empty, the name is "Empty Conversation".

    If truncate is true, only show up to two names in a group conversation.

    If show_unread is True, if there are unread chat messages, show the number
    of unread chat messages in parentheses after the conversation name.


Add foreground and background colours to a color scheme


Sync all conversations by making paginated requests.

    Conversations are ordered by ascending sort timestamp.

    Args:
        client (Client): Connected client.

    Raises:
        NetworkError: If the requests fail.

    Returns:
        tuple of list of ``ConversationState`` messages and sync timestamp


Loaded events which are unread sorted oldest to newest.

        Some Hangouts clients don't update the read timestamp for certain event
        types, such as membership changes, so this may return more unread
        events than these clients will show. There's also a delay between
        sending a message and the user's own message being considered read.

        (list of :class:`.ConversationEvent`).


Handle a watermark notification.


Update the internal state of the conversation.

        This method is used by :class:`.ConversationList` to maintain this
        instance.

        Args:
            conversation: ``Conversation`` message.


Wrap hangouts_pb2.Event in ConversationEvent subclass.


Add an event to the conversation.

        This method is used by :class:`.ConversationList` to maintain this
        instance.

        Args:
            event_: ``Event`` message.

        Returns:
            :class:`.ConversationEvent` representing the event.


Send a message to this conversation.

        A per-conversation lock is acquired to ensure that messages are sent in
        the correct order when this method is called multiple times
        asynchronously.

        Args:
            segments: List of :class:`.ChatMessageSegment` objects to include
                in the message.
            image_file: (optional) File-like object containing an image to be
                attached to the message.
            image_id: (optional) ID of an Picasa photo to be attached to the
                message. If you specify both ``image_file`` and ``image_id``
                together, ``image_file`` takes precedence and ``image_id`` will
                be ignored.
            image_user_id: (optional) Picasa user ID, required only if
                ``image_id`` refers to an image from a different Picasa user,
                such as Google's sticker user.

        Raises:
            .NetworkError: If the message cannot be sent.


Leave this conversation.

        Raises:
            .NetworkError: If conversation cannot be left.


Set the notification level of this conversation.

        Args:
            level: ``NOTIFICATION_LEVEL_QUIET`` to disable notifications, or
                ``NOTIFICATION_LEVEL_RING`` to enable them.

        Raises:
            .NetworkError: If the request fails.


Update the timestamp of the latest event which has been read.

        This method will avoid making an API request if it will have no effect.

        Args:
            read_timestamp (datetime.datetime): (optional) Timestamp to set.
                Defaults to the timestamp of the newest event.

        Raises:
            .NetworkError: If the timestamp cannot be updated.


Get all the conversations.

        Args:
            include_archived (bool): (optional) Whether to include archived
                conversations. Defaults to ``False``.

        Returns:
            List of all :class:`.Conversation` objects.


Leave a conversation.

        Args:
            conv_id (str): ID of conversation to leave.


Add new conversation from hangouts_pb2.Conversation


Get a cached conversation or fetch a missing conversation.

        Args:
            conv_id: string, conversation identifier

        Raises:
            NetworkError: If the request to fetch the conversation fails.

        Returns:
            :class:`.Conversation` with matching ID.


Receive a hangouts_pb2.Event and fan out to Conversations.

        Args:
            event_: hangouts_pb2.Event instance


Receive Conversation delta and create or update the conversation.

        Args:
            conversation: hangouts_pb2.Conversation instance

        Raises:
            NetworkError: A request to fetch the complete conversation failed.


Receive SetTypingNotification and update the conversation.

        Args:
            set_typing_notification: hangouts_pb2.SetTypingNotification
                instance


Receive WatermarkNotification and update the conversation.

        Args:
            watermark_notification: hangouts_pb2.WatermarkNotification instance


Sync conversation state and events that could have been missed.


Construct user from ``Entity`` message.

        Args:
            entity: ``Entity`` message.
            self_user_id (~hangups.user.UserID or None): The ID of the current
                user. If ``None``, assume ``entity`` is the current user.

        Returns:
            :class:`~hangups.user.User` object.


Get a user by its ID.

        Args:
            user_id (~hangups.user.UserID): The ID of the user.

        Raises:
            KeyError: If no such user is known.

        Returns:
            :class:`~hangups.user.User` with the given ID.


Add or upgrade User from ConversationParticipantData.


Add an observer to this event.

        Args:
            callback: A function or coroutine callback to call when the event
                is fired.

        Raises:
            ValueError: If the callback has already been added.


Remove an observer from this event.

        Args:
            callback: A function or coroutine callback to remove from this
                event.

        Raises:
            ValueError: If the callback is not an observer of this event.


Fire this event, calling all observers with the same arguments.


Run a hangups example coroutine.

    Args:
        example_coroutine (coroutine): Coroutine to run with a connected
            hangups client and arguments namespace as arguments.
        extra_args (str): Any extra command line arguments required by the
            example.


Return ArgumentParser with any extra arguments.


Print column headers and rows as a reStructuredText table.

    Args:
        col_tuple: Tuple of column name strings.
        row_tuples: List of tuples containing row data.


Generate doc for an enum.

    Args:
        enum_descriptor: descriptor_pb2.EnumDescriptorProto instance for enum
            to generate docs for.
        locations: Dictionary of location paths tuples to
            descriptor_pb2.SourceCodeInfo.Location instances.
        path: Path tuple to the enum definition.
        name_prefix: Optional prefix for this enum's name.


Create a directory if it does not exist.


Show the overlay menu.


Handle connecting for the first time.


Open conversation tab for new messages & pass events to notifier.


Put a coroutine in the queue to be executed.


Consume coroutines from the queue by executing them.


Rename conversation and call callback.


Re-order the conversations when an event occurs.


Make users stop typing when they send a message.


Update status text.


Return MessageWidget representing a ConversationEvent.

        Returns None if the ConversationEvent does not have a widget
        representation.


Handle updating and scrolling when a new event is added.

        Automatically scroll down to show the new text if the bottom is
        showing. This allows the user to scroll up to read previous messages
        while new messages are arriving.


Load more events for this conversation.


Return the menu widget associated with this widget.


Update this conversation's tab title.


Update tab display.


Add or modify a tab.

        If widget is not a tab, it will be added. If switch is True, switch to
        this tab. If title is given, set the tab's title.


Use the access token to get session cookies.

    Raises GoogleAuthError if session cookies could not be loaded.

    Returns dict of cookies.


Populate and submit a form on the current page.

        Raises GoogleAuthError if form can not be submitted.


Parse response format for request for new channel SID.

    Example format (after parsing JS):
    [   [0,["c","SID_HERE","",8]],
        [1,[{"gsid":"GSESSIONID_HERE"}]]]

    Returns (SID, gsessionid) tuple.


Listen for messages on the backwards channel.

        This method only returns when the connection has been closed due to an
        error.


Open a long-polling request and receive arrays.

        This method uses keep-alive to make re-opening the request faster, but
        the remote server will set the "Connection: close" header once an hour.

        Raises hangups.NetworkError or ChannelSessionError.


Parse push data and trigger events.


Decode optional or required field.


Decode repeated field.


Decode pblite to Protocol Buffer message.

    This method is permissive of decoding errors and will log them as warnings
    and continue decoding where possible.

    The first element of the outer pblite list must often be ignored using the
    ignore_first_item parameter because it contains an abbreviation of the name
    of the protobuf message (eg.  cscmrp for ClientSendChatMessageResponseP)
    that's not part of the protobuf.

    Args:
        message: protocol buffer message instance to decode into.
        pblite: list representing a pblite-serialized message.
        ignore_first_item: If True, ignore the item at index 0 in the pblite
            list, making the item at index 1 correspond to field 1 in the
            message.


Sets the Elasticsearch hosts to use

    Args:
        hosts (str): A single hostname or URL, or list of hostnames or URLs
        use_ssl (bool): Use a HTTPS connection to the server
        ssl_cert_path (str): Path to the certificate chain


Updates index mappings

    Args:
        aggregate_indexes (list): A list of aggregate index names
        forensic_indexes (list): A list of forensic index names


Saves aggregate DMARC reports to Kafka

        Args:
            aggregate_reports (list):  A list of aggregate report dictionaries
            to save to Kafka
            aggregate_topic (str): The name of the Kafka topic


Extracts xml from a zip or gzip file at the given path, file-like object,
    or bytes.

    Args:
        input_: A path to a file, a file like object, or bytes

    Returns:
        str: The extracted XML


Parses a file at the given path, a file-like object. or bytes as a
    aggregate DMARC report

    Args:
        _input: A path to a file, a file like object, or bytes
        nameservers (list): A list of one or more nameservers to use
        (Cloudflare's public DNS resolvers by default)
        dns_timeout (float): Sets the DNS timeout in seconds
        parallel (bool): Parallel processing

    Returns:
        OrderedDict: The parsed DMARC aggregate report


Converts one or more parsed forensic reports to flat CSV format, including
    headers

    Args:
        reports: A parsed forensic report or list of parsed forensic reports

    Returns:
        str: Parsed forensic report data in flat CSV format, including headers


Parses a DMARC aggregate or forensic file at the given path, a
    file-like object. or bytes

    Args:
        input_: A path to a file, a file like object, or bytes
        nameservers (list): A list of one or more nameservers to use
        (Cloudflare's public DNS resolvers by default)
        dns_timeout (float): Sets the DNS timeout in seconds
        strip_attachment_payloads (bool): Remove attachment payloads from
        forensic report results
        parallel (bool): Parallel processing

    Returns:
        OrderedDict: The parsed DMARC report


Returns a list of an IMAP server's capabilities

    Args:
        server (imapclient.IMAPClient): An instance of imapclient.IMAPClient

    Returns (list): A list of capabilities


Emails parsing results as a zip file

    Args:
        results (OrderedDict): Parsing results
        host: Mail server hostname or IP address
        mail_from: The value of the message from header
        mail_to : A list of addresses to mail to
        port (int): Port to use
        ssl (bool): Require a SSL connection from the start
        user: An optional username
        password: An optional password
        subject: Overrides the default message subject
        attachment_filename: Override the default attachment filename
        message: Override the default plain text body
        ssl_context: SSL context options


Saves aggregate DMARC reports to Splunk

        Args:
            aggregate_reports: A list of aggregate report dictionaries
            to save in Splunk


Decodes a base64 string, with padding being optional

    Args:
        data: A base64 encoded string

    Returns:
        bytes: The decoded bytes


Gets the base domain name for the given domain

    .. note::
        Results are based on a list of public domain suffixes at
        https://publicsuffix.org/list/public_suffix_list.dat.

    Args:
        domain (str): A domain or subdomain
        use_fresh_psl (bool): Download a fresh Public Suffix List

    Returns:
        str: The base domain of the given domain


Resolves an IP address to a hostname using a reverse DNS query

    Args:
        ip_address (str): The IP address to resolve
        cache (ExpiringDict): Cache storage
        nameservers (list): A list of one or more nameservers to use
        (Cloudflare's public DNS resolvers by default)
        timeout (float): Sets the DNS query timeout in seconds

    Returns:
        str: The reverse DNS hostname (if any)


Converts a human-readable timestamp into a Python ``DateTime`` object

    Args:
        human_timestamp (str): A timestamp string
        to_utc (bool): Convert the timestamp to UTC

    Returns:
        DateTime: The converted timestamp


Returns reverse DNS and country information for the given IP address

    Args:
        ip_address (str): The IP address to check
        cache (ExpiringDict): Cache storage
        nameservers (list): A list of one or more nameservers to use
        (Cloudflare's public DNS resolvers by default)
        timeout (float): Sets the DNS timeout in seconds
        parallel (bool): parallel processing

    Returns:
        OrderedDict: ``ip_address``, ``reverse_dns``


Uses the ``msgconvert`` Perl utility to convert an Outlook MS file to
    standard RFC 822 format

    Args:
        msg_bytes (bytes): the content of the .msg file

    Returns:
        A RFC 822 string


Separated this function for multiprocessing


Sends a PUB command to the server on the specified subject.

          ->> PUB hello 5
          ->> MSG_PAYLOAD: world
          <<- MSG hello 2 5


Publishes a message tagging it with a reply subscription
        which can be used by those receiving the message to respond.

           ->> PUB hello   _INBOX.2007314fe0fcb2cdc2a2914c1 5
           ->> MSG_PAYLOAD: world
           <<- MSG hello 2 _INBOX.2007314fe0fcb2cdc2a2914c1 5


Sets the subcription to use a task per message to be processed.

        ..deprecated:: 7.0
          Will be removed 9.0.


Sends a ping to the server expecting a pong back ensuring
        what we have written so far has made it to the server and
        also enabling measuring of roundtrip time.
        In case a pong is not returned within the allowed timeout,
        then it will raise ErrTimeout.


Looks up in the server pool for an available server
        and attempts to connect.


Process errors which occured while reading or parsing
        the protocol. If allow_reconnect is enabled it will
        try to switch the server to which it is currently connected
        otherwise it will disconnect.


Process PONG sent by server.


Coroutine which continuously tries to consume pending commands
        and then flushes them to the socket.


Coroutine which gathers bytes sent by the server
        and feeds them to the protocol parser.
        In case of error while reading, it will stop running
        and its task has to be rescheduled.


Generates a timezone aware datetime if the 'USE_TZ' setting is enabled

    :param value: The datetime value
    :return: A locale aware datetime


Load feature data from a 2D ndarray on disk.


Load feature image data from image files.

        Args:
          images: A list of image filenames.
          names: An optional list of strings to use as the feature names. Must
            be in the same order as the images.


Decode images using Pearson's r.

        Computes the correlation between each input image and each feature
        image across voxels.

        Args:
            imgs_to_decode: An ndarray of images to decode, with voxels in rows
                and images in columns.

        Returns:
            An n_features x n_images 2D array, with each cell representing the
            pearson correlation between the i'th feature and the j'th image
            across all voxels.


Decoding using the dot product.


Set up data for a classification task given a set of masks

        Given a set of masks, this function retrieves studies associated with
        each mask at the specified threshold, optionally removes overlap and
        filters by studies and features, and returns studies by feature matrix
        (X) and class labels (y)

        Args:
            dataset: a Neurosynth dataset
            maks: a list of paths to Nifti masks
            threshold: percentage of voxels active within the mask for study
                to be included
            remove_overlap: A boolean indicating if studies studies that
                appear in more than one mask should be excluded
            studies: An optional list of study names used to constrain the set
                used in classification. If None, will use all features in the
                dataset.
            features: An optional list of feature names used to constrain the
                set used in classification. If None, will use all features in
                the dataset.
            regularize: Optional boolean indicating if X should be regularized

        Returns:
            A tuple (X, y) of np arrays.
            X is a feature by studies matrix and y is a vector of class labels


Returns a list with the order that features requested appear in
    dataset


Sets the class_weight of the classifier to match y


Given a dataset, fits either features or voxels to y


Aggregates over all voxels within each ROI in the input image.

    Takes a Dataset and a Nifti image that defines distinct regions, and
    returns a numpy matrix of  ROIs x mappables, where the value at each
    ROI is the proportion of active voxels in that ROI. Each distinct ROI
    must have a unique value in the image; non-contiguous voxels with the
    same value will be assigned to the same ROI.

    Args:
        dataset: Either a Dataset instance from which image data are
            extracted, or a Numpy array containing image data to use. If
            the latter, the array contains voxels in rows and
            features/studies in columns. The number of voxels must be equal
            to the length of the vectorized image mask in the regions
            image.
        regions: An image defining the boundaries of the regions to use.
            Can be one of:
            1) A string name of the NIFTI or Analyze-format image
            2) A NiBabel SpatialImage
            3) A list of NiBabel images
            4) A 1D numpy array of the same length as the mask vector in
                the Dataset's current Masker.
        masker: Optional masker used to load image if regions is not a
            numpy array. Must be passed if dataset is a numpy array.
        threshold: An optional float in the range of 0 - 1 or integer. If
            passed, the array will be binarized, with ROI values above the
            threshold assigned to True and values below the threshold
            assigned to False. (E.g., if threshold = 0.05, only ROIs in
            which more than 5% of voxels are active will be considered
            active.) If threshold is integer, studies will only be
            considered active if they activate more than that number of
            voxels in the ROI.
        remove_zero: An optional boolean; when True, assume that voxels
            with value of 0 should not be considered as a separate ROI, and
            will be ignored.

    Returns:
        A 2D numpy array with ROIs in rows and mappables in columns.


Return top forty words from each topic in trained topic model.


Correlates row vector x with each row vector in 2D array y.


Determine FDR threshold given a p value array and desired false
    discovery rate q.


Create and store a new ImageTable instance based on the current
        Dataset. Will generally be called privately, but may be useful as a
        convenience method in cases where the user wants to re-generate the
        table with a new smoothing kernel of different radius.

        Args:
            r (int): An optional integer indicating the radius of the smoothing
                kernel. By default, this is None, which will keep whatever
                value is currently set in the Dataset instance.


Construct a new FeatureTable from file.

        Args:
            features: Feature data to add. Can be:
                (a) A text file containing the feature data, where each row is
                a study in the database, with features in columns. The first
                column must contain the IDs of the studies to match up with the
                image data.
                (b) A pandas DataFrame, where studies are in rows, features are
                in columns, and the index provides the study IDs.
            append (bool): If True, adds new features to existing ones
                incrementally. If False, replaces old features.
            merge, duplicates, min_studies, threshold: Additional arguments
                passed to FeatureTable.add_features().


Load a pickled Dataset instance from file.


Given a list of features, returns features in order that they
        appear in database.

        Args:
            features (list): A list or 1D numpy array of named features to
            return.

        Returns:
            A list of features in order they appear in database.


Returns a list of all studies in the table that meet the desired
        feature-based criteria.

        Will most commonly be used to retrieve studies that use one or more
        features with some minimum frequency; e.g.,:

            get_ids(['fear', 'anxiety'], threshold=0.001)

        Args:
            features (lists): a list of feature names to search on.
            threshold (float): optional float indicating threshold features
                must pass to be included.
            func (Callable): any numpy function to use for thresholding
                (default: sum). The function will be applied to the list of
                features and the result compared to the threshold. This can be
                used to change the meaning of the query in powerful ways. E.g,:
                    max: any of the features have to pass threshold
                        (i.e., max > thresh)
                    min: all features must each individually pass threshold
                        (i.e., min > thresh)
                    sum: the summed weight of all features must pass threshold
                        (i.e., sum > thresh)
            get_weights (bool): if True, returns a dict with ids => weights.

        Returns:
            When get_weights is false (default), returns a list of study
                names. When true, returns a dict, with study names as keys
                and feature weights as values.


Returns all features that match any of the elements in the input
        list.

        Args:
            search (str, list): A string or list of strings defining the query.

        Returns:
            A list of matching feature names.


Convert FeatureTable to SciPy CSR matrix.


Deprecation warning decorator. Takes optional deprecation message,
    otherwise will use a generic warning.


Convert coordinates from one space to another using provided
    transformation matrix.


Convert an N x 3 array of XYZ coordinates to matrix indices.


Perform an ADC read with the provided mux, gain, data_rate, and mode
        values and with the comparator enabled as specified.  Returns the signed
        integer result of the read.


Read a single ADC channel and return the ADC value as a signed integer
        result.  Channel must be a value within 0-3.


Expand the given address into one or more normalized strings.

    Required
    --------
    @param address: the address as either Unicode or a UTF-8 encoded string

    Options
    -------
    @param languages: a tuple or list of ISO language code strings (e.g. "en", "fr", "de", etc.)
                      to use in expansion. If None is passed, use language classifier
                      to detect language automatically.
    @param address_components: an integer (bit-set) of address component expansions
                               to use e.g. ADDRESS_NAME | ADDRESS_STREET would use
                               only expansions which apply to venue names or streets.
    @param latin_ascii: use the Latin to ASCII transliterator, which normalizes e.g. æ => ae
    @param transliterate: use any available transliterators for non-Latin scripts, e.g.
                          for the Greek phrase διαφορετικούς becomes diaphoretikoús̱
    @param strip_accents: strip accented characters e.g. é => e, ç => c. This loses some
                          information in various languags, but in general we want
    @param decompose: perform Unicode normalization (NFD form)
    @param lowercase: UTF-8 lowercase the string
    @param trim_string: trim spaces on either side of the string
    @param replace_word_hyphens: add version of the string replacing hyphens with space
    @param delete_word_hyphens: add version of the string with hyphens deleted
    @param replace_numeric_hyphens: add version of the string with numeric hyphens replaced
                                    e.g. 12345-6789 => 12345 6789
    @param delete_numeric_hyphens: add version of the string with numeric hyphens removed
                                   e.g. 12345-6789 => 123456789
    @param split_alpha_from_numeric: split tokens like CR17 into CR 17, helps with expansion
                                     of certain types of highway abbreviations
    @param delete_final_periods: remove final periods on abbreviations e.g. St. => St
    @param delete_acronym_periods: remove periods in acronyms e.g. U.S.A. => USA
    @param drop_english_possessives: normalize possessives e.g. Mark's => Marks
    @param delete_apostrophes: delete other types of hyphens e.g. O'Malley => OMalley
    @param expand_numex: converts numeric expressions e.g. Twenty sixth => 26th,
                         using either the supplied languages or the result of
                         automated language classification.
    @param roman_numerals: normalize Roman numerals e.g. IX => 9. Since these can be
                           ambiguous (especially I and V), turning this on simply
                           adds another version of the string if any potential
                           Roman numerals are found.


Normalizes a string, tokenizes, and normalizes each token
    with string and token-level options.

    This version only uses libpostal's deterministic normalizations
    i.e. methods with a single output. The string tree version will
    return multiple normalized strings, each with tokens.

    Usage:
        normalized_tokens(u'St.-Barthélemy')


Parse address into components.

    @param address: the address as either Unicode or a UTF-8 encoded string
    @param language (optional): language code
    @param country (optional): country code


Hash the given address into normalized strings that can be used to group similar
    addresses together for more detailed pairwise comparison. This can be thought of
    as the blocking function in record linkage or locally-sensitive hashing in the
    document near-duplicate detection.

    Required
    --------
    @param labels: array of component labels as either Unicode or UTF-8 encoded strings
                   e.g. ["house_number", "road", "postcode"]
    @param values: array of component values as either Unicode or UTF-8 encoded strings
                   e.g. ["123", "Broadway", "11216"]. Note len(values) must be equal to
                   len(labels).

    Options
    -------
    @param languages: a tuple or list of ISO language code strings (e.g. "en", "fr", "de", etc.)
                      to use in expansion. If None is passed, use language classifier
                      to detect language automatically.
    @param with_name: use name in the hashes
    @param with_address: use house_number & street in the hashes
    @param with_unit: use secondary unit as part of the hashes
    @param with_city_or_equivalent: use the city, city_district, suburb, or island name as one of
                                    the geo qualifiers
    @param with_small_containing_boundaries: use small containing boundaries (currently state_district)
                                             as one of the geo qualifiers
    @param with_postal_code: use postal code as one of the geo qualifiers
    @param with_latlon: use geohash + neighbors as one of the geo qualifiers
    @param latitude: latitude (Y coordinate)
    @param longitude: longitude (X coordinate)
    @param geohash_precision: geohash tile size (default = 6)
    @param name_and_address_keys: include keys with name + address + geo
    @param name_only_keys: include keys with name + geo
    @param address_only_keys: include keys with address + geo


Removed all dusty containers with 'Exited' in their status


Removes all dangling images as well as all images referenced in a dusty spec; forceful removal is not used


Write the given config to disk as a Dusty sub-config
    in the Nginx includes directory. Then, either start nginx
    or tell it to reload its config to pick up what we've
    just written.


We require the list of all remote repo paths to be passed in
        to this because otherwise we would need to import the spec assembler
        in this module, which would give us circular imports.


Daemon-side command to ensure we're running the latest
    versions of any managed repos, including the
    specs repo, before we do anything else in the up flow.


This command will use the compilers to get compose specs
    will pass those specs to the systems that need them. Those
    systems will in turn launch the services needed to make the
    local environment go.


Restart any containers associated with Dusty, or associated with
    the provided app_or_service_names.


Return a dictionary containing the Compose spec required to run
    Dusty's nginx container used for host forwarding.


Given the assembled specs and app_name, this function will return all apps and services specified in
    'conditional_links' if they are specified in 'apps' or 'services' in assembled_specs. That means that
    some other part of the system has declared them as necessary, so they should be linked to this app


This function returns a dictionary of the docker-compose.yml specifications for one app


This function returns a dictionary of the docker_compose specifications
    for one service. Currently, this is just the Dusty service spec with
    an additional volume mount to support Dusty's cp functionality.


Returns a list of formatted port mappings for an app


This returns formatted volume specifications for a docker-compose app. We mount the app
    as well as any libs it needs so that local code is used in our container, instead of whatever
    code was in the docker image.

    Additionally, we create a volume for the /cp directory used by Dusty to facilitate
    easy file transfers using `dusty cp`.


Expands specs.libs.depends.libs to include any indirectly required libs


Returns all libs that are referenced in specs.apps.depends.libs


Returns all services that are referenced in specs.apps.depends.services,
    or in specs.bundles.services


This function adds an assets key to the specs, which is filled in with a dictionary
    of all assets defined by apps and libs in the specs


This function takes an app or library name and will return the corresponding repo
    for that app or library


Given the spec of an app or library, returns all repos that are guaranteed
    to live in the same container


Given the name of an app or library, returns all repos that are guaranteed
    to live in the same container


Return a string of all host rules required to match
    the given spec. This string is wrapped in the Dusty hosts
    header and footer so it can be easily removed later.


Moves the temporary binary to the location of the binary that's currently being run.
    Preserves owner, group, and permissions of original binary


Context manager for setting up a TaskQueue. Upon leaving the
    context manager, all tasks that were enqueued will be executed
    in parallel subject to `pool_size` concurrency constraints.


This will output the nginx stream config string for specific port spec


Starting with Yosemite, launchd was rearchitected and now only one
    launchd process runs for all users. This allows us to much more easily
    impersonate a user through launchd and extract the environment
    variables from their running processes.


Will check the mac_username config value; if it is present, will load that user's
    SSH_AUTH_SOCK environment variable to the current environment.  This allows git clones
    to behave the same for the daemon as they do for the user


Recursively delete a path upon exiting this context
    manager. Supports targets that are files or directories.


Copy a path from the local filesystem to a path inside a Dusty
    container. The files on the local filesystem must be accessible
    by the user specified in mac_username.


Given a dictionary containing the expanded dusty DAG specs this function will
    return a dictionary containing the port mappings needed by downstream methods.  Currently
    this includes docker_compose, virtualbox, nginx and hosts_file.


Returns the Docker registry host associated with
    a given image name.


Reads the local Docker client config for the current user
    and returns all registries to which the user may be logged in.
    This is intended to be run client-side, not by the daemon.


Puts the client logger into streaming mode, which sends
    unbuffered input through to the socket one character at a time.
    We also disable propagation so the root logger does not
    receive many one-byte emissions. This context handler
    was originally created for streaming Compose up's
    terminal output through to the client and should only be
    used for similarly complex circumstances.


This is used to compile the command that will be run when the docker container starts
    up. This command has to install any libs that the app uses, run the `always` command, and
    run the `once` command if the container is being launched for the first time


Raise the open file handles permitted by the Dusty daemon process
    and its child processes. The number we choose here needs to be within
    the OS X default kernel hard limit, which is 10240.


Start the daemon's HTTP server on a separate thread.
    This server is only used for servicing container status
    requests from Dusty's custom 502 page.


Ripped off and slightly modified based on docker-py's
    kwargs_from_env utility function.


Get a list of containers associated with the list
    of services. If no services are provided, attempts to
    return all containers associated with Dusty.


This function is used with `dusty up`.  It will check all active repos to see if
    they are exported.  If any are missing, it will replace current dusty exports with
    exports that are needed for currently active repos, and restart
    the nfs server


Our exports file will be invalid if this folder doesn't exist, and the NFS server
    will not run correctly.


Given an existing consumer ID, return any new lines from the
    log since the last time the consumer was consumed.


This returns a list of formatted volume specs for an app. These mounts declared in the apps' spec
    and mounts declared in all lib specs the app depends on


Returns a list of the formatted volume specs for a lib


Returns a list of the formatted volume mounts for all libs that an app uses


Initialize the Dusty VM if it does not already exist.


Start the Dusty VM if it is not already running.


Using VBoxManage is 0.5 seconds or so faster than Machine.


Something in the VM chain, either VirtualBox or Machine, helpfully
    sets up localhost-to-VM forwarding on port 22. We can inspect this
    rule to determine the port on localhost which gets forwarded to
    22 in the VM.


Returns the MAC address assigned to the host-only adapter,
    using output from VBoxManage. Returned MAC address has no colons
    and is lower-cased.


Given the rather-complex output from an 'ip addr show' command
    on the VM, parse the output to determine the IP address
    assigned to the interface with the given MAC.


Determine the host-only IP of the Dusty VM through Virtualbox and SSH
    directly, bypassing Docker Machine. We do this because Docker Machine is
    much slower, taking about 600ms total. We are basically doing the same
    flow Docker Machine does in its own code.


Converts a python dict to a namedtuple, saving memory.


By default, return latest EOD Composite Price for a stock ticker.
           On average, each feed contains 3 data sources.

            Supported tickers + Available Day Ranges are here:
            https://apimedia.tiingo.com/docs/tiingo/daily/supported_tickers.zip

            Args:
                ticker (string): Unique identifier for stock ticker
                startDate (string): Start of ticker range in YYYY-MM-DD format
                endDate (string): End of ticker range in YYYY-MM-DD format
                fmt (string): 'csv' or 'json'
                frequency (string): Resample frequency


Return a pandas.DataFrame of historical prices for one or more ticker symbols.

            By default, return latest EOD Composite Price for a list of stock tickers.
            On average, each feed contains 3 data sources.

            Supported tickers + Available Day Ranges are here:
            https://apimedia.tiingo.com/docs/tiingo/daily/supported_tickers.zip
            or from the TiingoClient.list_tickers() method.

            Args:
                tickers (string/list): One or more unique identifiers for a stock ticker.
                startDate (string): Start of ticker range in YYYY-MM-DD format.
                endDate (string): End of ticker range in YYYY-MM-DD format.
                metric_name (string): Optional parameter specifying metric to be returned for each
                    ticker.  In the event of a single ticker, this is optional and if not specified
                    all of the available data will be returned.  In the event of a list of tickers,
                    this parameter is required.
                frequency (string): Resample frequency (defaults to daily).


Make a local copy of the sqlite cookie database and return the new filename.
    This is necessary in case this database is still being written to while the user browses
    to avoid sqlite locking errors.


Try to load cookies from all supported browsers and return combined cookiejar
    Optionally pass in a domain name to only load cookies from the specified domain


Decrypt encoded cookies


Get the application bearer token from client_id and client_secret.


Make a request to the spotify API with the current bearer credentials.

        Parameters
        ----------
        route : Union[tuple[str, str], Route]
            A tuple of the method and url or a :class:`Route` object.
        kwargs : Any
            keyword arguments to pass into :class:`aiohttp.ClientSession.request`


Get an albums tracks by an ID.

        Parameters
        ----------
        spotify_id : str
            The spotify_id to search by.
        limit : Optional[int]
            The maximum number of items to return. Default: 20. Minimum: 1. Maximum: 50.
        offset : Optiona[int]
            The offset of which Spotify should start yielding from.
        market : Optional[str]
            An ISO 3166-1 alpha-2 country code.


Get a spotify artist by their ID.

        Parameters
        ----------
        spotify_id : str
            The spotify_id to search by.


Get an artists tracks by their ID.

        Parameters
        ----------
        spotify_id : str
            The spotify_id to search by.
        include_groups : INCLUDE_GROUPS_TP
            INCLUDE_GROUPS
        limit : Optional[int]
            The maximum number of items to return. Default: 20. Minimum: 1. Maximum: 50.
        offset : Optiona[int]
            The offset of which Spotify should start yielding from.
        market : Optional[str]
            An ISO 3166-1 alpha-2 country code.


Get an artists top tracks per country with their ID.

        Parameters
        ----------
        spotify_id : str
            The spotify_id to search by.
        country : COUNTRY_TP
            COUNTRY


Get related artists for an artist by their ID.

        Parameters
        ----------
        spotify_id : str
            The spotify_id to search by.


Get a single category used to tag items in Spotify.

        Parameters
        ----------
        category_id : str
            The Spotify category ID for the category.
        country : COUNTRY_TP
            COUNTRY
        locale : LOCALE_TP
            LOCALE


Get a list of Spotify playlists tagged with a particular category.

        Parameters
        ----------
        category_id : str
            The Spotify category ID for the category.
        limit : Optional[int]
            The maximum number of items to return. Default: 20. Minimum: 1. Maximum: 50.
        offset : Optional[int]
            The index of the first item to return. Default: 0
        country : COUNTRY_TP
            COUNTRY


Get a list of categories used to tag items in Spotify.

        Parameters
        ----------
        limit : Optional[int]
            The maximum number of items to return. Default: 20. Minimum: 1. Maximum: 50.
        offset : Optional[int]
            The index of the first item to return. Default: 0
        country : COUNTRY_TP
            COUNTRY
        locale : LOCALE_TP
            LOCALE


Get a list of Spotify featured playlists.

        Parameters
        ----------
        locale : LOCALE_TP
            LOCALE
        country : COUNTRY_TP
            COUNTRY
        timestamp : TIMESTAMP_TP
            TIMESTAMP
        limit : Optional[int]
            The maximum number of items to return. Default: 20. Minimum: 1. Maximum: 50.
        offset : Optional[int]
            The index of the first item to return. Default: 0


Get a list of new album releases featured in Spotify.

        Parameters
        ----------
        limit : Optional[int]
            The maximum number of items to return. Default: 20. Minimum: 1. Maximum: 50.
        offset : Optional[int]
            The index of the first item to return. Default: 0
        country : COUNTRY_TP
            COUNTRY


Get Recommendations Based on Seeds.

        Parameters
        ----------
        seed_artists : str
            A comma separated list of Spotify IDs for seed artists. Up to 5 seed values may be provided.
        seed_genres : str
            A comma separated list of any genres in the set of available genre seeds. Up to 5 seed values may be provided.
        seed_tracks : str
            A comma separated list of Spotify IDs for a seed track. Up to 5 seed values may be provided.
        limit : Optional[int]
            The maximum number of items to return. Default: 20. Minimum: 1. Maximum: 50.
        market : Optional[str]
            An ISO 3166-1 alpha-2 country code.
        max_* : Optional[Keyword arguments]
            For each tunable track attribute, a hard ceiling on the selected track attribute’s value can be provided.
        min_* : Optional[Keyword arguments]
            For each tunable track attribute, a hard floor on the selected track attribute’s value can be provided.
        target_* : Optional[Keyword arguments]
            For each of the tunable track attributes (below) a target value may be provided.


Check to see if the current user is following one or more artists or other Spotify users.

        Parameters
        ----------
        ids : List[str]
            A comma-separated list of the artist or the user Spotify IDs to check.
            A maximum of 50 IDs can be sent in one request.
        type : Optional[str]
            The ID type: either "artist" or "user".
            Default: "artist"


Get the albums of a Spotify artist.

        Parameters
        ----------
        limit : Optional[int]
            The maximum number of items to return. Default: 20. Minimum: 1. Maximum: 50.
        offset : Optiona[int]
            The offset of which Spotify should start yielding from.
        include_groups : INCLUDE_GROUPS_TP
            INCLUDE_GROUPS
        market : Optional[str]
            An ISO 3166-1 alpha-2 country code.

        Returns
        -------
        albums : List[Album]
            The albums of the artist.


get the total amout of tracks in the album.

        Parameters
        ----------
        market : Optional[str]
            An ISO 3166-1 alpha-2 country code.

        Returns
        -------
        total : int
            The total amount of albums.


Get the users currently playing track.

        Returns
        -------
        context, track : Tuple[Context, Track]
            A tuple of the context and track.


Get information about the users avaliable devices.

        Returns
        -------
        devices : List[Device]
            The devices the user has available.


Get tracks from the current users recently played tracks.

        Returns
        -------
        playlist_history : List[Dict[str, Union[Track, Context, str]]]
            A list of playlist history object.
            Each object is a dict with a timestamp, track and context field.


Create a playlist for a Spotify user.

        Parameters
        ----------
        name : str
            The name of the playlist.
        public : Optional[bool]
            The public/private status of the playlist.
            `True` for public, `False` for private.
        collaborative : Optional[bool]
            If `True`, the playlist will become collaborative and other users will be able to modify the playlist.
        description : Optional[str]
            The playlist description

        Returns
        -------
        playlist : Playlist
            The playlist that was created.


get the albums tracks from spotify.

        Parameters
        ----------
        limit : Optional[int]
            The limit on how many tracks to retrieve for this album (default is 20).
        offset : Optional[int]
            The offset from where the api should start from in the tracks.

        Returns
        -------
        tracks : List[Track]
            The tracks of the artist.


loads all of the albums tracks, depending on how many the album has this may be a long operation.

        Parameters
        ----------
        market : Optional[str]
            An ISO 3166-1 alpha-2 country code. Provide this parameter if you want to apply Track Relinking.

        Returns
        -------
        tracks : List[Track]
            The tracks of the artist.


Retrive an album with a spotify ID.

        Parameters
        ----------
        spotify_id : str
            The ID to search for.
        market : Optional[str]
            An ISO 3166-1 alpha-2 country code

        Returns
        -------
        album : Album
            The album from the ID


Retrive an track with a spotify ID.

        Parameters
        ----------
        spotify_id : str
            The ID to search for.

        Returns
        -------
        track : Track
            The track from the ID


Retrive multiple albums with a list of spotify IDs.

        Parameters
        ----------
        ids : List[str]
            the ID to look for
        market : Optional[str]
            An ISO 3166-1 alpha-2 country code

        Returns
        -------
        albums : List[Album]
            The albums from the IDs


Retrive multiple artists with a list of spotify IDs.

        Parameters
        ----------
        ids : List[str]
            the IDs to look for

        Returns
        -------
        artists : List[Artist]
            The artists from the IDs


decorator to assert an object has an attribute when run.


Construct a OAuth2 object from a `spotify.Client`.


Construct a OAuth2 URL instead of an OAuth2 object.


Attributes used when constructing url parameters.


URL parameters used.


Execute the logic behind the meaning of ExpirationDate + return the matched status.

        :return:
            The status of the tested domain.
            Can be one of the official status.
        :rtype: str


Read the code and update all links.


Check if the current version is greater as the older older one.


Check if the current branch is `dev`.


Check if we have to put the previous version into the deprecated list.


Backup the current execution state.


Restore data from the given path.


Check if we have to ignore the given line.

        :param line: The line from the file.
        :type line: str


Handle the data from the options.

        :param options: The list of options from the rule.
        :type options: list

        :return: The list of domains to return globally.
        :rtype: list


Format the exctracted adblock line before passing it to the system.

        :param to_format: The extracted line from the file.
        :type to_format: str

        :param result: A list of the result of this method.
        :type result: list

        :return: The list of domains or IP to test.
        :rtype: list


Return the HTTP code status.

        :return: The matched and formatted status code.
        :rtype: str|int|None


Check if the given domain is a subdomain.

    :param domain: The domain we are checking.
    :type domain: str

    :return: The subdomain state.
    :rtype: bool

    .. warning::
        If an empty or a non-string :code:`domain` is given, we return :code:`None`.


Check the syntax of the given IPv4.

    :param ip: The IPv4 to check the syntax for.
    :type ip: str

    :return: The syntax validity.
    :rtype: bool

    .. warning::
        If an empty or a non-string :code:`ip` is given, we return :code:`None`.


Check if the given information is a URL.
        If it is the case, it download and update the location of file to test.

        :param passed: The url passed to the system.
        :type passed: str

        :return: The state of the check.
        :rtype: bool


Manage the loading of the url system.


Decide if we print or not the header.


Manage the database, autosave and autocontinue systems for the case that we are reading
        a file.

        :param current: The currently tested element.
        :type current: str

        :param last: The last element of the list.
        :type last: str

        :param status: The status of the currently tested element.
        :type status: str


Manage the case that we want to test only a domain.

        :param domain: The domain or IP to test.
        :type domain: str

        :param last_domain:
            The last domain to test if we are testing a file.
        :type last_domain: str

        :param return_status: Tell us if we need to return the status.
        :type return_status: bool


Manage the case that we want to test only a given url.

        :param url_to_test: The url to test.
        :type url_to_test: str

        :param last_url:
            The last url of the file we are testing
            (if exist)
        :type last_url: str


Format the extracted domain before passing it to the system.

        :param extracted_domain: The extracted domain.
        :type extracted_domain: str

        :return: The formatted domain or IP to test.
        :rtype: str

        .. note:
            Understand by formating the fact that we get rid
            of all the noises around the domain we want to test.


Extract all non commented lines from the file we are testing.

        :return: The elements to test.
        :rtype: list


Manage the case that need to test each domain of a given file path.

        .. note::
            1 domain per line.


Manage the case that we have to test a file

        .. note::
            1 URL per line.


Switch PyFunceble.CONFIGURATION variables to their opposite.

        :param variable:
            The variable name to switch.
            The variable should be an index our configuration system.
            If we want to switch a bool variable, we should parse
            it here.
        :type variable: str|bool

        :param custom:
            Let us know if have to switch the parsed variable instead
            of our configuration index.
        :type custom: bool

        :return:
            The opposite of the configuration index or the given variable.
        :rtype: bool

        :raises:
            :code:`Exception`
                When the configuration is not valid. In other words,
                if the PyFunceble.CONFIGURATION[variable_name] is not a bool.


Get the status while testing for an IP or domain.

        .. note::
            We consider that the domain or IP we are currently testing
            is into :code:`PyFunceble.INTERN["to_test"]`.


Handle the backend of the given status.


Get the structure we are going to work with.

        :return: The structure we have to work with.
        :rtype: dict


Creates the given directory if it does not exists.

        :param directory: The directory to create.
        :type directory: str

        :param loop: Tell us if we are in the creation loop or not.
        :type loop: bool


Delete the directory which are not registered into our structure.


Set the paths to the configuration files.

        :param path_to_config: The possible path to the config to load.
        :type path_to_config: str

        :return:
            The path to the config to read (0), the path to the default
            configuration to read as fallback.(1)
        :rtype: tuple


Download `public-suffix.json` if not present.


Download the latest version of `dir_structure_production.json`.


Execute the logic behind the merging.


Convert the versions to a shorter one.

        :param version: The version to split.
        :type version: str

        :param return_non_digits:
            Activate the return of the non-digits parts of the splitted
            version.
        :type return_non_digits: bool

        :return: The splitted version name/numbers.
        :rtype: list


Compare the given versions.

        :param local: The local version converted by split_versions().
        :type local: list

        :param upstream: The upstream version converted by split_versions().
        :type upstream: list

        :return:
            - True: local < upstream
            - None: local == upstream
            - False: local > upstream
        :rtype: bool|None


Let us know if we are currently in the cloned version of
        PyFunceble which implicitly mean that we are in developement mode.


Handle and check that some configuration index exists.


Generate unified file. Understand by that that we use an unified table
        instead of a separate table for each status which could result into a
        misunderstanding.


Generate a file according to the domain status.


Check if we are allowed to produce a file based from the given
        information.

        :return:
            The state of the production.
            True: We do not produce file.
            False: We do produce file.
        :rtype: bool


Implement the standard and alphabetical sorting.

        :param element: The element we are currently reading.
        :type element: str

        :return: The formatted element.
        :rtype: str


The idea behind this method is to sort a list of domain hierarchicaly.

        :param element: The element we are currently reading.
        :type element: str

        :return: The formatted element.
        :rtype: str

        .. note::
            For a domain like :code:`aaa.bbb.ccc.tdl`.

            A normal sorting is done in the following order:
                1. :code:`aaa`
                2. :code:`bbb`
                3. :code:`ccc`
                4. :code:`tdl`

            This method allow the sorting to be done in the following order:
                1. :code:`tdl`
                2. :code:`ccc`
                3. :code:`bbb`
                4. :code:`aaa`


Initiate the IANA database if it is not the case.


Extract the extention from the given block.
        Plus get its referer.


Update the content of the `iana-domains-db` file.


Retrieve the mining informations.


Backup the mined informations.


Remove the currently tested element from the mining
        data.


Provide the list of mined so they can be added to the list
        queue.

        :return: The list of mined domains or URL.
        :rtype: list


Get and return the content of the given log file.

        :param file: The file we have to get the content from.
        :type file: str

        :return The content of the given file.
        :rtype: dict


Write the content into the given file.

        :param content: The dict to write.
        :type content: dict

        :param file: The file to write.
        :type file: str


Logs the case that the referer was not found.

        :param extension: The extension of the domain we are testing.
        :type extension: str


Construct header of the table according to template.

        :param data_to_print:
            The list of data to print into the header of the table.
        :type data_to_print: list

        :param header_separator:
            The separator to use between the table header and our data.
        :type header_separator: str

        :param colomn_separator: The separator to use between each colomns.
        :type colomn_separator: str

        :return: The data to print in a list format.
        :rtype: list


Management and creation of templates of header.
        Please consider as "header" the title of each columns.

        :param do_not_print:
            Tell us if we have to print the header or not.
        :type do_not_print: bool


Construct the table of data according to given size.

        :param size: The maximal length of each string in the table.
        :type size: list

        :return:
            A dict with all information about the data and how to which what
            maximal size to print it.
        :rtype: OrderedDict

        :raises:
            :code:`Exception`
                If the data and the size does not have the same length.


Get the size of each columns from the header.

        :param header:
            The header template we have to get the size from.
        :type header: dict

        :return: The maximal size of the each data to print.
        :rtype: list


Management and input of data to the table.

        :raises:
            :code:`Exception`
                When self.data_to_print is not a list.


Save the current time to the file.

        :param last:
            Tell us if we are at the very end of the file testing.
        :type last: bool


Set the databases files to delete.


Delete almost all discovered files.

        :param clean_all:
            Tell the subsystem if we have to clean everything instesd
            of almost everything.
        :type clean_all: bool


Get hash of the given data.

        :param algo: The algorithm to use.
        :type algo: str


Return the hash of the given file


Remove a given key from a given dictionary.

        :param key_to_remove: The key(s) to delete.
        :type key_to_remove: list|str

        :return: The dict without the given key(s).
        :rtype: dict|None


Rename the given keys from the given dictionary.

        :param key_to_rename:
            The key(s) to rename.
            Expected format: :code:`{old:new}`
        :type key_to_rename: dict

        :param strict:
            Tell us if we have to rename the exact index or
            the index which looks like the given key(s)

        :return: The well formatted dict.
        :rtype: dict|None


Merge the content of to_merge into the given main dictionnary.

        :param to_merge: The dictionnary to merge.
        :type to_merge: dict

        :param strict:
            Tell us if we have to strictly merge lists.

            :code:`True`: We follow index
            :code`False`: We follow element (content)
        :type strict: bool

        :return: The merged dict.
        :rtype: dict


Save a dictionnary into a JSON file.

        :param destination:
            A path to a file where we're going to
            write the converted dict into a JSON format.
        :type destination: str


Fix the path of the given path.

        :param splited_path: A list to convert to the right path.
        :type splited_path: list

        :return: The fixed path.
        :rtype: str


Read a given file path and return its content.

        :return: The content of the given file path.
        :rtype: str


Return a well formatted list. Basicaly, it's sort a list and remove duplicate.

        :return: A sorted, without duplicate, list.
        :rtype: list


Return a list of string which don't match the
        given regex.


Used to get exploitable result of re.search

        :return: The data of the match status.
        :rtype: mixed


Used to replace a matched string with another.

        :return: The data after replacement.
        :rtype: str


Print on screen and on file the percentages for each status.


Check if the given URL is valid.

        :param url: The url to validate.
        :type url: str

        :param return_base:
            Allow us the return of the url base (if URL formatted correctly).
        :type return_formatted: bool

        :param return_formatted:
            Allow us to get the URL converted to IDNA if the conversion
            is activated.
        :type return_formatted: bool

        :return: The validity of the URL or its base.
        :rtype: bool|str


Check if the given subdomain is a subdomain.

        :param domain: The domain to validate.
        :type domain: str

        :return: The validity of the subdomain.
        :rtype: bool


Execute the logic behind the Syntax handling.

        :return: The syntax status.
        :rtype: str


Return the current content of the inactive-db.json file.


Save the current database into the inactive-db.json file.


Get the timestamp where we are going to save our current list.

        :return: The timestamp to append with the currently tested element.
        :rtype: int|str


Get the content of the database.

        :return: The content of the database.
        :rtype: list


Check if the currently tested element is into the database.


Backup the database into its file.


Check if the current time is older than the one in the database.


Implementation of UNIX whois.

        :param whois_server: The WHOIS server to use to get the record.
        :type whois_server: str

        :param domain: The domain to get the whois record from.
        :type domain: str

        :param timeout: The timeout to apply to the request.
        :type timeout: int

        :return: The whois record from the given whois server, if exist.
        :rtype: str|None


Execute the logic behind the URL handling.

        :return: The status of the URL.
        :rtype: str


Return the referer aka the WHOIS server of the current domain extension.


docstring for _randone


Wrapper for Zotero._cleanup


Add a retrieved template to the cache for 304 checking
        accepts a dict and key name, adds the retrieval time, and adds both
        to self.templates as a new dict using the specified key


Remove keys we added for internal use


Return the contents of My Publications


Return the total number of items in the specified collection


Return the total number of items for the specified tag


General method for returning total counts


Retrieve info about the permissions associated with the
        key associated to the given Zotero instance


Get the last modified version


Retrieve all collections and subcollections. Works for top-level collections
        or for a specific collection. Works at all collection depths.


Get subcollections for a specific collection


Retrieve all items in the library for a particular query
        This method will override the 'limit' parameter if it's been set


Return a list of dicts which are dumped CSL JSON


Return a list of strings formatted as HTML citation entries


Get a template for a new item


Create attachments
        accepts a list of one or more attachment template dicts
        and an optional parent Item ID. If this is specified,
        attachments are created under this ID


Delete one or more saved searches by passing a list of one or more
        unique search keys


Add one or more tags to a retrieved item,
        then update it on the server
        Accepts a dict, and one or more tags to add to it
        Returns the updated item from the server


Update an existing item
        Accepts one argument, a dict containing Item data


Update existing items
        Accepts one argument, a list of dicts containing Item data


Validate saved search conditions, raising an error if any contain invalid operators


Split a multiline string into a list, excluding blank lines.


Split a string with comma or space-separated elements into a list.


Evaluate environment markers.


Get configuration value.


Set configuration value.


Compatibility helper to use setup.cfg in setup.py.


Get LanguageTool version.


Get supported languages.


Set LanguageTool directory.


Match text against enabled rules.


Return newest compatible version.

    >>> version = get_newest_possible_languagetool_version()
    >>> version in [JAVA_6_COMPATIBLE_VERSION,
    ...             JAVA_7_COMPATIBLE_VERSION,
    ...             LATEST_VERSION]
    True


Get common directory in a zip file if any.


Make a Qt async slot run on asyncio loop.


Class decorator to add a logger to a class.


Selector has delivered us an event.


Add more ASN.1 MIB source repositories.

        MibCompiler.compile will invoke each of configured source objects
        in order of their addition asking each to fetch MIB module specified
        by name.

        Args:
            sources: reader object(s)

        Returns:
            reference to itself (can be used for call chaining)


Add more transformed MIBs repositories to borrow MIBs from.

        Whenever MibCompiler.compile encounters MIB module which neither of
        the *searchers* can find or fetched ASN.1 MIB module can not be
        parsed (due to syntax errors), these *borrowers* objects will be
        invoked in order of their addition asking each if already transformed
        MIB can be fetched (borrowed).

        Args:
            borrowers: borrower object(s)

        Returns:
            reference to itself (can be used for call chaining)


Get current object.
        This is useful if you want the real
        object behind the proxy at a time for performance reasons or because
        you want to pass the object into a different context.


r"""Kullback information criterion

    .. math:: KIC(k) = log(\rho_k) + 3 \frac{k+1}{N}

    :validation: double checked versus octave.


r"""approximate corrected Kullback information

    .. math:: AKICc(k) = log(rho_k) + \frac{p}{N*(N-k)} + (3-\frac{k+2}{N})*\frac{k+1}{N-k-2}


r"""Final prediction error criterion

    .. math:: FPE(k) = \frac{N + k + 1}{N - k - 1} \rho_k

    :validation: double checked versus octave.


r"""Minimum Description Length

    .. math:: MDL(k) = N log \rho_k + p \log N

    :validation: results


Generate the Main examples gallery reStructuredText

    Start the sphinx-gallery configuration and recursively scan the examples
    directories in order to populate the examples gallery


Setup sphinx-gallery sphinx extension


r"""Correlation function

    This function should give the same results as :func:`xcorr` but it
    returns the positive lags only. Moreover the algorithm does not use
    FFT as compared to other algorithms.

    :param array x: first data array of length N
    :param array y: second data array of length N. If not specified, computes the
        autocorrelation.
    :param int maxlags: compute cross correlation between [0:maxlags]
        when maxlags is not specified, the range of lags is [0:maxlags].
    :param str norm: normalisation in ['biased', 'unbiased', None, 'coeff']

        * *biased*   correlation=raw/N,
        * *unbiased* correlation=raw/(N-`|lag|`)
        * *coeff*    correlation=raw/(rms(x).rms(y))/N
        * None       correlation=raw

    :return:
        * a numpy.array correlation sequence,  r[1,N]
        * a float for the zero-lag correlation,  r[0]

    The *unbiased* correlation has the form:

    .. math::

        \hat{r}_{xx} = \frac{1}{N-m}T \sum_{n=0}^{N-m-1} x[n+m]x^*[n] T

    The *biased* correlation differs by the front factor only:

    .. math::

        \check{r}_{xx} = \frac{1}{N}T \sum_{n=0}^{N-m-1} x[n+m]x^*[n] T

    with :math:`0\leq m\leq N-1`.

    .. doctest::

        >>> from spectrum import CORRELATION
        >>> x = [1,2,3,4,5]
        >>> res = CORRELATION(x,x, maxlags=0, norm='biased')
        >>> res[0]
        11.0

    .. note:: this function should be replaced by :func:`xcorr`.

    .. seealso:: :func:`xcorr`


Cross-correlation using numpy.correlate

    Estimates the cross-correlation (and autocorrelation) sequence of a random
    process of length N. By default, there is no normalisation and the output
    sequence of the cross-correlation has a length 2*N+1.

    :param array x: first data array of length N
    :param array y: second data array of length N. If not specified, computes the
        autocorrelation.
    :param int maxlags: compute cross correlation between [-maxlags:maxlags]
        when maxlags is not specified, the range of lags is [-N+1:N-1].
    :param str option: normalisation in ['biased', 'unbiased', None, 'coeff']

    The true cross-correlation sequence is

    .. math:: r_{xy}[m] = E(x[n+m].y^*[n]) = E(x[n].y^*[n-m])

    However, in practice, only a finite segment of one realization of the
    infinite-length random process is available.

    The correlation is estimated using numpy.correlate(x,y,'full').
    Normalisation is handled by this function using the following cases:

        * 'biased': Biased estimate of the cross-correlation function
        * 'unbiased': Unbiased estimate of the cross-correlation function
        * 'coeff': Normalizes the sequence so the autocorrelations at zero
           lag is 1.0.

    :return:
        * a numpy.array containing the cross-correlation sequence (length 2*N-1)
        * lags vector

    .. note:: If x and y are not the same length, the shorter vector is
        zero-padded to the length of the longer vector.

    .. rubric:: Examples

    .. doctest::

        >>> from spectrum import xcorr
        >>> x = [1,2,3,4,5]
        >>> c, l = xcorr(x,x, maxlags=0, norm='biased')
        >>> c
        array([ 11.])

    .. seealso:: :func:`CORRELATION`.


Finds the minimum eigenvalue of a Hermitian Toeplitz matrix

    The classical power method is used together with a  fast  Toeplitz
    equation solution routine.  The eigenvector is normalized to unit length.

    :param T0: Scalar corresponding to real matrix element t(0)
    :param T: Array of  M  complex matrix elements t(1),...,t(M) C            from the left column of the Toeplitz matrix
    :param TOL: Real scalar tolerance; routine exits when [ EVAL(k) - EVAL(k-1) ]/EVAL(k-1) < TOL , where the index k denotes the iteration number.

    :return:
        * EVAL - Real scalar denoting the minimum eigenvalue of matrix
        * EVEC - Array of  M  complex eigenvector elements associated

    .. note::
     * External array T must be dimensioned >= M
     * array EVEC must be >= M+1
     *  Internal array E must be dimensioned >= M+1 .

     * **dependencies**
        *  :meth:`spectrum.toeplitz.HERMTOEP`


r"""Generate the Morlet waveform

    The Morlet waveform is defined as follows:

    .. math:: w[x] = \cos{5x}  \exp^{-x^2/2}

    :param lb: lower bound
    :param ub: upper bound
    :param int n: waveform data samples

    .. plot::
        :include-source:
        :width: 80%

        from spectrum import morlet
        from pylab import plot
        plot(morlet(0,10,100))


convert reflection coefficients to prediction filter polynomial

    :param k: reflection coefficients


Convert reflection coefficients to log area ratios.

    :param k: reflection coefficients
    :return: inverse sine parameters

    The log area ratio is defined by G = log((1+k)/(1-k)) , where the K
    parameter is the reflection coefficient.

    .. seealso:: :func:`lar2rc`, :func:`rc2poly`, :func:`rc2ac`, :func:`rc2ic`.

    :References:
       [1] J. Makhoul, "Linear Prediction: A Tutorial Review," Proc. IEEE, Vol.63, No.4, pp.561-580, Apr 1975.


Convert log area ratios to reflection coefficients.

    :param g:  log area ratios
    :returns: the reflection coefficients

    .. seealso: :func:`rc2lar`, :func:`poly2rc`, :func:`ac2rc`, :func:`is2rc`.

    :References:
       [1] J. Makhoul, "Linear Prediction: A Tutorial Review," Proc. IEEE,  Vol.63, No.4, pp.561-580, Apr 1975.


Convert line spectral frequencies to prediction filter coefficients

    returns a vector a containing the prediction filter coefficients from a vector lsf of line spectral frequencies.

    .. doctest::

        >>> from spectrum import lsf2poly
        >>> lsf = [0.7842 ,   1.5605  ,  1.8776 ,   1.8984,    2.3593]
        >>> a = lsf2poly(lsf)

    # array([  1.00000000e+00,   6.14837835e-01,   9.89884967e-01,
    # 9.31594056e-05,   3.13713832e-03,  -8.12002261e-03 ])

    .. seealso:: poly2lsf, rc2poly, ac2poly, rc2is


Prediction polynomial to line spectral frequencies.

    converts the prediction polynomial specified by A,
    into the corresponding line spectral frequencies, LSF.
    normalizes the prediction polynomial by A(1).

    .. doctest::

        >>> from spectrum import poly2lsf
        >>> a = [1.0000,  0.6149, 0.9899, 0.0000 ,0.0031, -0.0082]
        >>> lsf = poly2lsf(a)
        >>> lsf =  array([0.7842, 1.5605, 1.8776, 1.8984, 2.3593])

    .. seealso:: lsf2poly, poly2rc, poly2qc, rc2is


Convert a one-sided PSD to a twosided PSD

    In order to keep the power in the onesided PSD the same
    as in the twosided version, the onesided values are twice
    as much as in the input data (except for the zero-lag value).

    ::

        >>> twosided_2_onesided([10, 2,3,3,2,8])
        array([ 10.,   4.,   6.,   8.])


Convert a two-sided PSD to a one-sided PSD

    In order to keep the power in the twosided PSD the same
    as in the onesided version, the twosided values are 2 times
    lower than the input data (except for the zero-lag and N-lag
    values).

    ::

        >>> twosided_2_onesided([10, 4, 6, 8])
        array([ 10.,   2.,   3.,   3., 2., 8.])


Convert a two-sided PSD to a center-dc PSD


Convert a center-dc PSD to a twosided PSD


A simple test example with two close frequencies


Plot the data set, using the sampling information to set the x-axis
        correctly.


Returns the autocovariance of signal s at all lags.

    Adheres to the definition
    sxx[k] = E{S[n]S[n+k]} = cov{S[n],S[n+k]}
    where E{} is the expectation operator, and S is a zero mean process


Separate `filename` content between docstring and the rest

    Strongly inspired from ast.get_docstring.

    Returns
    -------
    docstring: str
        docstring of `filename`
    rest: str
        `filename` content without the docstring


Returns md5sum of file


Returns True if src_file has a different md5sum


Test existence of image file and no change in md5sum of
    example


Save all open matplotlib figures of the example code-block

    Parameters
    ----------
    image_path : str
        Path where plots are saved (format string which accepts figure number)
    fig_count : int
        Previous figure number count. Figure number add from this number

    Returns
    -------
    list of strings containing the full path to each figure


Save the thumbnail image


Executes the code block of the example file


This function solve Ax=B directly without taking care of the input
    matrix properties.


Simple periodogram, but matrices accepted.

    :param x: an array or matrix of data samples.
    :param NFFT: length of the data before FFT is computed (zero padding)
    :param bool detrend: detrend the data before co,puteing the FFT
    :param float sampling: sampling frequency of the input :attr:`data`.

    :param scale_by_freq:
    :param str window:

    :return: 2-sided PSD if complex data, 1-sided if real.

    if a matrix is provided (using numpy.matrix), then a periodogram
    is computed for each row. The returned matrix has the same shape as the input
    matrix.

    The mean of the input data is also removed from the data before computing
    the psd.

    .. plot::
        :width: 80%
        :include-source:

        from pylab import grid, semilogy
        from spectrum import data_cosine, speriodogram
        data = data_cosine(N=1024, A=0.1, sampling=1024, freq=200)
        semilogy(speriodogram(data, detrend=False, sampling=1024), marker='o')
        grid(True)

    .. plot::
        :width: 80%
        :include-source:

        import numpy
        from spectrum import speriodogram, data_cosine
        from pylab import figure, semilogy, figure ,imshow
        # create N data sets and make the frequency dependent on the time
        N = 100
        m = numpy.concatenate([data_cosine(N=1024, A=0.1, sampling=1024, freq=x)
            for x in range(1, N)]);
        m.resize(N, 1024)
        res = speriodogram(m)
        figure(1)
        semilogy(res)
        figure(2)
        imshow(res.transpose(), aspect='auto')

    .. todo:: a proper spectrogram class/function that takes care of normalisation


r"""Simple periodogram wrapper of numpy.psd function.

    :param A: the input data
    :param int NFFT: total length of the final data sets (padded
        with zero if needed; default is 4096)
    :param str window:

    :Technical documentation:

    When we calculate the periodogram of a set of data we get an estimation
    of the spectral density. In fact as we use a Fourier transform and a
    truncated segments the spectrum is the convolution of the data with a
    rectangular window which Fourier transform is

    .. math::

        W(s)= \frac{1}{N^2} \left[ \frac{\sin(\pi s)}{\sin(\pi s/N)} \right]^2

    Thus oscillations and sidelobes appears around the main frequency. One aim of t he tapering is to reduced this effects. We multiply data by a window whose  sidelobes are much smaller than the main lobe. Classical window is hanning window.  But other windows are available. However we must take into account this energy and divide the spectrum by energy of taper used. Thus periodogram becomes :

    .. math::

        D_k \equiv \sum_{j=0}^{N-1}c_jw_j \; e^{2\pi ijk/N}  \qquad k=0,...,N-1

    .. math::

        P(0)=P(f_0)=\frac{1}{2\pi W_{ss}}\arrowvert{D_0}\arrowvert^2

    .. math::

        P(f_k)=\frac{1}{2\pi W_{ss}} \left[\arrowvert{D_k}\arrowvert^2+\arrowvert{D_{N-k}}\arrowvert^2\right]        \qquad k=0,1,...,     \left( \frac{1}{2}-1 \right)

    .. math::

        P(f_c)=P(f_{N/2})= \frac{1}{2\pi W_{ss}} \arrowvert{D_{N/2}}\arrowvert^2

    with

    .. math::

        {W_{ss}} \equiv N\sum_{j=0}^{N-1}w_j^2

    .. plot::
        :width: 80%
        :include-source:

        from spectrum import WelchPeriodogram, marple_data
        psd = WelchPeriodogram(marple_data, 256)


Return the centered frequency range as a generator.

        ::

            >>> print(list(Range(8).centerdc_gen()))
            [-0.5, -0.375, -0.25, -0.125, 0.0, 0.125, 0.25, 0.375]


Return the one-sided frequency range as a generator.

        If :attr:`N` is even, the length is N/2 + 1.
        If :attr:`N` is odd, the length is (N+1)/2.

        ::

            >>> print(list(Range(8).onesided()))
            [0.0, 0.125, 0.25, 0.375, 0.5]
            >>> print(list(Range(9).onesided()))
            [0.0, 0.1111, 0.2222, 0.3333, 0.4444]


r"""Return the power contained in the PSD

        if scale_by_freq is False, the power is:

        .. math:: P = N \sum_{k=1}^{N} P_{xx}(k)

        else, it is

        .. math:: P =  \sum_{k=1}^{N} P_{xx}(k) \frac{df}{2\pi}

        .. todo:: check these equations


Returns a dictionary with the elements of a Jupyter notebook


Converts the RST text from the examples docstrigs and comments
    into markdown text for the IPython notebooks


Saves the notebook to a file


Autoregressive and moving average estimators.

    This function provides an estimate of the autoregressive
    parameters, the moving average parameters, and the driving
    white noise variance of  an ARMA(P,Q) for a complex or real data sequence.

    The parameters are estimated using three steps:

        * Estimate the AR parameters from the original data based on a least
          squares modified Yule-Walker technique,
        * Produce a residual time sequence by filtering the original data
          with a filter based on the AR parameters,
        * Estimate the MA parameters from the residual time sequence.

    :param array X: Array of data samples (length N)
    :param int P: Desired number of AR parameters
    :param int Q: Desired number of MA parameters
    :param int lag: Maximum lag to use for autocorrelation estimates

    :return:
        * A     - Array of complex P AR parameter estimates
        * B     - Array of complex Q MA parameter estimates
        * RHO   - White noise variance estimate

    .. note::
      *  lag must be >= Q (MA order)

    **dependencies**:
        * :meth:`spectrum.correlation.CORRELATION`
        * :meth:`spectrum.covar.arcovar`
        * :meth:`spectrum.arma.ma`

    .. plot::
        :width: 80%
        :include-source:

        from spectrum import arma_estimate, arma2psd, marple_data
        import pylab

        a,b, rho = arma_estimate(marple_data, 15, 15, 30)
        psd = arma2psd(A=a, B=b, rho=rho, sides='centerdc', norm=True)
        pylab.plot(10 * pylab.log10(psd))
        pylab.ylim([-50,0])

    :reference: [Marple]_


Moving average estimator.

    This program provides an estimate of the moving average parameters
    and driving noise variance for a data sequence based on a
    long AR model and a least squares fit.

    :param array X: The input data array
    :param int Q: Desired MA model order (must be >0 and <M)
    :param int M: Order of "long" AR model (suggest at least 2*Q )

    :return:
        * MA    - Array of Q complex MA parameter estimates
        * RHO   - Real scalar of white noise variance estimate

    .. plot::
        :width: 80%
        :include-source:

        from spectrum import arma2psd, ma, marple_data
        import pylab

        # Estimate 15 Ma parameters
        b, rho = ma(marple_data, 15, 30)
        # Create the PSD from those MA parameters
        psd = arma2psd(B=b, rho=rho, sides='centerdc')
        # and finally plot the PSD
        pylab.plot(pylab.linspace(-0.5, 0.5, 4096), 10 * pylab.log10(psd/max(psd)))
        pylab.axis([-0.5, 0.5, -30, 0])

    :reference: [Marple]_


PSD estimate using correlogram method.

    :param array X: complex or real data samples X(1) to X(N)
    :param array Y: complex data samples Y(1) to Y(N). If provided, computes
        the cross PSD, otherwise the PSD is returned
    :param int lag: highest lag index to compute. Must be less than N
    :param str window_name: see :mod:`window` for list of valid names
    :param str norm: one of the valid normalisation of :func:`xcorr` (biased,
        unbiased, coeff, None)
    :param int NFFT: total length of the final data sets (padded with zero
        if needed; default is 4096)
    :param str correlation_method: either `xcorr` or `CORRELATION`.
        CORRELATION should be removed in the future.

    :return:
        * Array of real (cross) power spectral density estimate values. This is
          a two sided array with negative values following the positive ones
          whatever is the input data (real or complex).

    .. rubric:: Description:

    The exact power spectral density is the Fourier transform of the
    autocorrelation sequence:

    .. math:: P_{xx}(f) = T \sum_{m=-\infty}^{\infty} r_{xx}[m] exp^{-j2\pi fmT}

    The correlogram method of PSD estimation substitutes a finite sequence of
    autocorrelation estimates :math:`\hat{r}_{xx}` in place of :math:`r_{xx}`.
    This estimation can be computed with :func:`xcorr` or :func:`CORRELATION` by
    chosing a proprer lag `L`. The estimated PSD is then

    .. math:: \hat{P}_{xx}(f) = T \sum_{m=-L}^{L} \hat{r}_{xx}[m] exp^{-j2\pi fmT}

    The lag index must be less than the number of data samples `N`. Ideally, it
    should be around `L/10` [Marple]_ so as to avoid greater statistical
    variance associated with higher lags.

    To reduce the leakage of the implicit rectangular window and therefore to
    reduce the bias in the estimate, a tapering window is normally used and lead
    to the so-called Blackman and Tukey correlogram:

    .. math:: \hat{P}_{BT}(f) = T \sum_{m=-L}^{L} w[m] \hat{r}_{xx}[m] exp^{-j2\pi fmT}

    The correlogram for the cross power spectral estimate is

    .. math:: \hat{P}_{xx}(f) = T \sum_{m=-L}^{L} \hat{r}_{xx}[m] exp^{-j2\pi fmT}

    which is computed if :attr:`Y` is not provide. In such case,
    :math:`r_{yx} = r_{xy}` so we compute the correlation only once.

    .. plot::
        :width: 80%
        :include-source:

        from spectrum import CORRELOGRAMPSD, marple_data
        from spectrum.tools import cshift
        from pylab import log10, axis, grid, plot,linspace

        psd = CORRELOGRAMPSD(marple_data, marple_data, lag=15)
        f = linspace(-0.5, 0.5, len(psd))
        psd = cshift(psd, len(psd)/2)
        plot(f, 10*log10(psd/max(psd)))
        axis([-0.5,0.5,-50,0])
        grid(True)

    .. seealso:: :func:`create_window`, :func:`CORRELATION`, :func:`xcorr`,
        :class:`pcorrelogram`.


Select first block delimited by start_tag and end_tag


Parse a dictionary from the search index


Parse a Sphinx search index

    Parameters
    ----------
    searchindex : str
        The Sphinx search index (contents of searchindex.js)

    Returns
    -------
    filenames : list of str
        The file names parsed from the search index.
    objects : dict
        The objects parsed from the search index.


Get a valid link, False if not found


r"""Return polynomial transfer function representation from zeros and poles

    :param ndarray z: Zeros of the transfer function.
    :param ndarray p: Poles of the transfer function.
    :param float k: System gain.

    :return:
        b : ndarray Numerator polynomial.
        a : ndarray Numerator and denominator polynomials.

    :func:`zpk2tf` forms transfer function polynomials from the zeros, poles, and gains
    of a system in factored form.

    zpk2tf(z,p,k) finds a rational transfer function

    .. math:: \frac{B(s)}{A(s)} = \frac{b_1 s^{n-1}+\dots b_{n-1}s+b_n}{a_1 s^{m-1}+\dots a_{m-1}s+a_m}

    given a system in factored transfer function form

    .. math:: H(s) = \frac{Z(s)}{P(s)} = k \frac{(s-z_1)(s-z_2)\dots(s-z_m)}{(s-p_1)(s-p_2)\dots(s-p_n)}

    with p being the pole locations, and z the zero locations, with as many.
    The gains for each numerator transfer function are in vector k.
    The zeros and poles must be real or come in complex conjugate pairs.
    The polynomial denominator coefficients are returned in row vector a and
    the polynomial numerator coefficients are returned in matrix b, which has
    as many rows as there are columns of z.

    Inf values can be used as place holders in z if some columns have fewer zeros than others.

    .. note:: wrapper of scipy function zpk2tf


Zero-pole-gain representation to state-space representation

    :param sequence z,p: Zeros and poles.
    :param float k: System gain.

    :return:
        * A, B, C, D : ndarray State-space matrices.

    .. note:: wrapper of scipy function zpk2ss


A Window visualisation tool

    :param N: length of the window
    :param name: name of the window
    :param NFFT: padding used by the FFT
    :param mindB: the minimum frequency power in dB
    :param maxdB: the maximum frequency power in dB
    :param kargs: optional arguments passed to :func:`create_window`

    This function plot the window shape and its equivalent in the Fourier domain.

    .. plot::
        :width: 80%
        :include-source:

        from spectrum import window_visu
        window_visu(64, 'kaiser', beta=8.)


r"""Kaiser window

    :param N: window length
    :param beta: kaiser parameter (default is 8.6)

    To obtain a Kaiser window that designs an FIR filter with
    sidelobe attenuation of :math:`\alpha` dB, use the following :math:`\beta` where
    :math:`\beta = \pi \alpha`.

    .. math::

        w_n = \frac{I_0\left(\pi\alpha\sqrt{1-\left(\frac{2n}{M}-1\right)^2}\right)} {I_0(\pi \alpha)}

    where

      * :math:`I_0` is the zeroth order Modified Bessel function of the first kind.
      * :math:`\alpha` is a real number that determines the shape of the
        window. It determines the trade-off between main-lobe width and side
        lobe level.
      * the length of the sequence is N=M+1.

    The Kaiser window can approximate many other windows by varying
    the :math:`\beta` parameter:

    ===== ========================
    beta  Window shape
    ===== ========================
    0     Rectangular
    5     Similar to a Hamming
    6     Similar to a Hanning
    8.6   Similar to a Blackman
    ===== ========================

    .. plot::
        :width: 80%
        :include-source:

        from pylab import plot, legend, xlim
        from spectrum import window_kaiser
        N = 64
        for beta in [1,2,4,8,16]:
            plot(window_kaiser(N, beta), label='beta='+str(beta))
        xlim(0,N)
        legend()

    .. plot::
        :width: 80%
        :include-source:

        from spectrum import window_visu
        window_visu(64, 'kaiser', beta=8.)

    .. seealso:: numpy.kaiser, :func:`spectrum.window.create_window`