Skip to content

Entropy

entropy 

entropy(
    d: List[float], in_bits=False, miditoolbox_compatible: bool = False
) -> float

Calculate the relative entropy of a distribution.

This function computes the relative entropy (also known as normalized entropy) of a given distribution d. The result is a value between 0 and 1, where 0 indicates no uncertainty (all probability mass is concentrated in one outcome) and 1 indicates maximum uncertainty (uniform distribution).

If in_bits is set to True, the result will be in bits (base 2 logarithm) instead of relative entropy. The in_bits result gives the expected number of bits needed to encode an outcome drawn from the distribution.

The distribution d will be normalized if it does not already sum to 1. If d is all zeros, the function will return 1.0 to avoid division by zero. If miditoolbox_compatible is set to True, division by zero is avoided by adding a small constant to the distribution, which will slightly alter the result. (This is to maintain compatibility with the original MIDI Toolbox implementation.)

Parameters:

  • d (List[float]) –

    The input distribution.

  • in_bits (bool, default: False ) –

    If True, returns unnormalized entropy in bits (base 2 logarithm). Default is False: returns relative entropy (0 <= H <= 1).

  • miditoolbox_compatible (bool, default: False ) –

    If True, uses the original MIDI Toolbox method of calculation. Default is False.

Returns:

  • float –

    The relative entropy (0 <= H <= 1) unless in_bits is True, in which case the unnormalized entropy in bits is returned.

Notes

Implementation based on the original MATLAB code from: https://github.com/miditoolbox/1.1/blob/master/miditoolbox/entropy.m

Examples:

Entropy is maximized when all outcomes are equally likely:

>>> entropy([0.5, 0.5])
1.0

Entropy is minimized when one outcome is certain:

>>> entropy([0.0, 1.0])
0.0
Source code in amads/algorithms/entropy.py 
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
def entropy(
    d: List[float], in_bits=False, miditoolbox_compatible: bool = False
) -> float:
    """
    Calculate the relative entropy of a distribution.

    This function computes the relative entropy (also known as normalized
    entropy) of a given distribution `d`. The result is a value between 0
    and 1, where 0 indicates no uncertainty (all probability mass is
    concentrated in one outcome) and 1 indicates maximum uncertainty
    (uniform distribution).

    If `in_bits` is set to True, the result will be in bits
    (base 2 logarithm) instead of relative entropy. The `in_bits`
    result gives the expected number of bits needed to encode an outcome
    drawn from the distribution.

    The distribution `d` will be normalized if it does not already sum
    to 1. If `d` is all zeros, the function will return 1.0 to avoid
    division by zero. If `miditoolbox_compatible` is set to True, division
    by zero is avoided by adding a small constant to the distribution,
    which will slightly alter the result. (This is to maintain compatibility
    with the original MIDI Toolbox implementation.)

    Parameters
    ----------
    d : List[float]
        The input distribution.

    in_bits : bool, optional
        If True, returns unnormalized entropy in bits (base 2 logarithm).
        Default is False: returns relative entropy (0 <= H <= 1).

    miditoolbox_compatible : bool, optional
        If True, uses the original MIDI Toolbox method of calculation.
        Default is False.

    Returns
    -------
    float
        The relative entropy (0 <= H <= 1) unless `in_bits` is True,
        in which case the unnormalized entropy in bits is returned.

    Notes
    -----
    Implementation based on the original MATLAB code from:
    https://github.com/miditoolbox/1.1/blob/master/miditoolbox/entropy.m

    Examples
    --------
    Entropy is maximized when all outcomes are equally likely:
    >>> entropy([0.5, 0.5])
    1.0

    Entropy is minimized when one outcome is certain:
    >>> entropy([0.0, 1.0])
    0.0
    """
    darray = np.asarray(d).flatten()  # Convert to a 1D numpy array
    sum = np.sum(darray)
    if miditoolbox_compatible:
        sum += 1e-12  # Avoid division by zero
    elif sum == 0:
        return 1.0  # Avoid division by zero; return maximum entropy
    darray = darray / sum  # Normalize
    if miditoolbox_compatible:
        logd = np.log(darray + 1e-12)  # Avoid log(0)
    else:
        logd = np.where(
            darray > 0, np.log(darray), 0
        )  # Compute log(d), treating log(0) as 0
    h = -np.sum(darray * logd)
    if in_bits:
        h = h / np.log(2)  # Unnormalized entropy in bits
    else:
        h = h / np.log(len(darray))  # Normalize to relative entropy
    return float(h)