"Truth is not always in a well. In fact, as regards the more important knowledge, I do believe that she is invariably superficial. The depth lies in the valleys where we seek her, and not upon the mountain-tops where she is found."

"By undue profundity we perplex and enfeeble thought; and it is possible to make even Venus herself vanish from the firmament by a scrutiny too sustained, too concentrated, or too direct."

"It will be found, in fact, that the ingenious are always fanciful, and the truly imaginative never otherwise than analytic."

— Edgar Allan Poe, The Murders in the Rue Morgue (1841)

Be like a sniffer dog

Not a fun fact: in 2017, a violent crime occurred every 24.6 seconds according to the FBI Crime Clock [1]. This is the kind of statistics found "upon the mountain-tops", by collecting mountains and mountains of daily crime data never ending. And the final result is reduced to this curious snapshot, like a nightmarish memento mori, namely the "FBI Crime Clock": time is money, but danger lurks around every corner... A travel advisory issued by your embassy about a failed state would not have been more explicit. Do Not Travel! Reconsider!

It is possible to make even the Six Grandfathers themselves vanish from the firmament of Mount Rushmore by such a sustained scrutiny into war on crime. Terrific decimal: 24.6. Better to never look at the watch, in fact. Nobody can breathe with such a number in mind. And there are a lot of records equally disturbing: 30.5 minutes (murders), 3.9 minutes (rapes), 1.7 minutes (robbery), etc. Ignorance of time is bliss.

Another quote from the same authority:

The 2017 violent crime rate was 382.9 per 100,000 inhabitants, down 0.9 percent when compared with the 2016 violent crime rate. [2]

How do you feel? Better, right?

Change the scale and you tell another story.

The truth that matters, the more important knowledge, says the detective Dupin-Poe, is invariably superficial, and lies in the valleys where we seek her. If not, we perplex and enfeeble thought. The Bostonian expert in solving enigmas is pro-Occam's razor. No need to undue profundity. Since "The Purloined letter" is hidden in plain sight [3], why should I look in a well? "Show the data!", warms Jeef Leek [4].

So, this is a story of peaks and valleys, like in a Dow Jones chart. "Pictures are undervalued in science", wrote Benoît Mandelbrot in his excellent book The (Mis)Behavior of Markets. We agree with the French mathematician. We believe that pictures, good pictures and not ridiculograms [5], are the royal road to epiphanies.

In this study, charts helped us a lot. Despite the noisy data, an unexpected pattern was showing up insistently from the very start:like temperature or influenza, crime is seasonal. Peak fever usually occurs in summer. Evidence pop up all over the data. Maybe it will be found, to paraphrase again Edgar Allan Poe, that data analysts are never otherwise than sniffer dogs following a promising track.

First clues of seasonality and lockdown effect

Seasonality is a classic pattern in criminology though still without being well understood by academics. The subject has been studied for over a century, starting with Quetelet [6], Mister "Average Man", who was an ingenuous believer, like almost all the scientists of his time (19th century), in "natural" or inborn biological causes behind deviance (climate and/or other stupid factors like race, heredity, etc.). In his (narrow) mind, statistics will save the world and the Average Man will replace the old imperfect man great-grandpa was. Here we are. More domestic disputes in summer, why? Maybe the sun making new highs removes some unpleasant ancestral barriers. As a result, the Average Man deviates from the boring statistical mean and the standard deviation of his expected human behavior turns crazy... Maybe. Quetelet's claim: "the violence of the passions predominating in summer, excites to more frequent personal collisions". In other no mother tongue terms: the passions predominating in his brain (whatever the season), excites Quetelet to very frequent confusions between correlation and causation.

More seriously, as mentioned by Dong et al. (2017) [7]:

Considering one of the most noticeable consequences of global warming is a shift in weather patterns, either locally warming or cooling, there is a need to understand how crime rates may change in response to the new weather patterns.

Less water in some countries, for example, and I bet that the Average Man will shift too. He will deviate to survive and become unpredictable. Welcome to the Extremistan Man. The extremistan studies are born. Bye-bye Quetelet and Mediocristan! Easy.

Fig. 1. — Crime Incident Reports overview. While the total number of incidents in Boston rarely exceeds 300 events on a single day (top left), following what looks like a perfect Gaussian distribution, the raw daily data is obscured by seasonal oscillations and a certain amount of stochasticity (bottom left). Note the sudden drop on bottom left: it occurs right after the State of Emergency is declared on March 23, 2020. When the Reopening is announced on May 18, the number of daily incidents increases first slowly then quickly a few days after, hitting a local maximum on June 1.

So this is not very surprising whether the BPD Crime Incident Reports show seasonal variations though obscured by noisy data (bottom left, Fig. 1). We need to build a tool to extract the long-term trend and not be fooled by randomness.

If we look carefully at the same bottom panel on left in Fig. 1, a substantial hole is drawn at the end of the series. The phenomenon starts in fact shortly before the State of Emergency is declared on March 23, 2020 by Governor Baker, and ends after the Reopening announcement [8] on May 18: this is what we call the lockdown effect (Fig. 2), i.e. an abnormal drop in the number of incidents due to the major part of the population being under strict control, empty streets and shops closed.

After the economy reopen, the crime quickly reopened too, and climaxed on June 1 for then coming back to its old routine made of sudden peaks and drops – the seasonal oscillations.

We will show infra that the June 1 spike is most likely related to George Floyd protests maybe not so peaceful but running hot in response to police brutality.

From this point of view, crime – crimes against property more precisely – is nothing but a kind of thermometer. You can use it for analysis purposes or ignore it, the thermometer says that burglary, auto-theft and vandalism have been the best riots indicators during George Floyd protests (Fig. 6). Rinse, repeat: the data shows that these people didn't target persons, but property, at least in Boston. This is why we need good data science. Because truth matters. Don't be afraid to tell the truth. She is often an outlier, a surprising and beautiful outlier. In other words, truth is a "dramatic change" in the data. The truth is that George Floyd died under the ruthless knee of a white police officer.

[...] social catastrophes can override any seasonality and lead to dramatic changes in future crime rates. (Dong et al., 2017)

That's it.

So, please, be careful when you remove outliers!

Fig. 2. — Zoom on lockdown effect. A significant drop in crime rate occurs during the lockdown in Boston (orange shape). The short-term oscillations (seasonality) are not immune to brutal social changes. We'll show infra that the spike occurring on June 1 is due to George Floyd protests.

After plotting a 2D heatmap (Fig. 3), we see even better the seasonal pattern. The color interpolation indicates a three/four-month range where incidents are reported at a higher frequency.

Next step: remove irrelevant offense types from the data.

Fig. 3. — Daily incidents heatmap. The data have been reorganized in an array where each cell is the recipient of the total count of incidents that occurred a given weekday of a given month of the year. The darkest zone around the middle of the figure indicates that the number of incidents is usually highest between June and September.

Size doesn't matter: all we need is love and good data

The right panel in Fig. 1 shows also that offense types distribution is heavily unbalanced, exhibiting a large fat tail typical of a Pareto distribution skewed to the right. As a matter of fact, 10,5% of offense types account for 72,4% of the incidents, which is very close to the 80/20 rule [9].

Pareto (work on the 20% that brings the 80%) is not only about prioritization.

It’s also about avoiding working on spurious coincidences - the noise that caused that small correlation which won’t repeat.
— Luca Dellanna (@DellAnnaLuca) October 1, 2020

Therefore, if we want to evaluate correctly the success —or the failure— of the public safety policy at Boston, it would be worth focusing only on the most relevant offense categories: those ones belonging to the said "Crime Index".

[...] they are the crimes most likely to be reported and most likely to occur with sufficient frequency to provide an adequate basis for comparison. [...] The offenses included in the Crime Index total are the violent crimes of Murder and Nonnegligent Manslaughter, Forcible Rape, Robbery, and Aggravated Assault, and the property crimes of Burglary, Larceny-theft, and Motor Vehicle Theft. Arson is excluded due to inconsistent reporting. [10]

By reducing our sample size and selecting carefully the offense types, our goal is to remove bias in the data collected by agents potentially involved in a race for results ("get those numbers and come back with them"), which may lead to serious misconducts [11].

The UCR Part I crimes [12], or Crime index, are divided into two categories: violent crimes (crime against persons) and property crimes (crime against property):

Violent Crimes	Property Crimes
ASSAULT - AGGRAVATED	LARCENY
MURDER	AUTO THEFT
ROBBERY	ARSON
RAPE	BURGLARY

As mentioned above, Arson offenses have been excluded from the Index Crimes due to inconsistent reporting. But we are also forced to exclude the rapes from this study due to inconsistency in the BPD dataset regarding sexual assault (change in law).

Fig. 4. — The most serious crimes. At first glance, Black Friday sales for crime shopping occurs in July. Spot seven differences between the left and the right heatmap is certainly a tough game. Sorry, no prize for the winner! It means that the weight of property crimes far exceeds the weight of violent crimes.

The Part I Crimes as well as the property crimes heatmaps (Fig. 4) look like a reduced snapshot of Fig. 3.

The same peak in (late) summer is still showing up despite we have strongly narrowed the data: the seasonality pattern is strong enough to be sample reduction resistant.

Does it mean that the most serious crimes are something —somewhere in the time or in the space— predictable? The way to reduce the crime has been factorized by Lewitt (2004) [13], it is a significant contribution though controversial. But the empire of crime is a complex space. Like a black hole with halo effect: easy to fall into, hard to quit without stigma for life regardless the extent of the criminal record [14]. And we have now examples of Minority Report emulators harassing innocent people based on big data and machine learning predictions [15]. The fact is that more and more studies are finding hidden bias in the data used during decades for training algorithms that finally are malicious enough to discretely discriminates minorities [16].

However, the violent crimes rate is not converging so well toward a clear seasonal peak, as if spotting the crimes against persons was in fact the real challenge here.

Is Boston doing better or worse than expected compared to other American cities?

By performing SSA, we will be able to answer the question in the next section.

The trajectory space of the crime

We start by quoting again Dong et al. (2017), our main source of inspiration for this work:

The first step in the SSA involves creating a trajectory matrix. For a time series $X = (x_1$, $x_2$,...,$x_N)$ and a window length $L = 365$, we define the trajectory matrix as:

$$ {\bf X} = \begin{pmatrix} x_1 & x_2 & x_3 & \cdots & x_K\\ x_2 & x_3 & x_4 & \cdots & x_{K+1}\\ x_3 & x_4 & x_5 & \cdots & x_{K+2}\\ \vdots & \vdots & \vdots & \ddots & \vdots\\ x_L & x_{L+1} & x_{L+2} & \vdots & x_N\\ \end{pmatrix} $$

with $K = N - L + 1$. Then a singular value decomposition (SVD) is performed on the trajectory matrix, providing a decomposition $X = X_1 + ⋯ + X_L$. Each of our modes then comes from averaging these matrices $X_i$ along their anti-diagonal. The mode with the largest singular value always corresponds to the average crime trend. We then typically would look for any modes with an annual periodicity in the first twenty singular values and sum these to produce our seasonal component.

Since we work under two constraints, time $t \ne \infty$ and budget $b\approx0$, we don't go into detail. For those interested in learning more about SSA, see [9] and [10].

Our purpose is to apply the SSA technique to Boston following [4] guidelines.

Fig. 5. — The most serious crimes with their SSA components. Daily aggregates are plotted in grey, long-term trends in yellow, and the green curves are the long-term trends plus the first two seasonal components (ET1-3). See the big outlier on the right?

Recap:

SSA technique makes an additive decomposition of a time series $X$ of length $N$ (the diagonal of the trajectory matrix) into a sum of $L$ components of length $N$. To reconstruct the time series, simply sum the $L$ components and make a sanity check.
The mode with the largest values, i.e. the first elementary reconstructed series or first eigentriple (ET1), is the long-term trend.
We sum the first other modes with annual periodicity to produce the seasonal component.

Long-term trends extraction (the signal)

The Index Crimes in Boston are slowly falling accordingly to the national tendency [11], but this is due above all to a sharp decline in property crimes frequency, while violent crimes long-term trend is flat as a hell (Fig. 5). Due to very low amplitude, annual variations for violent crimes would be almost impossible to detect without the SSA technique.

Back to the original dataset. Three offense types exhibit the same outlier (Fig. 6):

BURGLARY - COMMERICAL (sic)
AUTO THEFT
VANDALISM

Quick research confirms that George Floyd protests in Boston streets on May 31 have ended in violence [12]. The many incidents have been reported by BDP one day later, on June 1, resulting in the outlier in the data.

We will now emphasize both violent crimes and property crimes long trends by zooming on their respective subcategories.

Fig. 6. — Boston protests against George Floyd killing ending in violence. Commercial burglary, auto theft and vandalism exhibit the same spike on June 1, showing that these three offense types are the best markers for riots and looting.

Though we observe a general decrease for Part I Crimes (Fig. 5), the aggravated assaults are slightly on the rise and auto thefts have ceased to decrease after 2019, while the rate of decrease for burglary seems to be slowing (Fig. 7). As mentioned by [4]:

One possible cause is that crime is at very low basal level, and it is hard to decrease it below this level.

<ipython-input-22-8788afb0e16c>:27: UserWarning: FixedFormatter should only be used together with FixedLocator
  ax[i].set_yticklabels(["",0.6,0.8,1.,1.2,""])
<ipython-input-22-8788afb0e16c>:14: UserWarning: FixedFormatter should only be used together with FixedLocator
  ax[1].set_xticklabels(years)

Fig. 7. — Long-term trends by subcategories. Following Donet et al. (2017), the data has been normalized "to be the fraction of the initial crime level for the first time point $C_0$". The Murder trend is pretty erratic, but keep in mind that homicides occur very rarely (less than 1 by day on average), so the variance is higher.

This is a paradox: once reached very low levels of crime (proof of agents' good work), it becomes harder to lower the curve, and more likely to get bad results in the future (not proof of agents' bad work), losing therefore public funds. From this point of view, a public safety policy based only on quantified results is suboptimal and unfair to the agents, if not stressful for them (incentivizing the dangerous "race for results" pointed supra).

Seasonal components extraction (the noise)

To reconstruct the seasonal data, we will use the first $n$ leading SSA order modes, where $2 \leq n \leq4$, depending on the offense type, each one having a different periodicity.

Again, we have followed the procedure described in [4]:

The trend is subtracted from the data and then the data is smoothed using a moving window average. As the oscillations are related to the total amount of crime, the data and seasonal component have been divided by the trend pointwise in time to normalize the oscillations.

Since our long-term window length $L = 365$, we have smoothed the data with a monthly window average of length $l = 30$ before normalizing the oscillations.

Fig. 8. — Seasonal crime component. Even if the raw data has been smoothed, we see that the seasonal oscillations are fairly consistent, especially for aggravated assault, larceny and auto theft. In other words, each crime index type has a seasonal component. A large aberration appears for burglary during George Floyd protests, while a significant drop happens for robbery category during the lockdown, showing that robbery is the crime type that has been the most affected by State of Emergency.

Fig. 9. — Differences in seasonality.

References

[1] FBI Crime Clock: https://ucr.fbi.gov/crime-in-the-u.s/2017/crime-in-the-u.s.-2017/topic-pages/crime-clock
[2] Cf: https://ucr.fbi.gov/crime-in-the-u.s/2017/crime-in-the-u.s.-2017/topic-pages/tables/table-1
[3] Wikipedia - The Purloined Letter: https://en.wikipedia.org/wiki/The_Purloined_Letter
[4] Prudent analyst: http://jtleek.com/ads2020/week-5.html
[5] Introduction to ridiculogram by M. E. J. Newman. Web. 2 July 2007: https://www.youtube.com/watch?v=YS-asmU3p_4
[6] Adolphe Quetelet - A Treatise on Man and the Development of His Faculties (1842): https://archive.org/details/treatiseonmandev00quet/page/122/mode/2up
[7] Dong, K., Cao, Y., Siercke, B., Wilber, M., & McCalla, S. G. (2017). Advising caution in studying seasonal oscillations in crime rates. PloS one, 12(9), e0185432: https://doi.org/10.1371/journal.pone.0185432
[8] Mass.gov - Reopening Massachusetts. May 18, 2020. [PDF]: https://www.mass.gov/doc/reopening-massachusetts-may-18-2020/download
[9] Wikipedia - Pareto principle: https://en.wikipedia.org/wiki/Pareto_principle
[10] USLegal - Index Crimes Law and Legal Definition. https://definitions.uslegal.com/i/index-crimes/
[11] The New York Times. By Michelle Alexander. Why Police Lie Under Oath. Web. Feb. 2, 2013. https://www.nytimes.com/2013/02/03/opinion/sunday/why-police-officers-lie-under-oath.html?pagewanted=all&_r=0
[12] FBI:UCR • Crimes in the U.S. 2010 • Offenses Definitions. https://ucr.fbi.gov/crime-in-the-u.s/2010/crime-in-the-u.s.-2010/offense-definitions
[13] Steven D. Levitt (2014) - Understanding Why Crime Fell in the 1990s: Four Factors that Explain the Decline and Six that Do Not Lewitt: https://www.aeaweb.org/articles?id=10.1257/089533004773563485
[14] Michele Berger - Stigma of criminal record stays with individual, regardless of crime type, conviction. August 17, 2016: https://phys.org/news/2016-08-stigma-criminal-individual-crime-conviction.html
[15] Kathellen McGrory and Neil Bedi - Targeted. Sept. 3, 2020: https://projects.tampabay.com/projects/2020/investigations/police-pasco-sheriff-targeted/intelligence-led-policing
[16] Cf: http://web.cs.ucla.edu/~kwchang/publications_area/

@lru_cache(maxsize=512)
def calculate_variance(totalhands: int, winrate: float = 0.0, dev: float = 100.0, samples: int = 1000):
    """
    Another Poker Variance Calculator like Primedope's one
    Simulate randomly the amount of samples specified with winrate, totalhands and dev as parameters
    Display 20 samples, worst & best samples, 95% and 68% confidence intervals, expected winnings,
         as well as risk of ruin up to 20%
    Print useful numbers about (un)expected variance
     
    Inputs:
        totalhands (int): number of hands e.g. desired length of the samples
        winrate (float): observed (or true) winrate we want to calculate the variance
        dev (float): standard deviation ("Std Dev bb per 100 hands" stat in HM2)
        sample (int): number of samples we want to simulate
    
    """
    #limit number of samples
    if samples > 1_000_000:
        samples = 1_000_000
    
    start = time.time()
       
    h = totalhands//100
    EV = winrate*h
    
    #standard error of the mean
    std_dev = dev/(h**0.5)
    
    #probability of loss with such a winrate and standard error
    prob_loss = stats.norm.cdf(0,loc=winrate,scale=std_dev)
    
    #x-axis    
    x = np.arange(0, totalhands + 100, 100)
    
    #expected winnings
    y = [i * winrate for i in range(h+1)]
    
    #put ev values in a dataframe
    df = pd.DataFrame(y, columns=["ev"], index=x)
    
    #confidence intervals
    df["conf1"]= [i * (winrate + std_dev) for i in range(h+1)]
    df["conf2"]= [i * (winrate - std_dev) for i in range(h+1)]
    df["conf3"]= [i * (winrate + 2*std_dev) for i in range(h+1)]
    df["conf4"]= [i * (winrate - 2*std_dev) for i in range(h+1)]
    
    
    #building ramdom samples with loc = winrate and scale = dev
    arr = np.zeros(shape=(samples,h+1)) 
    for i in range(samples):
        arr[i,:]=np.cumsum(np.random.normal(winrate, dev, h+1))
    
    #best and worst EV
    bottom_ten = np.argsort(arr[:,-1])[:10]
    top_ten = np.argsort(arr[:,-1])[-10:]
    
    #bottom line for each sample
    worst_downs = np.array([i for i in np.amin(arr, axis=1)])
    
    #minimum bankroll required for each risk of ruin percentile
    #if the i-th percentile returns a positive value, minimum bankroll required is set to zero
    rr = range(0,101)
    min_bkr = [int(-np.percentile(worst_downs,i)) if np.percentile(worst_downs,i) < 0 else 0 for i in rr]
    rr_df = pd.DataFrame({"Risk of ruin": rr, "Minimum Bankroll": min_bkr})
    rr_df = rr_df.set_index("Risk of ruin")
    
    #best and worst samples
    df["best"] = arr[top_ten[-1],:]
    df["worst"] = arr[bottom_ten[-1],:]
    
    #print computing duration
    print(f'Duration: {time.time() - start}s')
        
    #path to save image
    IMAGE_DIR = Path.cwd().parent / "images"

    plt.figure(figsize=(20,10))
    #select randomly 20 samples to display excluding best and worst samples
    idx_array = [i for i in range(samples) if i not in [top_ten[-1],bottom_ten[-1]]]
    random_idx = np.random.choice(idx_array, 20, replace = False)
    for i in random_idx:    
        plt.plot(x, arr[i,:],linewidth=1)
   
    #display confidence intervals as well as best and worst samples
    fmt=["b","g--","g--","g","g","c","y"]
    labels=["EV","68% confidence interval","","95% confidence interval","","Best", "Worst"]
    for idx, col in enumerate(df.columns):
        df[col] = df[col].astype(int)
        plt.plot(df[col],fmt[idx],linewidth=3,label=labels[idx])
        
    plt.title("Samples over %d hands with confidence intervals" %totalhands)
    plt.xlabel("total hands")
    plt.ylabel("win/loss in big blinds")
    plt.legend(bbox_to_anchor=(0.0, 1.0), loc='upper left')        
    plt.savefig(IMAGE_DIR / "variance.png")
    plt.show()
    
    
    #some numbers to print
    ws = [winrate + std_dev, winrate - std_dev, winrate + (2*std_dev), winrate - (2*std_dev)]
    bb = [df.conf1.iloc[-1],df.conf2.iloc[-1],df.conf3.iloc[-1],df.conf4.iloc[-1]] 
    std_dev_bb = int(std_dev*h)    
    
    print (f"Expected winnings: {winrate} bb/100 ({df.ev.iloc[-1]} bb)")
    print (f"Standard deviation after {totalhands} hands: {std_dev: .2f} bb/100 ({std_dev_bb} bb)")
    print("\n")
    print (f"68% confidence interval: {ws[0]:.2f} bb/100 ({bb[0]} bb), {ws[1]:.2f} bb/100 ({bb[1]} bb)")
    print (f"95% confidence interval: {ws[2]:.2f} bb/100 ({bb[2]} bb), {ws[3]:.2f} bb/100 ({bb[3]} bb)")
    print("\n")
    print (f"Top-10 over {samples} samples (bb/100):\n{np.around(arr[top_ten][:,-1]/h, 2)}")
    print("\n")
    print (f"Bottom-10 over {samples} samples (bb/100):\n{np.around(arr[bottom_ten][:,-1]/h, 2)}")
    print("\n")
    print (f"Probability of loss after {totalhands} hands: {prob_loss:.2%}")
    print("\n")
       
    #display the first 20th percentiles of risk of ruin dataframe
    plt.figure(figsize=(20,10))    
    plt.plot(rr_df)    
    plt.title("Risk of ruin\n(q-th percentile over %s samples merged by their bottom line)" %samples)
    plt.xlabel("risk of ruin %")
    plt.xlim(0,20)
    plt.xticks(range(0,22,2))
    plt.ylabel("minimum bankroll in big blinds")
    plt.savefig(IMAGE_DIR / "risk_of_ruin.png")
    plt.show()    
    
    # return

calculate_variance(totalhands = 80000, winrate = 5.5, dev = 100, samples=1000)

Duration: 0.06800365447998047s

Expected winnings: 5.5 bb/100 (4400 bb)
Standard deviation after 80000 hands:  3.54 bb/100 (2828 bb)


68% confidence interval: 9.04 bb/100 (7228 bb), 1.96 bb/100 (1571 bb)
95% confidence interval: 12.57 bb/100 (10056 bb), -1.57 bb/100 (-1256 bb)


Top-10 over 1000 samples (bb/100):
[14.41 14.41 14.66 14.74 14.74 14.81 15.29 15.77 16.81 17.01]


Bottom-10 over 1000 samples (bb/100):
[-5.4  -5.02 -3.94 -3.3  -3.28 -3.11 -2.94 -2.84 -2.82 -2.72]


Probability of loss after 80000 hands: 5.99%

fastpages

Seasonality in Boston Index Crime rates

Be like a sniffer dog

First clues of seasonality and lockdown effect

Size doesn't matter: all we need is love and good data

The trajectory space of the crime

Long-term trends extraction (the signal)

Seasonal components extraction (the noise)

References

The (un)expected variance at poker

Table of Contents

An Example Markdown Post

Example Markdown Post

Basic setup

Basic formatting

Lists

Boxes and stuff

Images

Code

Tables

Tweetcards

Footnotes