.png)
7 hours ago
1
Executive Summary
Purpose: To analyze TTFF in CI processes over the past three months. Methods: Data analysis using R, focusing on data cleaning, EDA, statistical tests, and time series analysis.
Key Findings: Identification of a statistically significant decrease in TTFF after optimizations. Statistical evidence supporting the effectiveness of recent improvements. Slight increase in the number outliers in the month of September.
Introduction
Continuous Integration (CI) is a critical component of modern software development practices at OP Labs. The Time to First Failure (TTFF) metric is particularly important as it impacts the developer experience by indicating how quickly feedback is received after code changes. Over the past three weeks, the team has implemented optimizations aimed at reducing TTFF. This report analyzes the impact of these optimizations on TTFF over the past three months.
Methodology
Data Collection
Source: CI attempt records from the past three months.
Data File: 'ci analysis sheet1.csv'.
Data Fields:
Workflow ID: Unique identifier for each CI run.
Date: Date of the CI attempt.
Time (Seconds): TTFF for each attempt.
Data Preparation and Cleaning
Steps Taken:
Excluded weekends and days with fewer than 5 attempts to focus on representative data.
Identified and handled missing values.
Detected and managed outliers using the IQR method.
Note on Outliers:
Found an attempt with a negative TTFF (Workflow ID:97851cf0-1b57-4d36-922e-e35863d08776on September 13th, 2024).
Decision: Excluded this data point from the analysis due to its invalid value.
Setting up the R Environment
Loading the Data
#Loading the necessary libraries library(tidyverse) ## ── Attaching core tidyverse packages ──────────────────────── tidyverse 2.0.0 ── ## ✔ dplyr 1.1.4 ✔ readr 2.1.5 ## ✔ forcats 1.0.0 ✔ stringr 1.5.1 ## ✔ ggplot2 3.5.1 ✔ tibble 3.2.1 ## ✔ lubridate 1.9.3 ✔ tidyr 1.3.1 ## ✔ purrr 1.0.2 ## ── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ── ## ✖ dplyr::filter() masks stats::filter() ## ✖ dplyr::lag() masks stats::lag() ## ℹ Use the conflicted package (<http://conflicted.r-lib.org/>) to force all conflicts to become errors library(lubridate) library(scales) ## ## Attaching package: 'scales' ## ## The following object is masked from 'package:purrr': ## ## discard ## ## The following object is masked from 'package:readr': ## ## col_factor library(cowplot) ## ## Attaching package: 'cowplot' ## ## The following object is masked from 'package:lubridate': ## ## stamp library(effsize) library(zoo) ## ## Attaching package: 'zoo' ## ## The following objects are masked from 'package:base': ## ## as.Date, as.Date.numeric library(segmented) ## Loading required package: MASS ## ## Attaching package: 'MASS' ## ## The following object is masked from 'package:dplyr': ## ## select ## ## Loading required package: nlme ## ## Attaching package: 'nlme' ## ## The following object is masked from 'package:dplyr': ## ## collapse library(dplyr) #Loading the data ci_data <- read_csv("ci analysis sheet1.csv", col_types = cols( `Workflow ID` = col_character(), Date = col_date(format = "%m/%d/%Y"), `Time (Seconds)` = col_double() ))Data Preparation and Cleaning
# Add day of the week ci_data <- ci_data %>% mutate(DayOfWeek = wday(Date, label = TRUE)) # Exclude weekends (Saturday and Sunday) and negative Time (Seconds) ci_data_weekdays <- ci_data %>% filter( !DayOfWeek %in% c("Sat", "Sun"), `Time (Seconds)` > 0 ) # Exclude days with fewer than 5 attempts attempt_counts <- ci_data_weekdays %>% group_by(Date) %>% tally(name = "AttemptCount") ci_data_filtered <- ci_data_weekdays %>% inner_join(attempt_counts, by = "Date") %>% filter(AttemptCount >= 5)Checking for missing values
# Check for missing values sum(is.na(ci_data_filtered$`Time (Seconds)`)) ## [1] 0 # Identify outliers using the IQR method Q1 <- quantile(ci_data_filtered$`Time (Seconds)`, 0.25) Q3 <- quantile(ci_data_filtered$`Time (Seconds)`, 0.75) IQR <- Q3 - Q1 # Define outlier thresholds lower_bound <- Q1 - 1.5 * IQR upper_bound <- Q3 + 1.5 * IQR # Filter out outliers ci_data_no_outliers <- ci_data_filtered %>% filter(`Time (Seconds)` >= lower_bound & `Time (Seconds)` <= upper_bound) ## Separate outliers into a new table ci_outliers <- ci_data_filtered %>% filter(`Time (Seconds)` < lower_bound | `Time (Seconds)` > upper_bound)Exploratory Data Analysis and Descriptive Statistics
# Calculate Summary Statistics for Both Datasets summary_stats_combined <- bind_rows( ci_data_filtered %>% summarise( Data = "Including Outliers", Mean = mean(`Time (Seconds)`, na.rm = TRUE), Median = median(`Time (Seconds)`, na.rm = TRUE), SD = sd(`Time (Seconds)`, na.rm = TRUE), Min = min(`Time (Seconds)`, na.rm = TRUE), Max = max(`Time (Seconds)`, na.rm = TRUE) ), ci_data_no_outliers %>% summarise( Data = "Excluding Outliers", Mean = mean(`Time (Seconds)`, na.rm = TRUE), Median = median(`Time (Seconds)`, na.rm = TRUE), SD = sd(`Time (Seconds)`, na.rm = TRUE), Min = min(`Time (Seconds)`, na.rm = TRUE), Max = max(`Time (Seconds)`, na.rm = TRUE) ) ) # Print Combined Summary Statistics print(summary_stats_combined) ## # A tibble: 2 × 6 ## Data Mean Median SD Min Max ## <chr> <dbl> <dbl> <dbl> <dbl> <dbl> ## 1 Including Outliers 852. 527. 2375. 10.8 18572. ## 2 Excluding Outliers 521. 487. 382. 10.8 1838.Plotting Histograms and density plots
# Histogram ggplot(ci_data_no_outliers, aes(x = `Time (Seconds)`)) + geom_histogram(binwidth = 10, fill = "skyblue", color = "black") + labs(title = "Distribution of TTFF", x = "Time (Seconds)", y = "Frequency") + theme_minimal(base_size = 15) + theme(plot.title = element_text(hjust = 0.5))Time series visualization
# Time Series Without Extreme Outliers and y-axis Limit ggplot(ci_data_no_outliers, aes(x = Date, y = `Time (Seconds)`)) + geom_point(alpha = 0.5) + geom_smooth(method = "loess", color = "red") + # Smooth trend line geom_smooth(method = "lm", color = "blue", linetype = "dashed") + # Additional linear trend line labs(title = "TTFF Over Time\n(Excluding Outliers)", x = "Date", y = "Time (Seconds)") + scale_y_continuous(limits = c(0, 2000)) + # Adjust y-axis limit if needed theme_minimal(base_size = 15) + theme(plot.title = element_text(hjust = 0.5, face = "bold")) ## `geom_smooth()` using formula = 'y ~ x' ## `geom_smooth()` using formula = 'y ~ x'This time series visualization illustrates the fluctuation of Time to First Failure (TTFF) values from July to September. TTFF represents the duration in seconds required for the automated CI tests to give some feedback regarding things that need to be addressed prior to human review and the Pull Request being accepted. High TTFF correspond to a waste of time, as well as a poor developer experience, because no work can reasonably be done while they are waiting for feedback.
In the visualization, two trend lines are displayed:
Red Line: A loess trend line that smoothly captures fluctuations in TTFF over time. Blue Dashed Line: A linear trend line (lm) for a straightforward trend comparison. Key Insights: The loess trend line initially rises in July, stabilizes in August, and then declines sharply starting from early September, indicating an improvement in signal acquisition times. The average TTFF falls below the 1-minute benchmark around mid-September, suggesting enhancements in GPS signal performance. By identifying these key dates, we can correlate them with potential system updates or changes to understand what factors contributed to the improved performance.
# Calculate loess fitted values with the filtered data loess_fit <- loess(`Time (Seconds)` ~ as.numeric(Date), data = ci_data_no_outliers) fitted_values <- data.frame(Date = ci_data_no_outliers$Date, Fitted = predict(loess_fit)) # Identify the date where the loess trend reaches its peak and then starts declining turning_point <- fitted_values %>% arrange(Date) %>% mutate(Trend_Change = c(0, diff(Fitted))) %>% filter(Trend_Change > 0) %>% slice(n()) # Last date where trend is still increasing (before decline) # Use dplyr::select to avoid conflicts turning_point <- turning_point %>% dplyr::select(Date, Fitted) # Calculate the date when average TTFF rises above 60 seconds and falls back below # Step 1: Find the first date when average TTFF rises above 60 above_60 <- ci_data_no_outliers %>% group_by(Date) %>% summarize(Mean_TTFF = mean(`Time (Seconds)`)) %>% filter(Mean_TTFF >= 60) %>% slice(1) # First date when average TTFF rises above 60 seconds # Step 2: Find the first date after above_60 when TTFF falls below 60 again average_fall_below <- ci_data_no_outliers %>% group_by(Date) %>% summarize(Mean_TTFF = mean(`Time (Seconds)`)) %>% filter(Date > above_60$Date, Mean_TTFF < 60) %>% slice(1) # First date after rise when average TTFF falls back below 60 # Print key dates print(turning_point) ## Date Fitted ## 1 2024-07-26 668.894 print(above_60) ## # A tibble: 1 × 2 ## Date Mean_TTFF ## <date> <dbl> ## 1 2024-06-24 679. print(average_fall_below) ## # A tibble: 0 × 2 ## # ℹ 2 variables: Date <date>, Mean_TTFF <dbl>The ‘turning_point’ identified the point as which the CI process started to get slightly faster, as it seems to have been getting bloated from the point that it went above 1 minute TTFF on average at the end of June, up until, July 26th when it hit an inflection point. As of September 10th, the TTFF average has fallen below 1 minute once again.
# Fit both loess and linear models over the same date range loess_fit <- loess(`Time (Seconds)` ~ as.numeric(Date), data = ci_data_no_outliers) lm_fit <- lm(`Time (Seconds)` ~ as.numeric(Date), data = ci_data_no_outliers) # Generate a sequence of dates for prediction date_sequence <- data.frame(Date = seq(min(ci_data_no_outliers$Date), max(ci_data_no_outliers$Date), by = "day")) # Predict fitted values for both models date_sequence$Loess_Fitted <- predict(loess_fit, newdata = date_sequence) date_sequence$LM_Fitted <- predict(lm_fit, newdata = date_sequence) # Find the intersection point where the two lines cross date_sequence <- date_sequence %>% mutate(Difference = Loess_Fitted - LM_Fitted, # Calculate difference between loess and lm Cross = Difference * lag(Difference) < 0) # Check if the difference changes sign # Get the date of the first intersection in September intersection <- date_sequence %>% filter(Cross, format(Date, "%Y-%m") == "2024-09") %>% # Filter for crossings in September slice(1) # First intersection date # Print the date and values at the intersection intersection ## Date Loess_Fitted LM_Fitted Difference Cross ## 1 2024-09-10 408.8887 416.2849 -7.396144 TRUEComparing TTFF across different segments
Monthly Summary
# Adding a 'Month' indicator to the data ci_data_no_outliers <- ci_data_no_outliers %>% mutate(Month = format(Date, "%Y-%m")) # Extract year and month # Calculate summary statistics by month monthly_summary_no_outliers <- ci_data_no_outliers %>% group_by(Month) %>% summarize( Mean_TTFF = mean(`Time (Seconds)`, na.rm = TRUE), Median_TTFF = median(`Time (Seconds)`, na.rm = TRUE), SD_TTFF = sd(`Time (Seconds)`, na.rm = TRUE), Count = n() ) # Display monthly summary monthly_summary_no_outliers ## # A tibble: 4 × 5 ## Month Mean_TTFF Median_TTFF SD_TTFF Count ## <chr> <dbl> <dbl> <dbl> <int> ## 1 2024-06 596. 702. 334. 91 ## 2 2024-07 642. 740. 409. 287 ## 3 2024-08 607. 745. 397. 564 ## 4 2024-09 373. 348. 308. 616 # Visualization by Month ggplot(ci_data_no_outliers, aes(x = Month, y = `Time (Seconds)`, fill = Month)) + geom_boxplot() + labs(title = "TTFF Distribution by Month", x = "Month", y = "Time (Seconds)") + theme_minimal(base_size = 15) + theme(plot.title = element_text(hjust = 0.5, face = "bold"))Defining Custom Segments based on Visual Cues
Previously we were able to determine that there was a turning point at the end of July, as well as major improvements noticed at the beginning of September, most notably the TTFF dropping below 1 minute on average on September 10th. Focusing on those dates specifically:
Inflection points including outliers
Inflections points excluding outliers
# Define visual cue segments (based on observation from the chart) cutoff_date_1 <- as.Date("2024-07-26") # Example date when trend changes cutoff_date_2 <- as.Date("2024-09-01") # Example date before a noticeable decline # Adding a 'Visual_Segment' indicator based on the visual analysis ci_data_no_outliers <- ci_data_no_outliers %>% mutate(Visual_Segment = case_when( Date <= cutoff_date_1 ~ "Early Period", Date > cutoff_date_1 & Date < cutoff_date_2 ~ "Mid Period", Date >= cutoff_date_2 ~ "Late Period" )) # Calculate summary statistics by visual segment visual_segment_summary_no_outliers <- ci_data_no_outliers %>% group_by(Visual_Segment) %>% summarize( Mean_TTFF = mean(`Time (Seconds)`, na.rm = TRUE), Median_TTFF = median(`Time (Seconds)`, na.rm = TRUE), SD_TTFF = sd(`Time (Seconds)`, na.rm = TRUE), Count = n() ) # Display visual segment summary visual_segment_summary_no_outliers ## # A tibble: 3 × 5 ## Visual_Segment Mean_TTFF Median_TTFF SD_TTFF Count ## <chr> <dbl> <dbl> <dbl> <int> ## 1 Early Period 631. 711. 392. 338 ## 2 Late Period 373. 348. 308. 616 ## 3 Mid Period 609. 746. 397. 604 # Reorder Visual_Segment column ci_data_no_outliers$Visual_Segment <- factor(ci_data_no_outliers$Visual_Segment, levels = c("Early Period", "Mid Period", "Late Period")) # Visualization by Visual Segment ggplot(ci_data_no_outliers, aes(x = Visual_Segment, y = `Time (Seconds)`, fill = Visual_Segment)) + geom_boxplot() + labs(title = "TTFF Distribution by Visual Segments", x = "Segment", y = "Time (Seconds)") + theme_minimal(base_size = 15) + theme(plot.title = element_text(hjust = 0.5, face = "bold"))Shapiro-Wilk / Mann-Whitney U tests
# Define a function to perform Shapiro-Wilk and Mann-Whitney U tests perform_tests <- function(data, group_var, value_var) { # Shapiro-Wilk test for each group shapiro_results <- data %>% group_by(!!sym(group_var)) %>% summarise(Shapiro_p_value = shapiro.test(!!sym(value_var))$p.value) # Mann-Whitney U test for pairwise comparisons pairwise_comparisons <- combn(unique(data[[group_var]]), 2, simplify = FALSE) mann_whitney_results <- lapply(pairwise_comparisons, function(pair) { group1 <- data %>% filter(!!sym(group_var) == pair[1]) %>% pull(!!sym(value_var)) group2 <- data %>% filter(!!sym(group_var) == pair[2]) %>% pull(!!sym(value_var)) test_result <- wilcox.test(group1, group2) data.frame( Group1 = pair[1], Group2 = pair[2], Mann_Whitney_p_value = test_result$p.value ) }) # Combine results into a single dataframe mann_whitney_results <- do.call(rbind, mann_whitney_results) return(list(Shapiro_Results = shapiro_results, Mann_Whitney_Results = mann_whitney_results)) } # Monthly Analysis with Outliers #monthly_test_outliers <- perform_tests( # data = ci_data_filtered, # group_var = "Month", # value_var = "Time (Seconds)" #) # Monthly Analysis without Outliers monthly_test_no_outliers <- perform_tests( data = ci_data_no_outliers, group_var = "Month", value_var = "Time (Seconds)" ) # Visual Segment Analysis with Outliers #visual_test_outliers <- perform_tests( # data = ci_data_filtered, # group_var = "Visual_Segment", # value_var = "Time (Seconds)" #) # Visual Segment Analysis without Outliers visual_test_no_outliers <- perform_tests( data = ci_data_no_outliers, group_var = "Visual_Segment", value_var = "Time (Seconds)" ) # Print test results list( # Monthly_With_Outliers = monthly_test_outliers, Monthly_Without_Outliers = monthly_test_no_outliers, # Visual_With_Outliers = visual_test_outliers, Visual_Without_Outliers = visual_test_no_outliers ) ## $Monthly_Without_Outliers ## $Monthly_Without_Outliers$Shapiro_Results ## # A tibble: 4 × 2 ## Month Shapiro_p_value ## <chr> <dbl> ## 1 2024-06 1.08e- 7 ## 2 2024-07 1.05e-12 ## 3 2024-08 2.84e-17 ## 4 2024-09 7.35e-20 ## ## $Monthly_Without_Outliers$Mann_Whitney_Results ## Group1 Group2 Mann_Whitney_p_value ## 1 2024-06 2024-07 3.254879e-01 ## 2 2024-06 2024-08 5.376250e-01 ## 3 2024-06 2024-09 1.216689e-09 ## 4 2024-07 2024-08 1.931853e-02 ## 5 2024-07 2024-09 7.500467e-22 ## 6 2024-08 2024-09 6.790153e-24 ## ## ## $Visual_Without_Outliers ## $Visual_Without_Outliers$Shapiro_Results ## # A tibble: 3 × 2 ## Visual_Segment Shapiro_p_value ## <fct> <dbl> ## 1 Early Period 1.38e-13 ## 2 Mid Period 5.86e-18 ## 3 Late Period 7.35e-20 ## ## $Visual_Without_Outliers$Mann_Whitney_Results ## Group1 Group2 Mann_Whitney_p_value ## 1 Early Period Mid Period 6.703805e-02 ## 2 Early Period Late Period 4.306223e-24 ## 3 Mid Period Late Period 3.196324e-25Results
Evaluation of Statistical Tests
The statistical tests conducted for the Time to First Fix (TTFF) data provide insights into the distribution and comparison of TTFF across different periods and monthly segments. Here we evaluate the results of the Shapiro-Wilk and Mann-Whitney U tests, both including and excluding outliers.
Shapiro-Wilk Test for Normality
The Shapiro-Wilk test was used to assess whether the TTFF data follows a normal distribution for each period and month.
Monthly Analysis (Without Outliers)
| 2024-06 | 1.08×10−7 | Data is not normally distributed. |
| 2024-07 | 1.05×10−12 | Data is not normally distributed. |
| 2024-08 | 2.84×10−17 | Data is not normally distributed. |
| 2024-09 | 7.35×10−20 | Data is not normally distributed. |
Conclusion: Excluding outliers does not change the non-normal distribution of TTFF data. All months still exhibit significant deviation from normality.
Visual Segments (Without Outliers)
| Early Period | 1.38×10−13 | Data is not normally distributed. |
| Mid Period | 5.86×10−18 | Data is not normally distributed. |
| Late Period | 7.35×10−20 | Data is not normally distributed. |
Conclusion: Similar to the case with outliers, all visual segments still deviate significantly from normality even after excluding outliers.
Mann-Whitney U Test for Group Comparisons
The Mann-Whitney U test was used to compare the TTFF distributions between different months and visual segments. This non-parametric test is appropriate due to the non-normal distribution of the data.
Monthly Analysis (Without Outliers)
| 2024-06 | 2024-07 | 0.325 | No significant difference. |
| 2024-06 | 2024-08 | 0.538 | No significant difference. |
| 2024-06 | 2024-09 | 1.22×10−9 | Significant difference (June vs. September). |
| 2024-07 | 2024-08 | 0.019 | Significant difference (July vs. August). |
| 2024-07 | 2024-09 | 7.50×10−22 | Significant difference (July vs. September). |
| 2024-08 | 2024-09 | 6.79×10−24 | Significant difference (August vs. September). |
Conclusion: Excluding outliers still shows significant differences for comparisons involving September, indicating a substantial change in TTFF values in that month.
Visual Segments (Without Outliers)
| Early Period | Mid Period | 0.067 | Marginally significant difference. |
| Early Period | Late Period | 4.31×10−24 | Significant difference (Early vs. Late). |
| Mid Period | Late Period | 3.20×10−25 | Significant difference (Mid vs. Late). |
Conclusion: Similar to the case with outliers, excluding outliers still shows significant differences between the Late period and the earlier segments.
Overall Evaluation
The Shapiro-Wilk test confirms that the TTFF data does not follow a normal distribution, regardless of the inclusion or exclusion of outliers. This justifies the use of non-parametric tests like the Mann-Whitney U test for group comparisons. The Mann-Whitney results indicate that significant changes in TTFF values occur primarily in September and during the Late period, likely due to operational changes or optimizations. These findings suggest that further investigation into the causes of these changes is warranted.
Calculating Effect Size using Cliff’s Delta
Cliff’s delta is a non-parametric effect size measure that quantifies the difference between two distributions. It is particularly useful when dealing with non-normal data and provides a value between -1 and 1:
+1: All values in the first group are greater than those in the second group.
-1: All values in the first group are less than those in the second group.
0: No difference between the two groups.
Evaluating the Results from Cliff’s Delta
Cliff’s delta analysis confirms that September and the Late period exhibit significant changes in TTFF values compared to previous months and periods. The effect sizes are mostly medium when outliers are excluded, suggesting that operational changes during these times had a noticeable impact on the system’s performance. Including outliers tends to dilute these effects, showing smaller effect sizes, particularly in the earlier months.
This analysis, combined with the statistical tests, provides a comprehensive view of the data’s behavior over time and highlights critical periods where further investigation is warranted. These results can guide operational improvements and help understand the factors influencing TTFF variations.
Visualizing the Results for Monthly Segments
# Boxplot for Monthly Comparison without Outliers ggplot(ci_data_no_outliers, aes(x = Month, y = `Time (Seconds)`, fill = Month)) + geom_boxplot() + labs(title = "TTFF Distribution by Month\n(Excluding Outliers)", x = "Month", y = "Time (Seconds)") + theme_minimal(base_size = 15) + theme(plot.title = element_text(hjust = 0.5, face = "bold"))Visualizing Results for Visual Segments
# Reorder Visual_Segment column in ci_data_no_outliers ci_data_no_outliers$Visual_Segment <- factor(ci_data_no_outliers$Visual_Segment, levels = c("Early Period", "Mid Period", "Late Period")) # Boxplot for Visual Segments without Outliers ggplot(ci_data_no_outliers, aes(x = Visual_Segment, y = `Time (Seconds)`, fill = Visual_Segment)) + geom_boxplot() + labs(title = "TTFF Distribution by Visual Segments\n(Excluding Outliers)", x = "Segment", y = "Time (Seconds)") + theme_minimal(base_size = 15) + theme(plot.title = element_text(hjust = 0.5, face = "bold"))Time Series Analysis
Calculating Moving Averages
# Calculate daily average TTFF for dataset without outliers Daily_Average_No_Outliers <- ci_data_no_outliers %>% group_by(Date) %>% summarise(Average_TTFF = mean(`Time (Seconds)`, na.rm = TRUE)) %>% arrange(Date) %>% mutate(MA_7 = rollmean(Average_TTFF, k = 7, fill = NA, align = "right")) # 7-day moving averageVisualizing moving averages
# Define the cutoff date for visualization Cutoff_Date <- as.Date("2024-09-01") # Adjust as necessary based on your context # Visualization for dataset without outliers ggplot(Daily_Average_No_Outliers, aes(x = Date)) + geom_line(aes(y = Average_TTFF), color = "grey70") + # Daily average TTFF geom_line(aes(y = MA_7), color = "blue", size = 1) + # 7-day moving average geom_vline(xintercept = Cutoff_Date, linetype = "dashed", color = "red") + annotate( "text", x = Cutoff_Date + days(1), y = max(Daily_Average_No_Outliers$Average_TTFF, na.rm = TRUE), label = "Optimizations Begin", hjust = 0, vjust = 1, color = "red" ) + labs(title = "TTFF Over Time with 7-Day Moving Average (Excluding Outliers)", x = "Date", y = "Average TTFF (Seconds)") + theme_minimal(base_size = 15) ## Warning: Using `size` aesthetic for lines was deprecated in ggplot2 3.4.0. ## ℹ Please use `linewidth` instead. ## This warning is displayed once every 8 hours. ## Call `lifecycle::last_lifecycle_warnings()` to see where this warning was ## generated. ## Warning: Removed 6 rows containing missing values or values outside the scale range ## (`geom_line()`).Regression Analysis
# Convert Date to numeric for linear models ci_data_no_outliers <- ci_data_no_outliers %>% mutate(Date_numeric = as.numeric(Date)) # Initial Linear Model with Numeric Date (Excluding Outliers) lm_model_no_outliers <- lm(`Time (Seconds)` ~ Date_numeric, data = ci_data_no_outliers) # Segmented Regression Model (Excluding Outliers) segmented_model_no_outliers <- segmented( lm_model_no_outliers, seg.Z = ~Date_numeric, psi = 19967 # Initial guess for the breakpoint ) # Simple Linear Regression (Excluding Outliers) simple_lm_no_outliers <- lm(`Time (Seconds)` ~ Date_numeric, data = ci_data_no_outliers) # Summary of Segmented Models summary(segmented_model_no_outliers) ## ## ***Regression Model with Segmented Relationship(s)*** ## ## Call: ## segmented.lm(obj = lm_model_no_outliers, seg.Z = ~Date_numeric, ## psi = 19967) ## ## Estimated Break-Point(s): ## Est. St.Err ## psi1.Date_numeric 19949 3.234 ## ## Coefficients of the linear terms: ## Estimate Std. Error t value Pr(>|t|) ## (Intercept) -2.762e+04 1.690e+04 -1.635 0.1023 ## Date_numeric 1.418e+00 8.478e-01 1.673 0.0946 . ## U1.Date_numeric -1.199e+01 1.502e+00 -7.987 NA ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 ## ## Residual standard error: 356.5 on 1554 degrees of freedom ## Multiple R-Squared: 0.1321, Adjusted R-squared: 0.1305 ## ## Boot restarting based on 9 samples. Last fit: ## Convergence attained in 2 iterations (rel. change 5.1227e-08) # Summary of Simple Linear Models summary(simple_lm_no_outliers) ## ## Call: ## lm(formula = `Time (Seconds)` ~ Date_numeric, data = ci_data_no_outliers) ## ## Residuals: ## Min 1Q Median 3Q Max ## -711.42 -322.88 24.92 261.41 1360.28 ## ## Coefficients: ## Estimate Std. Error t value Pr(>|t|) ## (Intercept) 90711.6847 7287.1412 12.45 <2e-16 *** ## Date_numeric -4.5202 0.3652 -12.38 <2e-16 *** ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 ## ## Residual standard error: 364.9 on 1556 degrees of freedom ## Multiple R-squared: 0.08962, Adjusted R-squared: 0.08904 ## F-statistic: 153.2 on 1 and 1556 DF, p-value: < 2.2e-16 # Extract breakpoints from segmented models and convert to date breakpoint_no_outliers <- summary(segmented_model_no_outliers)$psi[1, "Est."] cutoff_date_no_outliers <- as.Date(breakpoint_no_outliers, origin = "1970-01-01") # Print the cutoff dates print(paste("Cutoff Date (Excluding Outliers):", cutoff_date_no_outliers)) ## [1] "Cutoff Date (Excluding Outliers): 2024-08-14" # Extract breakpoints for visualization breakpoint_no_outliers <- as.Date(summary(segmented_model_no_outliers)$psi[1, "Est."], origin = "1970-01-01") # Visualization for Linear and Segmented Regression (Excluding Outliers) with Breakpoint ggplot(ci_data_no_outliers, aes(x = Date, y = `Time (Seconds)`)) + geom_point(alpha = 0.5) + geom_smooth(method = "lm", color = "blue", linetype = "dashed", size = 1.5, aes(y = fitted(simple_lm_no_outliers))) + # Simple linear trend line geom_line(aes(y = fitted(segmented_model_no_outliers)), color = "red", size = 1.5) + # Segmented regression fit geom_vline(xintercept = breakpoint_no_outliers, linetype = "dashed", color = "green", size = 1) + # Breakpoint line annotate("text", x = breakpoint_no_outliers, y = max(ci_data_no_outliers$`Time (Seconds)`, na.rm = TRUE), label = "Breakpoint", color = "green", hjust = -0.1, vjust = 1, size = 5) + # Breakpoint label labs(title = "TTFF Linear and Segmented Regression\n(Excluding Outliers)", x = "Date", y = "Time (Seconds)") + theme_minimal(base_size = 15) ## `geom_smooth()` using formula = 'y ~ x'Interpreting the Regression Outputs
Linear Regression Results
Including Outliers:
Coefficients: Estimate Std. Error t value Pr(>|t|) (Intercept) -92441.261 46887.981 -1.972 0.0488 * Date_numeric 4.676 2.350 1.990 0.0468 *
Interpretation:
The slope (Estimate for Date_numeric) is +4.676, indicating that TTFF is increasing over time when outliers are included.
The p-value for the slope is 0.0468, which is just below 0.05, suggesting the increase is statistically significant, but only marginally.
Multiple R-squared is 0.002476, indicating that the model explains only about 0.25% of the variance in TTFF.
Excluding Outliers:
Coefficients: Estimate Std. Error t value Pr(>|t|) (Intercept) 90711.6847 7287.1412 12.45 <2e-16 *** Date_numeric -4.5202 0.3652 -12.38 <2e-16 ***
Interpretation:
The slope is -4.5202, indicating that TTFF is decreasing over time when outliers are excluded.
The p-value is <2e-16, indicating a highly statistically significant negative trend.
Multiple R-squared is 0.08962, meaning the model explains about 8.96% of the variance in TTFF.
Segmented Regression Results
Including Outliers:
Copy code
`Estimated Break-Point(s): Est. St.Err psi1.Date_numeric 19981.84 0.874
Coefficients of the linear terms: Estimate Std. Error t value
Pr(>|t|)
(Intercept) -2.139e+05 5.512e+04 -3.881 0.000108
Date_numeric 1.077e+01 2.763e+00 3.897
0.000101 U1.Date_numeric -3.651e+02 1.243e+02 -2.938
NA`
Breakpoint Estimate: Date_numeric = 19981.84, which corresponds to 2024-09-15.
Interpretation:
Before the breakpoint, TTFF is increasing over time.
After the breakpoint, the slope changes by -365.1, suggesting a decrease in TTFF post-breakpoint.
The change in slope is statistically significant (t value of -2.938).
Excluding Outliers:
plaintextCopy code
`Estimated Break-Point(s): Est. St.Err psi1.Date_numeric 19949 3.234
Coefficients of the linear terms: Estimate Std. Error t value
Pr(>|t|)
(Intercept) -2.762e+04 1.690e+04 -1.635 0.1023
Date_numeric 1.418e+00 8.478e-01 1.673 0.0946 . U1.Date_numeric
-1.199e+01 1.502e+00 -7.987 NA`
Breakpoint Estimate: Date_numeric = 19949, which corresponds to 2024-08-14.
Interpretation:
Before the breakpoint, TTFF is increasing slightly over time.
After the breakpoint, the slope decreases by -11.99, indicating a more pronounced decrease in TTFF after the breakpoint.
The change in slope is highly significant (t value of -7.987).
Understanding the Breakpoints
- Excluding Outliers: Breakpoint at August 14, 2024.
Key Observations
When excluding outliers:
TTFF shows a significant decreasing trend over time.
The breakpoint is earlier (mid-August), before Kelvin started his optimizations.
Summary of Findings
Significant Decrease in TTFF:
Excluding outliers, TTFF has decreased by an average of 4.52 seconds per day, a highly significant trend (p < 2e-16).
This suggests a meaningful improvement in the speed at which developers receive feedback from the CI process.
Alignment with Optimization Efforts:
- This change point marks a shift from an increasing to a decreasing trend in TTFF, highlighting the impact of the recent efforts.
Impact of Outliers:
Outliers significantly affect the analysis, underscoring the importance of monitoring and addressing extreme TTFF values.
Including outliers results in a marginally significant increase in TTFF over time, which may not reflect the overall improvements.
Conclusion
Effectiveness of Optimizations:
The evidence supports the conclusion that Kelvin’s work has led to a significant reduction in TTFF.
This improvement enhances the developer experience by reducing wait times for feedback, potentially increasing productivity.
Recommendations:
Continuous Monitoring: Implement ongoing monitoring of TTFF to maintain and further improve CI performance.
Outlier Investigation: Examine the causes of outliers to prevent extreme delays and ensure consistent performance.
Further Analysis: Consider additional factors that may influence TTFF, such as changes in code complexity or infrastructure.
Addressing the Discrepancy in Breakpoints
In your report, it’s important to discuss the discrepancy in the breakpoint estimation when including versus excluding outliers. Here’s how you might articulate this:
Discussion of Breakpoint Discrepancy
The segmented regression analysis yielded different estimated breakpoints depending on whether outliers were included:
Including Outliers: Breakpoint at September 15, 2024.
Excluding Outliers: Breakpoint at August 14, 2024.
Implications:
Kelvin’s Impact: Despite the earlier breakpoint when excluding outliers, the overall significant decrease in TTFF aligns with Kelvin’s optimization efforts.
Outlier Management: Managing and understanding outliers is crucial, as they can distort analyses and mask underlying trends.
