[RFC]: Developer Dashboard For Tracking Ecosystem Build Failures Project

[Rupa-Rd]

Full name

Roopa Dharshini

University status

Yes

University name

SRM Institute of Science and Technology

University program

Bachelors in Computer Science

Expected graduation

2025

Short biography

I am Roopa Dharshini, a final year computer science student at SRM University. I am currently contributing to the Season of KDE 2025 as a mentee for KEcoLab Project. I am a full-stack developer having experience in building a web application from scratch using MERN(MongoDB, ExpressJS, ReactJS and Node) stack. Apart from being a web developer, I am also a technical content writer at Guvi. I built and contributed to various open-source projects throughout my journey. Now, I want to take the next step in my open-source journey by contributing to the stdlib project in this GSoC.

Timezone

IST Indian Standard Time

Contact details

email: roopadharshini.rd@gmail.com github: Rupa-Rd

Platform

Windows

Editor

I use Visual Studio Code because of its simplicity and flexibility in opening a code editor, version control, command prompt and live editor.

Programming experience

I have experience in various programming languages such as Java, Python, and JavaScript. I built various projects using JavaScript.

JavaScript experience

For a change, I participated in Vinyasa Summer of Code (VSoC) as a project admin for my open source project Open-Source Galaxy. Open-Source Galaxy is a website to display the open-source contribution programs such as GSoC, LFX mentorship and so on. As a project admin, it was a challenging experience to learn how to manage issues, PRs and deployments during contributions. This project has received 16 forks, 12 stars and merged 25+ PRs.

Node.js experience

Create-front-end-app-html is my first npm package manager I created. When I joined the SheCodes community, I found it is difficult for women from non-tech backgrounds to structure a simple website project. So, I created this npm package for them to structure a simple website with html, css and javascript. This also provides a boilerplate code for html and css. The project structure can be easily created by triggering the command npx create-front-end-app-html <project_name> This project received 400 downloads in its first week of deployment.

C/Fortran experience

I've the basic knowledge in C such as functional programming, operators and other basic functions.

Interest in stdlib

I built applications for a smaller audience till now. When I came to know about stdlib and how big this community is, it really excited me to build a project that will help a larger audience.

Version control

Yes

Contributions to stdlib

PR #6465

I contributed to Stdlib by improving the doctests for complex number typed arrays in documentation examples (issue#4833). The pull request has been successfully merged (PR#6495). While working on this issue, I learned about the usage of complexArray. At first, it was challenging to understand the issue. Later I spent more time understanding the requirements and the solution to it. So, I did the respective changes under the blas/base/wasm/cscal folder.

I faced a lot of test failures during the contribution, so I had to go in depth of technical aspects of what are the failures and how it affects the results. One of the CI/CD fails is because the expected result and observed result of the example program of cscal was not matching. I resolved it by carefully studying and observing the code line by line.

stdlib showcase

To showcase the usage of stdlib and to learn fastify as a part of the GSOC project requirement, I built a web app called ‘case conversion’. This web app will convert the input sentence into various case conversion formats such as upper case, lower case, pascal case, camel case, kebab case, start case and header case by leveraging @stdlib/string library.

The frontend of the web app is built using a simple html and css file with minimalistic design. The backend of this project is achieved using npm, NodeJS and fastify. I have added the README file explaining the setting up and usage along with sample screenshots.

Project GitHub URL: stdlib case conversion showcase project

Goals

Developer Dashboard For Tracking Ecosystem Build Failures

Abstract

Maintaining the stability and integrity of the stdlib ecosystem that is composed of over 3,500 repositories may be tiring without any proper tool to identify the faults and work on it. This project proposes the development of a real-time developer dashboard to visualize build status data that is currently stored in PostgreSQL database. This dashboard will feature real-time build and workflow statuses, filterable views based on criteria, quick navigation to the repository, and summarized view of number of downloads(week average), stars, passed and failed job and workflow status. This allows developers to quickly identify and address build failures, ensuring the stability and integrity of the entire ecosystem.

Goals

The goal of this proposal is to create a materialized view table to store the necessary details for the dashboard from different tables. The data will be updated to the materialized view table on a daily basis using Github actions. The detailed architecture diagram for this solution is given below.

Figure 1: Architecture Diagram for Real-time Developer Dashboard

From the architecture diagram, the data will be displayed on the front-end using API calls. Every time, the backend API will make a request call to the materialized view table to fetch data. Requesting the data from a materialized view table is faster than requesting data from different tables at the same time. A trigger will be set for Github actions to update the data in the materialized view table everyday if there are any changes in any of the stdlib’s repositories statuses. This approach is scalable; the load on the database is spread out as the complex queries are done on a schedule rather than on every user request. Also, the frontend can only focus on displaying data and making simple API calls.

Filtering and Sorting

server-side filtering might be the best option for this project due to the large amount of repo data. The structure of the API endpoint would be something like GET /api/workflow-statuses?status=failure for filtering the failure workflow and GET /api/workflow-statuses?sortBy=createDate&sortOrder=asc to sort from latest to oldest workflows based on created date.

Design

The user interface of the developer dashboard will be a single page website. On the first line, it displays the recent data update by the scheduled GitHub actions. On the second line, it provides the search feature to search by project name and whole dashboard summary such as Number of job pass, job fails, workflow pass, workflow fail, total stars and total downloads. On the third line, a table represents a detailed overview of each stdlib’s repositories statuses. This includes repo link, npm package manager link, workflow overall status, workflow created timestamp, overall job status, event type, code coverage percentage, recent commit timestamp with interlink, number of stars, published timestamp, published version, node version, average week downloads and repository size. A clear view of the UI/UX design can be find here: Link

Figure 2: Prototype of Developer Dashboard

Why this project?

This project perfectly aligned with my skillset, which is MERN Stack. The most exciting part about this project is the approach I proposed. Because it is quite an optimized approach to handle data from various tables and summarize it to display in the dashboard.

Qualifications

I am a good fit for this project, because

1. I have similar experience in building a dashboard.
2. I built an npm package manager and published it.
3. I have experience in managing my own open source project as a project admin.

Similar to this project, I designed a Restaurant Management Dashboard that provides an interactive user interface and backend for managing restaurant data effectively. This dashboard displays the summarized statistics of yearly profits, number of visitors, orders received, liked and disliked dishes, user ratings, popular timings and net earning with the help of Chart JS library in React. It displays the menu (type, name, amount, rating), order details (order id, items, total amount, customer, order date) and customer details (customer id, name, order details) from the backend through REST API calls. I’ve used lists, forms, tables and charts for interactivity and Tailwind CSS for the UI design.

Prior art

NPM has a similar dashboard for their developers to track their repositories and build failures: https://npm.github.io/statusboard/.

The idea and approach are mostly similar to it.

Commitment

Since I am on my final year project vacation, I can work on this project full time (from Monday to Friday).
I will summarize my work each week and present it to the mentors during the GSoC. If any other emergency occurs during GSoC, I will intimate the mentors about it with proper reason and will resume work once the issue has been resolved.

Schedule

Assuming a 12 week schedule,

• Community Bonding Period:

Week 1: Foundations & stdlib Familiarization

• Explore the stdlib project structure and its 3500+ repositories.
• Study the existing build result database schema in PostgreSQL.
• Learn about GitHub Actions, particularly the schedule trigger (cron expressions).
• Review basic PostgreSQL concepts and SQL queries.
• Set up a local development environment (Node.js, PostgreSQL).

Week 2: PostgreSQL and Materialized Views (MV) Design

• Analyze the relevant tables and define data aggregation requirements.
• Design the structure of the materialized views for optimal performance.
• Write and test the SQL queries for creating the materialized views.
• Create the materialized views in the development PostgreSQL database.

Week 3: Node.js Backend API Setup

• Initialize a Node.js project with Express.js.
• Install necessary packages (pg, cors, etc.).
• Configure database connection settings.
• Define API endpoints for retrieving data from the materialized views.
• Create a basic API structure.

Week 4: API Endpoint Development (Data Retrieval)

• Write the Node.js code to query the materialized views.
• Format the data for API responses (JSON).
• Implement basic error handling.
• Test the api endpoints using postman.

Week 5: GitHub Actions for MV Refresh

• Create a .github/workflows/refresh-views.yml file.
• Write the Node.js or SQL script to refresh the views.
• Configure GitHub Secrets for database credentials.
• Test the workflow to ensure it refreshes the views correctly.

Week 6: (midterm)

• Submit the backend setup for data retriveal from materialized view.
• Initialize a React, Vue.js, or Svelte project.
• Install necessary frontend libraries (Axios, Tailwind CSS).
• Design the basic layout of the dashboard.
• Create basic components.

Week 7: Frontend Data Fetching and Display

• Write frontend code to fetch data from the API endpoints.
• Display the data in tables or other suitable formats.
• Implement loading states and basic error handling.

Week 8: Frontend Filtering and Sorting

• Add UI elements for filtering and sorting.
• Implement the logic to filter and sort the displayed data.
• Test the filtering and sorting functionality.

Week 9: Enhancements

• Adding summarized reports at the beginning of the dashboard for easy view
• Implement visual indicators for build statuses (colors, icons).
• Add links to repository resources (GitHub, build logs).
• Ensure the dashboard is responsive and user-friendly.

Week 10: Testing and Debugging (End-to-End)

• Test the interaction between the frontend, backend API, and database.
• Debug any issues and ensure data consistency.
• Test the github actions workflow.

Week 11 - 12: Deployment and Documentation

• Deploy the backend API and frontend to a hosting platform.
• Write documentation for the system architecture, API endpoints, and GitHub Actions workflow.
• Create a user manual and developer documentation

Final Week:

• Present the finished product.
• Submitting all the necessary documentation for the project (API docs, README, etc).
• Submitting the GSOC report and writing blogs about my contribution.

Notes:

• The community bonding period is a 3 week period built into GSoC to help you get to know the project community and participate in project discussion. This is an opportunity for you to setup your local development environment, learn how the project's source control works, refine your project plan, read any necessary documentation, and otherwise prepare to execute on your project project proposal.
• Usually, even week 1 deliverables include some code.
• By week 6, you need enough done at this point for your mentor to evaluate your progress and pass you. Usually, you want to be a bit more than halfway done.
• By week 11, you may want to "code freeze" and focus on completing any tests and/or documentation.
• During the final week, you'll be submitting your project.

Related issues

#4

Checklist

• I have read and understood the Code of Conduct.
• I have read and understood the application materials found in this repository.
• I understand that plagiarism will not be tolerated, and I have authored this application in my own words.
• I have read and understood the patch requirement which is necessary for my application to be considered for acceptance.
• I have read and understood the stdlib showcase requirement which is necessary for my application to be considered for acceptance.
• The issue name begins with [RFC]: and succinctly describes your proposal.
• I understand that, in order to apply to be a GSoC contributor, I must submit my final application to https://summerofcode.withgoogle.com/ before the submission deadline.

[Planeshifter]

Thanks for your proposal!

We already have an existing backend server built using Fastify. You can see the database schema here: #4 (comment)

• Were you planning to extend our current backend or adding a separate Express-based one? What are the trade-offs here?
• What specific columns would you include in your materialized view? Is there any additional data we should retrieve?
• How would you go about refreshing the materialized view?
• Would the filtering and sorting be done client- or server-side? Assuming the latter, how would you structure your API endpoints to support the filtering and sorting requirements of the dashboard
• How would you approach the implementation of the search functionality? Do you see any challenges?

[Rupa-Rd]

Thank you for your feedback, @Planeshifter.
I've seen the database schema mentioned in issue #4.

1. I can see that there is an existing backend for adding and extracting data from the Postgres database. We can either extend or create a new backend for this dashboard project. I prefer creating a new backend for this project because of two reasons:
   • Let's say I am using the existing backend (without materialized view), it needs to depend on several API calls from different databases (commit details from commit db, event failure details from events db and so on). Suppose there is any failure that happens during the data updation in the database; it will also affect the dashboard.
   • Let's say I am using the existing backend (with materialized view), the backend needs to do both processes, and it will add extra complexity to it. If any error happens, it might be difficult to find the cause (whether it is because of the dashboard or other reasons).

So, if it has a separate backend, the debugging process and the scheduling process are easier. @kgryte mentioned the fastify and the difficulty of introducing new backends (express and others) during the last office hours; I understand it completely. Well, I have experience in Express, but it won't be difficult for me to learn Fastify and work with it. I will change the tech stack to Fastify and learn it in the meantime.

2. For the columns in the materialized view, I think of including repo link, npm package manager link, workflow overall status, workflow created timestamp, overall job status, event type, code coverage percentage, recent commit timestamp with interlink, number of stars, published timestamp, published version, node version, average weekly downloads, and repository size from different databases. I need mentors input for this to specify the exact columns.

3. Since the changes in repos might happen every day, it would be great to update/refresh the materialized view every day. So, I planned to use GitHub actions to schedule the refreshing (probably at midnight).

4. Yes, server-side filtering might be the best option for this project due to the large amount of repo data. The structure of the API endpoint would be something like GET /api/workflow-statuses?status=failure for filtering the failure workflow and GET /api/workflow-statuses?sortBy=createDate&sortOrder=asc to sort from latest to oldest workflows based on created date.

5. I planned to implement the search functionality on the materialized view database using the query keyword LIKE for searching the repo names. If the developer frequently searches for the repo, maybe it will add some load to the db.

Overall, this approach (materialized view) would be faster compared to requesting data from actual databases and will be easy to debug and extend in the future. I am open to any suggestions and changes to my approach.