My First Personal Project of 2025: Golden Retriever

My First Personal Project of 2025

In 2025, I launched my first personal project. This initiative was not only an attempt at data analysis but also a hands-on exploration of different technology stacks.

Project Link: https://gloden.1funlab.com/dashboard

Background and Motivation

This year, I decided to start investing in Japanese stocks while continuing to monitor U.S. stock market data. To better analyze these markets, I wanted direct access to relevant data from both the Japanese and U.S. markets, enabling me to visualize and analyze it effectively.

However, I discovered that some critical data, such as the average and median PE, EPS, and market cap, were not readily available. While these statistics aren't directly accessible, they can be calculated by collecting raw data and leveraging Python pandas for statistical analysis. This realization was the driving force behind starting this project.

Project Architecture

The project is divided into the following modules:

1. Data Acquisition

• U.S. Stock Data
  • Initial data was sourced from nasdaq.com. However, issues like inaccurate PE data and missing fields arose. A switch to a more reliable source is planned to improve data quality.
• Japanese Stock Data
  • Data was retrieved from Yahoo Finance, which provides a stable and accurate data interface.
• Technical Implementation
  • The data acquisition scripts were developed in Python, utilizing:
    • httpx: For asynchronous requests, boosting scraping efficiency.
    • selenium: For handling dynamically loaded web pages.
  • Raw data is stored in SQLite for quick access and management.

2. Data Cleaning and Analysis

• Data cleaning and analysis are handled using Python tools:
  • pandas: For data manipulation and analysis.
  • numpy: For scientific calculations.
  • seaborn and matplotlib: To generate simple local analysis charts.
• Final Data Storage
  • Cleaned and aggregated data is stored in MongoDB for efficient querying and utilization.

3. API Development

• Technology Stack
  • APIs are built using Golang and the Echo framework.
  • Given the project's small API footprint, Golang's performance and simplicity made it a suitable choice.
• Features
  • APIs are designed to be lightweight, enabling quick data retrieval and seamless front-end integration.

4. Front-End Development

• Technology Stack
  • Built with Next.js and TypeScript.
  • UI Frameworks: MUI and Tailwind CSS.
• Development Insights
  • Having started using this front-end stack last year, I am now comfortable applying it to mid-sized projects. Leveraging Next.js's static generation and server-side rendering (SSR), I achieved a balance between performance and development efficiency.

5. Data Visualization

• Charting Tool: Highcharts
  • Why Highcharts?
    • I’ve been using Highcharts for over 10 years. While I’ve tried other libraries like ECharts and D3, Highcharts remains my top choice for standard charts due to its ease of use and robust feature set.
• Use Cases
  • Visualizing analysis results from both U.S. and Japanese stock markets with interactive, dynamic charts.

6. Continuous Integration and Deployment (CI/CD)

• Tools
  • Both the front-end and API utilize GitHub Actions for seamless CI/CD.
• Additional Features
  • Integrated GitHub Slack Action to send Slack notifications upon deployment, ensuring real-time project status tracking.

Future Plans

Looking ahead, I plan to introduce more features and technologies, including:

• Data Source Optimization
  • Improve data quality and streamline the acquisition process.
• Automated Analysis
  • Incorporate machine learning models to make simple predictions based on historical data.
• Exploration of New Tech Stacks
  • Experiment with Rust, Golang, or other emerging technologies for more efficient development.

This project's experiences will undoubtedly provide valuable insights and motivation for future endeavors.

Next up: I plan to rewrite my blog project using Rust!