As Solidsight matured, accuracy became paramount. One key improvement was ensuring generated Mermaid diagrams were accurate and meaningful, addressing the occasional production of invalid placeholder diagrams by the LLM.

Validation and Regeneration Logic

I developed a function to validate generated diagrams:

def is_valid_mermaid(diagram):
    return (diagram.strip().startswith(("flowchart TD", "sequenceDiagram"))
            and "Default Diagram" not in diagram)

When invalid diagrams were detected, the tool would prompt users to regenerate them using the original contract code.

Persistent Challenges Solved

Previously, invalid diagrams went unnoticed, causing confusion for users. With this validation step, such diagrams were caught and addressed immediately, ensuring consistently high-quality outputs.

Final Reflections

Implementing robust validation and regeneration was perhaps one of the most impactful improvements. It highlighted the critical importance of quality assurance processes, ultimately turning Solidsight into a dependable, polished tool suitable for serious contract analysis.

Where can you find it?

I took the to Github as a fellow SR on X showed some interest in using it, and the app made is now available for all to use and adapt to their liking. The overall code isn’t complex, I used AI and made adjustments where needed (it speeds up the process) - Manually creating is fun, but, my time is spent in codebases looking for bugs, I try not to spend too much time building unless its completely necessary.

I also changed the name to DeepCurrent, I like Solidsight, but, as a final version 1, this was the choice.

Have fun & happy hunting: App: DeepCurrent

As Solidsight's capabilities expanded, managing historical analysis data became increasingly important. The next logical step was introducing structured session management, allowing users to revisit previous analyses easily.

Structured Session Browsing

The session management system organised analyses neatly by timestamped directories. Users could browse and revisit any historical data effortlessly:

pythonCopyEditdef browse_sessions():
    sessions = [d for d in os.listdir(os.getcwd()) if d.startswith("analysis_")]
    if not sessions:
        console.print("[bold red]No analysis sessions found.[/bold red]")
        return
    table = Table(title="Analysis Sessions")
    for idx, session in enumerate(sessions, start=1):
        table.add_row(f"{idx}", session)
    console.print(table)

Navigating Complexities

Managing file paths, ensuring consistency in data storage, and handling incomplete analysis sessions were initial obstacles. The early implementations faced occasional crashes due to mislabelled files or corrupted data.

Insights Gained

By addressing these challenges, I learned the significance of comprehensive file validation and clear storage architecture. Effective session management ultimately allowed users to benefit more fully from historical analyses, making Solidsight highly practical.

We will stop here, and I’ll see you on the next one.

pxng0lin.

Smart contract analyses can vary greatly in complexity and purpose. Realising this, I added support for multiple LLM models, allowing users to tailor analyses according to specific needs. This flexibility transformed Solidsight into a much more versatile tool.

Multi-model Support

I introduced a simple yet effective model-selection prompt using Rich:

pythonCopyEditanalysis_model = Prompt.ask(
    "Select your analysis model", 
    choices=["deepseek-r1", "qwen2.5-coder:3b", "gemma3:4b"], 
    default="deepseek-r1"
)

Challenges in Implementation

Different models required slight prompt adjustments due to their varied interpretations and output formats. It took careful prompt engineering and extensive testing to achieve consistency across models.

Lessons Learned

The value of modular design became clear—flexibility allowed Solidsight to adapt easily to varying analysis requirements.

We will stop here, and I’ll see you on the next one.

pxng0lin.

Versions 4 and 5 saw significant strides in workflow optimisation, making the SFA more practical and user-friendly. Efficiency and redundancy elimination became central to these iterations.

Duplicates

A key feature introduced was a robust duplicate-checking mechanism, significantly cutting down on redundant processing:

def report_already_exists(report_id: str, db_path="vectorisation.db") -> bool:
    conn = sqlite3.connect(db_path)
    c = conn.cursor()
    c.execute("SELECT 1 FROM reports WHERE id=?", (report_id,))
    exists = c.fetchone()
    conn.close()
    return bool(exists)

A Little Extra

I also implemented substantial interface improvements, such as clearer feedback with progress bars and advanced logging systems, allowing easier debugging and providing greater transparency during processing tasks.

These improvements significantly enhanced the user experience, making the system easier and more intuitive to operate.

A short read, but progress! See you soon.

pxng0lin

Although Solidsight was becoming powerful, its plain text outputs lacked readability. To address this, I integrated the Rich library, significantly enhancing the visual appeal and clarity of the command-line interface.

Implementing Rich

Rich transformed simple outputs into coloured, visually structured messages and tables:

pythonCopyEditfrom rich.console import Console
console = Console()

console.print("[bold green]Analysis Complete![/bold green]")

Errors and Compatibility Issues

Initially, Rich caused display anomalies in certain terminals. Compatibility across different terminal emulators required additional tweaking.

Reflections

Adding Rich significantly improved user interaction, facilitating clearer communication of results and easier debugging during development.

We will stop here, and I’ll see you on the next one.

pxng0lin.

Following the initial success, versions 2 and 3 of the SFA were dedicated to improving stability, refining database functionality, and expanding data handling capabilities. This phase was crucial in ensuring that the system was robust enough to handle increasing complexity.

Improving Usability

Significant improvements involved enhancing the database to store structured JSON data for metadata, vector embeddings, and detailed AI-generated analysis summaries. This allowed queries to become significantly more flexible and efficient, enhancing overall usability.

Additionally, refining how Markdown content was split into sections greatly improved the precision of the analysis. The upgraded section-splitting function allowed the SFA to accurately handle diverse document structures, significantly improving the effectiveness of embedding and subsequent analysis:

def split_into_sections(text: str) -> list:
    sections = re.split(r'\n(?=#)', text)
    results = []
    for sec in sections:
        sec = sec.strip()
        if sec.startswith("#"):
            lines = sec.splitlines()
            heading = lines[0].lstrip('#').strip()
            content = "\n".join(lines[1:]).strip()
            results.append({"title": heading, "content": content})
        else:
            if sec:
                results.append({"title": "Introduction", "content": sec})
    return results

Learnings

Versions 2 and 3 highlighted critical areas for further development, particularly regarding duplicate detection and more robust error management.

See you in the next one

pxng0lin

I realised early on that functionality alone wasn't enough; Solidsight needed an intuitive way for users to interact with generated reports. This led to the development of interactive command-line menus, providing users with easy navigation options and clear choices.

Interactive Menu Implementation

The new menu allowed users to select specific analyses or regenerate outputs as needed:

pythonCopyEditdef main_menu(output_dir):
    while True:
        print("\n--- Main Menu ---")
        reports = [f for f in os.listdir(output_dir) if f.endswith("_analysis_report.md")]
        if not reports:
            print("No reports found in this directory.")
            break
        for idx, report in enumerate(reports, start=1):
            print(f"{idx}. {report.replace('_analysis_report.md', '')}")
        choice = input("Select a report by number or type 'exit': ")
        if choice.lower() == 'exit':
            break
        # additional logic here

Navigating Initial Errors

Initially, the menu would crash if no reports existed or when invalid inputs were entered. It became clear that thorough input validation and better user prompts were essential.

Lessons Learned

Addressing these user-experience issues taught me to anticipate user behaviours better and ensure the program remained stable, irrespective of how it was used.

We will stop here, and I’ll see you on the next one.

pxng0lin.

When I started building Solidsight, the idea was straightforward yet ambitious: automate the process of reviewing Solidity smart contracts by leveraging local Large Language Models (LLMs). I envisioned an app that could quickly dissect contracts and produce clear, informative reports detailing their functionality, potential vulnerabilities, and user interactions.

Building the Core Functionality

The initial goal was simple: parse Solidity files, generate a detailed functions report, describe the typical user journey, and visualise the function interactions using Mermaid diagrams.

Here’s the basic API interaction function I started with:

pythonCopyEditdef call_llm(prompt):
    headers = {"Content-Type": "application/json"}
    payload = {
        "model": "deepseek-r1",
        "prompt": prompt,
        "temperature": 0.7,
        "max_tokens": 8000
    }
    try:
        response = requests.post(LLM_API_URL, json=payload, headers=headers)
        response.raise_for_status()
        return response.json()["choices"][0]["text"].strip()
    except Exception as e:
        print(f"LLM API call failed: {e}")
        return ""

Challenges Encountered

The early versions struggled significantly with handling large Solidity files, often hitting token limits or experiencing API timeouts. Error handling at this stage was minimal, causing unexpected failures that required restarting the entire analysis—a frustrating user experience.

Reflections and Improvements

From these initial setbacks, I learned how crucial robust error handling was. Implementing structured error catching and refining my prompts reduced crashes and improved reliability, paving the way for more ambitious developments.

We will stop here, and I’ll see you on the next one.

pxng0lin.

Why SFA?

The tedious, repetitive nature of security auditing, especially in the blockchain space, convinced me there must be a better way. Traditionally, this process involved manual reading of lengthy Markdown audit reports, cross-referencing vulnerability databases, and performing in-depth smart contract audits—all of which were time-consuming and error-prone.

I aimed for a system that could:

• Automatically download and process Markdown-formatted security audit reports from GitHub.

• Compute semantic embeddings for quick and effective information retrieval.

• Store structured data locally in a SQLite database to ensure privacy and autonomy.

• Utilise a local language model (via Ollama) to summarise vulnerabilities and suggest mitigations in real-time.

Building a Strong Base

I began by setting up a local SQLite database with tables dedicated to reports and code audits:

pythonCopyEditdef init_db(db_path="vectorisation.db"):
    conn = sqlite3.connect(db_path)
    c = conn.cursor()
    c.execute('''
        CREATE TABLE IF NOT EXISTS reports (
            id TEXT PRIMARY KEY,
            source TEXT,
            content TEXT,
            overall_embedding TEXT,
            section_embeddings TEXT,
            analysis_summary TEXT,
            metadata TEXT
        )
    ''')
    conn.commit()
    conn.close()

This database ensured data integrity and made future data retrieval straightforward.

Early Challenges and Solutions

One initial challenge was handling GitHub Markdown URLs. Many links pointed to files that needed conversion from "blob" URLs to "raw" URLs for direct downloading. To handle this, I created a helper function:

pythonCopyEditdef convert_to_raw_url(url: str) -> str:
    if "github.com" in url and "/blob/" in url:
        return url.replace("github.com", "raw.githubusercontent.com").replace("/blob/", "/")
    return url

Initial Results

Although version 1 lacked sophisticated error handling, it successfully demonstrated that the system could autonomously download Markdown files, compute embeddings using the sentence-transformers library, and store structured data locally. While rudimentary, the initial results were promising, affirming that the core concept of a fully local, AI-driven tool was viable.

So, that makes a start, see you in the next one.

pxng0lin.

In the last article, I talked about how I collaborated with ChatGPT to introduce new features to my trading model, such as Recursive Feature Elimination (RFE), XGBoost, and hyperparameter tuning. With these enhancements, the next challenge was ensuring that my trading script could properly make use of the newly trained model. In this article, I'll discuss how I worked with ChatGPT to overcome the challenges of integrating the updated model into the trading logic.

Adapting the Trading Script

With the upgraded model featuring 17 new indicators, including the Average Directional Index (ADX), Momentum (MOM), and Rate of Change (ROC), it became apparent that my existing trading script needed significant changes to handle the expanded feature set. This was not just a simple update; it required a comprehensive alignment between the feature engineering used during model training and the real-time data preparation for trading decisions.

One of the most crucial aspects was ensuring consistency in feature names and their order. I noticed that, when the trading script tried to make predictions using real-time market data, it frequently ran into errors because the features did not match those expected by the model. The model would raise an error due to either missing features or incorrect ordering, resulting in frustrating setbacks.

After explaining these challenges to ChatGPT, it proposed a solution to ensure feature alignment throughout the scripts. Here's a snippet showing how I extracted and prepared the features for real-time predictions:

# Extracting features in the same order as during training
features_df = pd.DataFrame([features], columns=FEATURE_COLUMNS)
features_df = features_df.fillna(0)  # Handling any missing values

# Making prediction
prediction_proba = model.predict_proba(features_df)
buy_proba = prediction_proba[0][1]
sell_proba = prediction_proba[0][0]

Handling Feature Order and Consistency

To maintain consistency, I had to revisit the feature engineering steps I used during model training and replicate them exactly in the trading script. This meant incorporating calculations for all 17 features, such as the Exponential Moving Average (EMA), Bollinger Bands, and others, ensuring the order was identical.

Below is a snippet that shows the feature calculation in real-time within the trading script, which mirrors what was done during training:

# Calculate features for real-time data
latest_data['ema50'] = talib.EMA(latest_data['close'], timeperiod=50)
latest_data['ema200'] = talib.EMA(latest_data['close'], timeperiod=200)
latest_data['adx'] = talib.ADX(latest_data['high'], latest_data['low'], latest_data['close'], timeperiod=14)
latest_data['mom'] = talib.MOM(latest_data['close'], timeperiod=10)
# ... calculate other features as well

Once I had ensured the features were calculated consistently, ChatGPT suggested a way to handle any potential discrepancies by checking for missing or unordered features and then adjusting accordingly. This helped prevent runtime errors and allowed the model to predict as expected.

Managing Recent Market Activity

Another improvement ChatGPT helped me make was incorporating recent market activity to improve the accuracy of predictions. The idea was to add a "run rate" feature that considered recent volume trends and volatility. This helped the bot become more context-aware, allowing it to factor in the recent momentum of the market when making decisions. Here’s a simplified example of how this was integrated:

# Calculate recent volume trend
latest_data['volume_change'] = latest_data['volume'].pct_change(periods=3)

# Calculate rolling volatility
latest_data['volatility'] = latest_data['close'].rolling(window=10).std()

These additional metrics allowed the model to weigh recent market conditions more heavily, giving it a better edge in predicting buy and sell opportunities.

Overcoming Errors and Building Robustness

Throughout this process, I faced multiple errors. The infamous "feature mismatch" error returned multiple times, and the challenge of handling NaN values or incomplete data meant I had to be meticulous. ChatGPT was instrumental in suggesting techniques such as filling missing values with zeros or calculating forward fill methods to handle incomplete data, ensuring no features were left undefined.

Here is an example of how I handled missing values to avoid model errors:

# Handle missing values
features_df = features_df.fillna(0)  # Replace NaN values with zero to avoid prediction errors

ChatGPT's approach to problem-solving here was iterative: each time an error was encountered, it would analyse the traceback, suggest corrections, and propose enhancements. It felt like a collaborative debugging session, and over time, these iterative fixes made my bot much more resilient to runtime issues.

The Result

After many rounds of testing and troubleshooting, the new trading script, combined with the enhanced model, was finally able to operate smoothly. The bot could now consistently make predictions using all 17 features, providing a more comprehensive analysis of the market. The integration of recent market activity indicators added an extra layer of context, further improving the bot's decision-making abilities.

The improvements in prediction accuracy were evident—the bot was now making fewer false-positive trades and identifying profitable opportunities with more reliability. The journey was challenging, but having ChatGPT as a guide made all the difference, providing insights and coding solutions whenever I encountered roadblocks.

Stay tuned for the next article, where I'll share how I worked on optimizing the bot's transaction logic, minimizing transaction costs, and further improving overall profitability.

As I continue my journey to enhance my trading bot, I've reached a significant milestone—the integration of new machine learning features into my model. This has been both an exciting and challenging phase, where the focus has been on improving the prediction accuracy of buy and sell signals. In this post, I'll share the details of how I worked with ChatGPT to incorporate Recursive Feature Elimination (RFE), XGBoost, and hyperparameter tuning, and what these changes mean for the bot's performance.

Integrating RFE, XGBoost, and Hyperparameter Tuning

The journey started with a realization: the existing model's feature set wasn't providing enough granularity for accurate trading decisions. To address this, I asked ChatGPT to suggest improvements. It decided to add new features and apply RFE to determine which ones were most important. The goal was simple—let's keep only what truly matters. This led to an extensive feature engineering process that added indicators like Average Directional Index (ADX), Rate of Change (ROC), Momentum (MOM), and many others.

Once these features were ready, ChatGPT used RFE to identify which ones significantly contributed to model performance. This reduced the feature set to the most relevant indicators, which in turn helped improve model training efficiency and reduce overfitting.

Here's a snippet showing how RFE was applied to the model:

from sklearn.feature_selection import RFE
from sklearn.ensemble import RandomForestClassifier

# Initializing the model for feature selection
model_for_rfe = RandomForestClassifier(random_state=42)

# Selecting top 10 features using RFE
rfe = RFE(model_for_rfe, n_features_to_select=10)
X_train_rfe = rfe.fit_transform(X_train, y_train)

Next came XGBoost—a powerful tool in the machine learning arsenal. I asked ChatGPT if it could improve the model further, and it suggested experimenting with XGBoost alongside the traditional Random Forest approach, evaluating which performed better with the trading dataset. It also recommended hyperparameter tuning, using GridSearchCV to test different combinations for the Random Forest model and find the optimal setup. The resulting model showed noticeable improvements in performance metrics, boosting the accuracy of predictions and helping with more informed trading decisions.

Here's an example of how GridSearchCV was used for hyperparameter tuning:

from sklearn.model_selection import GridSearchCV

param_grid = {
    'n_estimators': [100, 200],
    'max_depth': [10, 20],
    'min_samples_split': [2, 5],
}

rf = RandomForestClassifier(random_state=42)
grid_search = GridSearchCV(rf, param_grid, cv=3, n_jobs=-1, verbose=2)
grid_search.fit(X_train_rfe, y_train)

# Best model after hyperparameter tuning
best_rf = grid_search.best_estimator_

Challenges Faced and Lessons Learned

This process wasn't without its challenges. Integrating these new features meant dealing with various compatibility issues between the training script and the prediction logic in the trading script. For example, after training the model with 17 features, I encountered repeated errors when using the model to make predictions in real-time trading.

One of the most frustrating errors was the infamous "feature mismatch" problem. The model expected a specific order and set of features, but the data I was providing during predictions was either incorrectly ordered or incomplete.

To solve this, ChatGPT suggested ensuring strict consistency between the feature set used during training and the real-time feature extraction in the trading script. This was a valuable lesson—the importance of keeping feature engineering consistent across all stages of model development.

Here's how I ensured feature consistency during prediction:

# Extracting features in the same order as during training
features_df = pd.DataFrame([features], columns=FEATURE_COLUMNS)
features_df = features_df.fillna(0)

# Making prediction
prediction_proba = model.predict_proba(features_df)
buy_proba = prediction_proba[0][1]
sell_proba = prediction_proba[0][0]

The Impact on Prediction Accuracy

These changes have had a substantial impact on the bot's performance. The new features have provided more depth for understanding market movements. RFE and XGBoost allowed us to focus on what's truly important while hyperparameter tuning made the model leaner and more effective. In practice, this means the bot is now better at identifying potential opportunities and minimizing false signals—a crucial aspect of successful algorithmic trading.

This stage of development marked a turning point, where the focus shifted from simply making predictions to making accurate, reliable predictions that could translate into profitable trades. While the journey is far from over, I'm excited about the improvements so far and ready to tackle the next challenges that arise.

Stay tuned for the next update, where I'll share how ChatGPT helped me align the trading script to properly consume the upgraded model features and how I managed to make it all work together seamlessly.

pxng0lin.

Now that the data was enriched with technical indicators and lag features, it was time to build a predictive model to forecast stock movements.

Building the Predictive Model

The AI assistant suggested using XGBoost, a powerful and efficient gradient boosting algorithm that's well-suited for tabular data.

What is XGBoost in Machine Learning?

XGBoost, or eXtreme Gradient Boosting, is a XGBoost algorithm in machine learning algorithm under ensemble learning. It is trendy for supervised learning tasks, such as regression and classification. XGBoost builds a predictive model by combining the predictions of multiple individual models, often decision trees, in an iterative manner.

The algorithm works by sequentially adding weak learners to the ensemble, with each new learner focusing on correcting the errors made by the existing ones. It uses a gradient descent optimization technique to minimize a predefined loss function during training.

Key features of XGBoost Algorithm include its ability to handle complex relationships in data, regularization techniques to prevent overfitting and incorporation of parallel processing for efficient computation.

source: What is the XGBoost algorithm and how does it work? (analyticsvidhya.com)

from xgboost import XGBClassifier

def prepare_data(data):
    # Define the target variable
    data['Future_Return'] = (data['Close'].shift(-1) - data['Close']) / data['Close']
    data['Target'] = (data['Future_Return'] > 0).astype(int)
    data.dropna(inplace=True)

    # Select features
    features = data.drop(['Target', 'Future_Return', 'Close'], axis=1).columns
    X = data[features]
    y = data['Target']
    return X, y

from sklearn.model_selection import TimeSeriesSplit

def train_model(X, y):
    tscv = TimeSeriesSplit(n_splits=5)
    model = XGBClassifier(use_label_encoder=False, eval_metric='logloss')
    for train_index, test_index in tscv.split(X):
        X_train, X_test = X.iloc[train_index], X.iloc[test_index]
        y_train, y_test = y.iloc[train_index], y.iloc[test_index]
        model.fit(X_train, y_train)
    return model

Train and Test Data

When building a machine learning model, it’s crucial to evaluate its performance on unseen data. This is where the concepts of train and test data come into play.

1. Training Data:

   • Purpose: Used to train the model.

   • Process: The model learns patterns, relationships, and features from this data.

   • Example: If you have a dataset of house prices, the training data would include features like the number of rooms, location, and the corresponding house prices.

2. Test Data:

   • Purpose: Used to evaluate the model’s performance.

   • Process: After training, the model makes predictions on the test data, and these predictions are compared to the actual values to assess accuracy.

   • Example: Continuing with the house prices example, the test data would also include features like the number of rooms and location, but the model would predict the house prices, which are then compared to the actual prices.

Using XGBoost with Train and Test Data

Here’s a quick example guide to using XGBoost with train and test data:

1. Import Libraries:

    import xgboost as xgb
    from sklearn.model_selection import train_test_split
    from sklearn.metrics import accuracy_score
   

2. Load and Prepare Data:

    # Example using a dataset
    from sklearn.datasets import load_breast_cancer
    data = load_breast_cancer()
    X = data.data
    y = data.target
   

3. Split Data into Train and Test Sets:

    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
   

4. Train the XGBoost Model:

    model = xgb.XGBClassifier()
    model.fit(X_train, y_train)
   

5. Make Predictions on Test Data:

    y_pred = model.predict(X_test)
   

6. Evaluate the Model:

    accuracy = accuracy_score(y_test, y_pred)
    print(f"Accuracy: {accuracy * 100:.2f}%")
   

Why Split Data?

• Avoid Overfitting: By evaluating the model on unseen data (test data), you can ensure it generalises well and isn’t just memorising the training data.

• Model Validation: It helps in validating the model’s performance and tuning hyperparameters effectively.

Conclusion

Using train and test data is essential for building robust machine learning models. XGBoost, with its powerful capabilities, can efficiently handle this process, ensuring high performance and accuracy.

Resources: How to train XGBoost models in Python (youtube.com)

Handling Overfitting

• Hyperparameter Tuning: Adjusting parameters like max_depth, n_estimators, and learning_rate to find the optimal combination.

• Cross-Validation: Using TimeSeriesSplit to perform cross-validation that respects the temporal order.

• Regularisation: Adding regularisation parameters like reg_alpha and reg_lambda to penalise complex models.

from sklearn.model_selection import RandomizedSearchCV

def train_model_with_cv(X, y):
    model = XGBClassifier(use_label_encoder=False, eval_metric='logloss')
    param_grid = {
        'n_estimators': [50, 100, 150],
        'max_depth': [3, 5, 7],
        'learning_rate': [0.01, 0.05, 0.1],
        'subsample': [0.8, 1.0],
        'colsample_bytree': [0.8, 1.0],
        'reg_alpha': [0, 0.1, 0.5],
        'reg_lambda': [1, 1.5, 2]
    }
    tscv = TimeSeriesSplit(n_splits=5)
    grid_search = RandomizedSearchCV(model, param_grid, cv=tscv, scoring='accuracy', n_iter=10)
    grid_search.fit(X, y)
    return grid_search.best_estimator_

This approach improved the model's generalisation to unseen data.

Handling Class Imbalance

What is SMOTE?

SMOTE is a technique used to create synthetic samples for the minority class in a dataset. This helps balance the class distribution, which is crucial for training machine learning models effectively on imbalanced data.

How Does SMOTE Work?

1. Identify Minority Class Samples: SMOTE starts by identifying the samples in the minority class.

2. Generate Synthetic Samples: It then generates new synthetic samples by interpolating between existing minority class samples. This is done by selecting two or more similar instances and creating a new instance that lies between them in the feature space.

3. Add Synthetic Samples to Dataset: These synthetic samples are added to the dataset, resulting in a more balanced class distribution.

Benefits of SMOTE

• Improves Model Performance: By balancing the dataset, models can learn better and perform more accurately on the minority class.

• Reduces Overfitting: Unlike simple oversampling (which duplicates minority class samples), SMOTE reduces the risk of overfitting by creating new, unique samples.

from imblearn.over_sampling import SMOTE

def balance_classes(X, y):
    smote = SMOTE(random_state=42)
    X_resampled, y_resampled = smote.fit_resample(X, y)
    return X_resampled, y_resampled

After balancing the classes, the model's performance improved significantly.

Encountered Error:

While training the model, I ran into an error:

ValueError: could not convert string to float: '2024-03-07'.

def select_numeric_features(data):
    numeric_cols = data.select_dtypes(include=[np.number]).columns
    return data[numeric_cols]

By selecting only numeric features, we eliminated the error.

Results so far

Excellent, so we have a working bot, with analysis, back-testing and predictions, but, the accuracy is quite low, so this will be my next item to work on (or rather AI to work on).

pxng0lin.

With the stock data successfully fetched and initial error handling in place, it was time to delve deeper into the data to make it more informative for our predictive models. The idea was to enrich the data with technical indicators—tools that traders use to analyse past market data to predict future price movements.

Calculating Technical Indicators

The AI assistant suggested utilising the ta library, a comprehensive technical analysis library in Python. This library provides a wide range of technical indicators ready to be used with minimal setup.

We aimed to calculate several key indicators:

• Simple Moving Average (SMA)

• Exponential Moving Average (EMA)

• Relative Strength Index (RSI)

• Moving Average Convergence Divergence (MACD)

• Bollinger Bands

• Average True Range (ATR)

• On-Balance Volume (OBV)

Here's how we implemented the function to add these indicators:

import ta

def add_technical_indicators(data):
    # Simple Moving Average
    data['SMA'] = ta.trend.SMAIndicator(data['Close'], window=14).sma_indicator()

    # Exponential Moving Average
    data['EMA'] = ta.trend.EMAIndicator(data['Close'], window=14).ema_indicator()

    # Relative Strength Index
    data['RSI'] = ta.momentum.RSIIndicator(data['Close'], window=14).rsi()

    # MACD
    macd = ta.trend.MACD(data['Close'])
    data['MACD'] = macd.macd()
    data['MACD_Signal'] = macd.macd_signal()

    # Bollinger Bands
    bb = ta.volatility.BollingerBands(data['Close'], window=20, window_dev=2)
    data['BB_High'] = bb.bollinger_hband()
    data['BB_Low'] = bb.bollinger_lband()

    # Average True Range
    data['ATR'] = ta.volatility.AverageTrueRange(data['High'], data['Low'], data['Close'], window=14).average_true_range()

    # On-Balance Volume
    data['OBV'] = ta.volume.OnBalanceVolumeIndicator(data['Close'], data['Volume']).on_balance_volume()

    # Drop initial rows with NaN values
    data.dropna(inplace=True)
    return data

data.dropna(inplace=True)

This adjustment ensured that the dataset was free of missing values and ready for further analysis.

Additional Consideration: To make the dataset even richer, the AI suggested adding more technical indicators like Momentum, Chaikin Money Flow (CMF), and Money Flow Index (MFI).

# Momentum
data['Momentum'] = data['Close'].diff(4)

# Chaikin Money Flow
data['CMF'] = ta.volume.ChaikinMoneyFlowIndicator(
    high=data['High'], low=data['Low'], close=data['Close'], volume=data['Volume'], window=20
).chaikin_money_flow()

# Money Flow Index
data['MFI'] = ta.volume.MFIIndicator(
    high=data['High'], low=data['Low'], close=data['Close'], volume=data['Volume'], window=14
).money_flow_index()

Feature Engineering with Lag Features

To capture temporal dependencies and provide the model with more context, we decided to create lag features. Lag features are previous time steps' values of a time series, which can help the model understand how past values influence future ones.

Temporal dependencies refer to the relationships and patterns between data points in a time series, where the value at a given time is influenced by its previous values. These dependencies are crucial in time series analysis and forecasting, as they help in understanding how past events affect future outcomes.

def create_lag_features(data, numeric_cols, lags):
    for col in numeric_cols:
        for lag in lags:
            data[f'{col}_lag{lag}'] = data[col].shift(lag)
    data.dropna(inplace=True)
    return data

We applied this function to our data, specifying the numeric columns and the number of lags we wanted to create (e.g., 1, 2).

Multicollinearity refers to a statistical phenomenon in which two or more predictor variables in a multiple regression model are highly correlated, meaning that one can be linearly predicted from the others with a substantial degree of accuracy. This can cause problems in estimating the coefficients of the regression model, leading to unreliable and unstable estimates. Multicollinearity can inflate the variance of the coefficient estimates and make it difficult to determine the individual effect of each predictor variable on the dependent variable.

def remove_highly_correlated_features(data, threshold=0.9):
    corr_matrix = data.corr().abs()
    upper_tri = corr_matrix.where(np.triu(np.ones(corr_matrix.shape), k=1).astype(bool))
    to_drop = [column for column in upper_tri.columns if any(upper_tri[column] > threshold)]
    data.drop(columns=to_drop, inplace=True)
    return data

Implementing this function helped in reducing the feature set to a more manageable size while retaining the most informative variables.

So far so good! I started to push the data out to my Telegram, bearing in mind, the entirety of the code is generated each time, so the model works and produces results. Changes come from me noticing issues with the output or errors that come up in the terminal.

Output of signals to Telegram

That’s it for this week, see you on the next one.

pxng0lin.

With the idea in place and my setup ready, it was time to start coding—or, more accurately, instructing AI to code for me.

Getting the Data

I needed historical stock data. The AI suggested using the yfinance library, which is a reliable source for stock market data.

import yfinance as yf

def get_stock_data(ticker, interval='1d', period='5y'):
    stock = yf.Ticker(ticker)
    data = stock.history(interval=interval, period=period)
    return data

It decided to fetch data for companies in the S&P 500's, I added a refinement of technology, energy and utilities sectors. It used Wikipedia's list and extracted the tickers I was interested in.

After running the initial version, I realized that for some companies, especially smaller ones or those less actively traded, the data returned was sparse or even nonexistent. This would obviously create issues for the machine learning models down the line.

import logging

def get_stock_data(ticker, interval='1d', period='5y'):
    stock = yf.Ticker(ticker)
    try:
        data = stock.history(interval=interval, period=period)
        if data.empty:
            logging.warning(f"No data for {ticker}")
            return None
        return data
    except Exception as e:
        logging.error(f"Error fetching data for {ticker}: {e}")
        return None

This modification allowed the script to log warnings or errors for tickers with issues and proceed with the rest, enhancing the robustness of the data fetching process.

Asynchronous Data Fetching

Fetching data for multiple tickers sequentially was time-consuming. The AI assistant recommended using asynchronous programming with asyncio to speed up the process.

import asyncio

async def fetch_all_data(tickers, interval='1d', period='5y'):
    data = {}
    for ticker in tickers:
        stock_data = await asyncio.to_thread(get_stock_data, ticker, interval, period)
        if stock_data is not None:
            data[ticker] = stock_data
    return data

Corrected Code:

pythonCopy codeasync def fetch_all_data(tickers, interval='1d', period='5y'):
    tasks = [asyncio.to_thread(get_stock_data, ticker, interval, period) for ticker in tickers]
    results = await asyncio.gather(*tasks)
    data = {ticker: result for ticker, result in zip(tickers, results) if result is not None}
    return data

This adjustment fixed the error, allowing for efficient, concurrent data fetching.

Scheduling with crontab

To automate the bot's execution, I used crontab on my Linode instance to schedule it to run daily. This way, the bot would fetch new data and perform analysis every day without manual intervention.

codecrontab -e
# Add the following line to run the bot every day at 00:10 AM
10 0 * * * /usr/bin/python3 /path/to/the/bot.py

This ensures that the bot fetches new data and performs analysis every day without manual intervention.

INFO:__main__:Telegram message sent.
INFO:__main__:Analyzing 91 tickers.
INFO:__main__:Fetched data for AKAM from yfinance, Data Shape: (262, 7)
INFO:__main__:Fetched data for ADBE from yfinance, Data Shape: (262, 7)
INFO:__main__:Fetched data for ACN from yfinance, Data Shape: (262, 7)
INFO:__main__:Fetched data for AMD from yfinance, Data Shape: (262, 7)
INFO:__main__:Fetched data for APH from yfinance, Data Shape: (262, 7)
INFO:__main__:Combined data for AKAM, Data Shape: (262, 139)
INFO:__main__:Combined data for ACN, Data Shape: (262, 142)
INFO:__main__:Combined data for APH, Data Shape: (262, 144)
INFO:__main__:Combined data for AMD, Data Shape: (262, 143)
INFO:__main__:Combined data for ADBE, Data Shape: (262, 143)

Preference

I run a few things on a schedule, but, particularly with my own bot for a similar concept, I use the .sh file to run the script. This is so I can activate the Python Virtual Environment before running it.

There may be better ways of doing it, but, like I mentioned, I’m not a developer, I just dabble, so I usually fall back to scripts, things I used to create whilst web2 ethical hacking.

#!/bin/bash
# change directory
cd /home/REDACTED/REDACTED/testing
# activate the virtual environment
source /home/REDACTED/REDACTED/testing/.TEST/bin/activate
# run the python script
python3 /home/REDACTED/REDACTED/REDACTED/bot_24_draft.py
# Deactivate the virtual environment
deactivate

We will stop here, and I’ll see you on the next one.

pxng0lin.

In this series of articles, I'm going to take you through a little journey that I've decided to label as 'NotADev'.

Why, you may ask? Well, I've dabbled in Python scripts for some years whilst doing web2 ethical hacking and recently transitioning into web3 security research. Many moons ago, I was an analyst, building propensity models, forecasting, and performing various methods of customer analysis. So, I still have an interest in data, algorithms, and statistics.

Lately, I've tried my hand at trading, both in crypto and stocks & shares—only casually—but I have my hand in where I can on the stock market.

Over the past few months, I've caught the bug to do some dev work, starting off by building things myself and utilising AI for new ideas or code improvements. Then, Replit came out with Replit Agent, and I jumped at the opportunity to build some apps based on ideas I'd had, and I was truly amazed! Fast forward to today, I'm still using it, but I came across the new 'ChatGPT o1 preview' model and thought I'd test it to improve my already AI-written analysis bot for the stocks and shares market..

The idea.

I wanted to come up with a way of trading on the stock market in an automated fashion—saving me time analysing, monitoring prices, and buying or selling, all the usual stuff. So, I aimed to build a bot that would do this and then relay the results back to me via message, hoping to get the bot to trade for me too eventually.

Using free resources for data, ChatGPT (paid) for code generation, Telegram API for sending results, and some personal preferences, I began the quest to build a bot to run in the cloud

Requirements.

• Automated extraction of historical stock market data

  • Filtered for the technology and energy industries only

  • Auto selection of most favourable tickers

• Incorporate trading indicators for signals

• Utilise Machine Learning to predict future prices for weekly and daily intervals.

  • Aim for accuracy over 80% (personal preference).

  • Weekly analysis being the focus, with daily used as a run rate to capture volatility and adjust the weekly if necessary.

• Build a back-testing model to test the algorithm/models historically

• Save a summary of the results by date for each ticker

• Message the signals on a daily basis

• NTH (Nice To Have):

  • Bot trading from signals on stock & shares market

My Setup

A primitive setup I’m sure, but since I’m not an actual developer, It’s suitable for the circumstances.

- o1-preview
- o1-mini
- 4o | https://chatgpt.com/ | This is the AI I'll be using to generate the code, build the models, and algorithm. | | Python 3 | Python | Language used for coding. I'm not advanced in this, and I'll lean on the AI for 95% of the code as I'm avoiding building it manually on purpose—also, it's a lot quicker than me. | | Python Virtual Environment | - Virtual Environments | | | How to Install | I always use a virtual environment for my Python coding. I encourage you to do the same—much cleaner, lfewer bugs and conflicts. | | | Linode | https://www.linode.com | I use this service for cloud instances at an inexpensive cost.

This is my referral link; you'll receive a $100 60-day credit once you've added a valid payment method to your account | | Wikipedia | List of S&P 500 companies - Wikipedia | List of the S&P 500 companies. I chose specific industries that I would trade in. | | WSL2 | Install WSL2 | This is my go-to for my setup. I have a Windows laptop, but I use Linux 80% of the time. Linking this to my VS Code IDE is a must—can't work any other way |

So, that makes a start, see you in the next one.

pxng0lin.

After setting up Radicle, we need to setup Foundry and start to build our contracts. 'Foundry 101' has a Github repo that we'll utilise for our code to form the base, then, following along make changes where needed. Radicle will replace any add, commits and pushes usually done to Github, lets hope we don't run into too many issues along the way.

forge init

Oops.. What's this?!

As we've created a README.md document in the working directory (previous article), we need to use the --force command when initialising Foundry.

forge init --force

Viola!

Target directory is not empty, but `--force` was specified
Initializing /home/pxng0lin/web3/projects/radicle_xyz/rad-foundry-fund-me...
Installing forge-std in /home/pxng0lin/web3/projects/radicle_xyz/rad-foundry-fund-me/lib/forge-std (url: Some("https://github.com/foundry-rs/forge-std"), tag: None)
Cloning into '/home/pxng0lin/web3/projects/radicle_xyz/rad-foundry-fund-me/lib/forge-std'...
remote: Enumerating objects: 2310, done.
remote: Counting objects: 100% (2305/2305), done.
remote: Compressing objects: 100% (805/805), done.
remote: Total 2310 (delta 1534), reused 2145 (delta 1428), pack-reused 5
Receiving objects: 100% (2310/2310), 658.95 KiB | 7.66 MiB/s, done.
Resolving deltas: 100% (1534/1534), done.
    Installed forge-std v1.8.2
    Initialized forge project

Nothing is really different at this stage, I noticed the .github/workflows was created with the test.yml file, will see if this has any significance later.

We make our first copy-paste here now, using the Github repo from the course resources.

• Creating a new file named FundMe.sol in the src directory

• Then we navigate to the remix-fund-me-f23/FundMe.sol contract in the repo and paste this into our new FundMe.sol file.

• Repeat the steps for the second contract PriceConverter.sol

Radicle push

Now, since we are using Radicle, I want to test that my changes are working.

Following the Radicle user guide, we will commit and push the changes to my Radicle repo.

Once you’re finished, add and commit your changes with git add and git commit just as you would when collaborating on any other Git repository. Then use git push rad master to synchronize the changes with your node (be sure to replace master with your default branch, in case that’s not it).

So for my rad push, I'll be pushing to the main, as this was the branch name I changed to upon initialisation.

git push rad main

Error!

error: error connecting to ssh-agent: Environment variable `SSH_AUTH_SOCK` not found
error: failed to push some refs to 'rad://z43pr3L72n8wT74KSHcty9fEY5JaL/z6Mkg3Tu7aGDn3pLrshRiaCFLQJwEHyFKTCwYFoKDHts1YV2'

Our first error! It seems not having ssh-agent running is causing issues. I ran the following command to start ssh-agent. The command will check the agent is running, if not, then it will start it; the latter is the case for us.

eval "$(ssh-agent -s)"

After this, we run the rad auth command to ensure our private key is added

rad auth
✓ Passphrase: ********
✓ Radicle key added to ssh-agent

So, lets try again to push to Radicle... Success!

git push rad main
✓ Canonical head updated to 9d054f3d641253a3babd505e0b05807e4065e670
✓ Synced with 4 node(s)

  https://app.radicle.xyz/nodes/seed.radicle.garden/rad:z43pr3L72n8wT74KSHcty9fEY5JaL/tree/9d054f3d641253a3babd505e0b05807e4065e670

To rad://z43pr3L72n8wT74KSHcty9fEY5JaL/z6Mkg3Tu7aGDn3pLrshRiaCFLQJwEHyFKTCwYFoKDHts1YV2
   9ff86ae..9d054f3  main -> main

Imports and Dependencies

Upon running forge build to compile our new contracts, we get some errors. The course explains why and gives the solution, so we will implement this now.

Firstly, we need the repo that we're going to install from, Chainlink. In the terminal we use the command forge install smartcontractkit/chainlink-brownie-contracts --no-commit (notice that we don't use the full address).

Now we need to implement, so our contracts can use them, this is done using remappings in the foundry.toml file. The remapping changes @chainlink/contracts to lib/chainlink/chainlink-brownie-contracts/contracts when importing contracts into our project. It's like a shortcut that lets you use a shorter name to refer to specific locations, making it easier to import contracts.

[profile.default]
src = "src"
out = "out"
libs = ["lib"]
remappings = ["@chainlink/contracts/=lib/chainlink/chainlink-brownie-contracts/contracts"]

# See more config options https://github.com/foundry-rs/foundry/blob/master/crates/config/README.md#all-options

Save the toml file, go to the terminal and run forge build, Success!

[⠊] Compiling...
[⠢] Compiling 3 files with Solc 0.8.25
[⠆] Solc 0.8.25 finished in 68.45ms
Compiler run successful!

Lastly, a little alpha from Patrick to help with identifying errors in our contracts easier. Adjusting the NotOwner() custom error and prefixing the contract name.

error FundMe__NotOwner();

Ok lets wrap up here, so far so good. I don't want to reinvent the wheel for the course. Rather, this is more to ensure my Radicle repo is capturing my changes, and we've achieved this so far.

What's an Attackathon?

Attackathons are education-based bughunting contests where whitehat hackers compete over a reward pool by submitting impactful bugs in the project's code.

Before the Attackathon
Immunefi works with the project to host a security-focused education period, providing top tier education and support to security researchers.

During the Attackathon
Whitehats experience ideal bughunting conditions, with direct project support, responsiveness, and duplicate rewards.

After the Attackathon
Immunefi spotlights the security accomplishments, with a custom leaderboard, Attackathon findings report, bug fix reviews, and NFT awards.

Ultimately, Attackathons serve to secure projects, develop their security ecosystem, and create new opportunities for whitehats.

source: Fuel Attackathon | Immunefi

Before going deep into Fuel, I thought I'd have a look and feel for the Sway programming language. Using the documentation, I'll get setup and follow the Developer Quickstart Quide for one of the development tutorials.

Smart Contract Quickstart: Fuel Injection Installation

NB: I'm using VS Code as my editor, and I will continue my journey with Radicle and use that as my repo instead of Github. Also, I installed the Sway extension as recommended by Fuel.

We firstly create a working directory and initialise a git repo.

mkdir counter-tutorial
git init

Then we install fuel following the instructions

curl https://install.fuel.network | sh

Straightforward and didn't run into any issues. I then did the update steps just in case.

fuelup update
updating the 'latest-x86_64-unknown-linux-gnu' toolchain
[00:00:00] [########################################] 4.84 KiB/4.84 KiB (0s) - Download complete                                                                                                   Downloading: forc forc-explore forc-wallet fuel-core fuel-core-keygen 

latest updated
  updated components:
  - forc 0.60.0
  - forc-explore 0.28.1
  - forc-wallet 0.7.1
  - fuel-core 0.26.0
  - fuel-core-keygen 0.26.0

The counter contract

So the following command will generate our Sway contract, I guess this is similar to any tutorials that provide pre-built examples to use.

forc template --template-name counter counter-contract

After this I can initialise my Radicle repo too.

rad node start
rad init

Output:

• The Radicle repo

rad init

Initializing radicle 👾 repository in /home/pxng0lin/web3/immunefi/fuel/fuel-tutorials..

✓ Name Fuel tutorials
✓ Description Fuel tutorials for development with Sway
✓ Default branch main
✓ Visibility public
✓ Passphrase: [REDACTED*]
✓ Unsealing key...
✓ Repository fuel-tutorials created.

...

counter-contract
├── Forc.toml
└── src
    └── main.sw

1 directory, 2 files

Building (compiling) the contract.

Following the instructions we can now build the contract

forc build
  Creating a new `Forc.lock` file. (Cause: lock file did not match manifest)
  Removing core
  Removing counter
  Removing std git+https://github.com/fuellabs/sway?tag=v0.31.1#c32b0759d25c0b515cbf535f9fb9b8e6fda38ff2
    Adding core
    Adding std git+https://github.com/fuellabs/sway?tag=v0.60.0#2f0392ee35a1e4dd80bd8034962d5b4083dfb8b6
   Created new lock file at /home/pxng0lin/web3/immunefi/fuel/fuel-tutorials/counter-contract/Forc.lock
  Finished debug [unoptimized + fuel] target(s) in 3.56s

Next we will setup a local wallet, this comes along side Fuel so we shouldn't have any issues here either.

forc wallet new

Since this is a tutorial I won't make the password complex, I'm more than likely to forget if I do.

Wallet mnemonic phrase: [REDACTED]

We will create a new wallet account to get our fuel address using the command forc wallet account new.

Please enter your wallet password to derive account 1: 
Wallet address: [REDACTED]

Deploying our contract

We will now deploy to the testnet by running the command forc deploy --testnet. Once we've provided the password for our wallet we should receive confirmation of deployment.

Error!

What's this? So we are unable to deploy since we have 0 funds in our wallet. So in the tutorial we can get some funds from the faucet

After fuelling up we will try that again.

 forc deploy --testnet
  Finished release [optimized + fuel] target(s) in 3.28s

Please provide the password of your encrypted wallet vault at "[REDACTED]": 

---------------------------------------------------------------------------
Account 1: [REDACTED]

Asset ID : f8f8b6283d7fa5b672b530cbb84fcccb4ff8dc40f8176ef4544ddb1f1952ad07
Amount   : 2000049
---------------------------------------------------------------------------

Please provide the index of account to use for signing: 1
Do you agree to sign this transaction with [REDACTED]? [y/N]: y


Contract counter-contract Deployed!

Network: https://testnet.fuel.network
Contract ID: 0x71a2a7efd28e8b80388105cb3bb52ca5e100b21cb034ee6e44a5ffb135f7f361
Deployed in block 0018e86c

Success!

This finished the tutorial! So I'll also wrap it up here. The next step they have are further tutorials.

I'll have a read through the Sway documentation to get a better feel of what I'm looking at in the contract, and potentially move on to Sway Program Types and build a Predicate following the tutorial; until the next article, thanks for reading.

Who Am I?

Purpose of activity?

I've used Github for quite some time, I'm no expert of course, but since I got into web2 hacking many moons ago, then in the past few years (as of May 2024) web3, I've been using it mainly for security research, a few python scripts, and early web3 learning (at the moment with Cyfrin's Updraft learning platform, go check it out here).

But, more recently I was hoping to build more of a portfolio whilst learning and stumbled across the video by Nader. Since decentralisation is thrown around frequently, even though it is relevant and has meaning; I thought I would steer more towards that with my own projects and see how it goes.

Getting setup

I already have WSL2 setup on my Windows laptop, and the VS Code editor installed - including the plugin to connect to WSL and the Radicle plugin. So, I'll only need to follow the setup of the Radicle repo from here.

Installation as per the instructions from the guide:

curl -sSf https://radicle.xyz/install | sh

Output:

 Welcome to Radicle

Detecting operating system...
Downloading https://files.radicle.xyz/releases/latest/radicle-x86_64-unknown-linux-musl.tar.xz...
######################################################################## 100.0%
Downloading https://files.radicle.xyz/releases/latest/radicle-x86_64-unknown-linux-musl.tar.xz.sig...
######################################################################## 100.0%
Verifying radicle-x86_64-unknown-linux-musl.tar.xz...
Good "file" signature for cloudhead with ED25519 key SHA256:iTDjRHSIaoL8dpHbQ0mv+y0IQqPufGl2hQwk4TbXFlw
Installing Radicle into /home/pxng0lin/.radicle...
Configuring path variable in ~/.bashrc...

✓ Radicle 1.0.0-rc.9 was installed successfully.

Before running Radicle for the first time,
run `source ~/.bashrc` or open a new terminal.

Then, create your Radicle key pair with `rad auth`.

As per the instructions, we need to execute the commands in my .bashrc file by calling source. This basically re-reads the file which is only read upon start-up of Bash. Alternatively, we could open a new terminal window too.

source ~/.bashrc

Ok so now to check that it was successful by running the version command.

rad --version

Output:

rad 1.0.0-rc.9 (d56d619f)

Great! I'm good so far.

Next is creating a Radicle identity aka Radicle DID (Decentralised Identifier). This is a cryptographic key pair, since we're on the network, which is used to identify and authenticate me/my node. Public is for the network and everyone can see, private, as indicated, is only used for authentication of my node, signing of code and other artifacts, and shouldn't be shared publicly.

rad auth

Output:

Initializing your radicle 👾 identity

✓ Enter your alias: rxdicle-1
✓ Enter a passphrase: [REDACTED *]
✓ Creating your Ed25519 keypair...
✓ Your Radicle DID is did:key:[REDACTED]. This identifies your device. Run `rad self` to show it at all times.
✓ You're all set.

✗ Hint: install ssh-agent to have it fill in your passphrase for you when signing.

To create a Radicle repository, run `rad init` from a Git repository with at least one commit.
To clone a repository, run `rad clone <rid>`. For example, `rad clone rad:z3gqcJUoA1n9HaHKufZs5FCSGazv5` clones the Radicle 'heartwood' repository.
To get a list of all commands, run `rad`.

NB: I didn't use ssh-agent, we will revisit this later.

Your Radicle DID is similar to your Node ID (NID), the difference is the former is formatted as a Decentralized Identifier, while the latter is just the encoded public key. Share your Radicle DID freely with collaborators.

You can check your rad details by running a few commands, reference below, I used the rad self to check mine

Alias           rxdicle-1
DID             did:key:z6Mkg3Tu7aGDn3pLrshRiaCFLQJwEHyFKTCwYFoKDHts1YV2
└╴Node ID (NID) [REDACTED]
SSH             not running
├╴Key (hash)    [REDACTED]
└╴Key (full)    [REDACTED]
Home            /home/pxng0lin/.radicle
├╴Config        /home/pxng0lin/.radicle/config.json
├╴Storage       /home/pxng0lin/.radicle/storage
├╴Keys          /home/pxng0lin/.radicle/keys
└╴Node          /home/pxng0lin/.radicle/node

Many of the other items you see in the rad self output can be viewed individually. Wondering about your alias? A quick rad self --alias has you covered. Need to pinpoint your Radicle home folder? rad self --home is your friend. And for your config file location, just hit up rad self --config.

If you’re ever feeling lost, rad self --help will lay out all your options.

NB: I'm in the habit of redacting information, even if it's said to be ok for the public, call me over-cautious, I guess.

Node On?

A quick check on my node status, I expect it to be off since its my first install.

rad node status

Output:

Node is stopped.
To start it, run `rad node start`.

As expected it's not running, so we will fire it up with the provided command

rad node start

Output:

✓ Passphrase: 
✓ Node started (841691)
To stay in sync with the network, leave the node running in the background.
To learn more, run `rad node --help`.

For extra commands, and to find out how to stop the node use rad node --help

Starting a new project

Moving to the Youtube video by Nader, we will now initiate a new repo, we will be using the tutorial from Updraft as my project, Fund Me. The only difference will be the replacement of Github for Radicle, everything else should be the same.

mkdir rad-foundry-fund-me-23

we then change directory into the project folder and initiate the Radicle project.

Because I'm creating a brand new project, we need to ensure have a git repo ready, so I'll initialise a git repo first, then the Radicle repo afterwards.

git init

Output:

hint: Using 'master' as the name for the initial branch. This default branch name
hint: is subject to change. To configure the initial branch name to use in all
hint: of your new repositories, which will suppress this warning, call:
hint: 
hint:   git config --global init.defaultBranch <name>
hint: 
hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and
hint: 'development'. The just-created branch can be renamed via this command:
hint: 
hint:   git branch -m <name>
Initialized empty Git repository in /home/pxng0lin/web3/projects/radicle_xyz/rad-foundry-fund-me/.git/

The Radicle repo requires at least 1 commit to be done, so I'll create a README.md file and commit this. I changed the initial branch name to 'main', this was only due to preference and not a requirement.

git branch -m main
touch README.md
git add README.md
git status

Output:

On branch main

No commits yet

Changes to be committed:
  (use "git rm --cached <file>..." to unstage)
        new file:   README.md

We will commit this using the command git commit. It will ask for a comment to be added for the commit, then press ctrl+o to save, followed by ctrl+x, and we are good; now I'm ready to initialise the Radicle repo.

Output:

[main (root-commit) 9ff86ae] Creation of README.md file only
 1 file changed, 0 insertions(+), 0 deletions(-)
 create mode 100644 README.md

Initialising the new Radicle repo.

rad init

TIP: to clear any entered commands and returned outputs from your terminal, use eitherctrl+l(ell) or typeclear, I like to clean up the screen from time-to-time.

So now we need to enter some information about the new project for the repo.

✓ Name foundry-fund-me
✓ Description A tutorial project in the Foundry 101 course on Cyfrin to learn how to professionally deploy code, master the art of creating fantastic tests, and gain insights into advanced debugging techniques.
✓ Default branch main
✓ Visibility public
✓ Passphrase: 
✓ Unsealing key...
✓ Repository foundry-fund-me created.

Your Repository ID (RID) is rad:z43pr3L72n8wT74KSHcty9fEY5JaL.
You can show it any time by running `rad .` from this directory.

◢ Upload done for rad:[REDACTED] to [REDACTED] : signal: 9 (SIGKILL)✓ Repository successfully synced to [REDACTED] 
◢ Uploading rad:[REDACTED] to [REDACTED] Compressing objects: 100% (8/8)✓ Repository successfully synced to [REDACTED] 
✓ Repository successfully synced to 2 node(s).

Your repository has been synced to the network and is now discoverable by peers.
View it in your browser at:

    https://app.radicle.xyz/nodes/seed.radicle.garden/rad:z43pr3L72n8wT74KSHcty9fEY5JaL

To push changes, run `git push`.

And we're off!

So the setup is done, and I'm ready to begin the development stage. this will be the follow along as per the course. I'll end it here for today, and begin a new article as I complete the stages of the course as not to make this too long of a post.

• pseudonym: pxng0lin (pangolin)

• location: United Kingdom

• interests: data, learning, privacy, web3, independence from 9-5 (don't we all yearn for that?)

Some deets.

• profession: Previously an analyst (data, forecasting, resource), all things data, forecasting resource and modelling used for predictions, over the past 4 years I've transitioned into cybersecurity and reignited my love of hacking and transferred my problem solving skills, and all things data over.

• experience: Mainly in data for analytics, from insight to forecasting resource, modelling and prediction of customer data, etc., this was all before the 'dream' that is now AI.

  • Web2: I started to dabble in hacking several years back and utilised platforms like Bugcrowd, Intigriti and HackerOne, great platforms still today. Earned some rep, learnt a lot more than I reported, built a lot of scripts in Bash and Python, and realised quite quickly, there's too many hackooors in that space, and knowledge is a scramble to get quickly, research takes a while too, especially to get to a top level, not for all, but for me and my circumstances, I didn't have the time (other responsibilities).

  • Web3: I discovered Immunefi c. 2020, when looking at crypto and thinking at the time, "can this stuff be hacked?". I joined their Discord server and saw conversations around using Slither to detect vulnerabilities in smart contracts and talk about false-positives being high in the results (smart contracts? What's one of them?). I saw this and cloned the Slither tool from Github, ran this over a mainnet smart contract from Etherscan (shh... naughty!) and then the results showed, what I thought, were valid vulnerabilities - "Oh great, just like using web2 tools, such as Nuclei..", were not really, and it wasn't that simple!

  • languages speaking/coding: I'm not expert in any, I have used/use several, and continue to learn, in no particular order

    • VBA

    • SQL

    • Python

    • R

    • Bash

    • Solidity

    • Javascript

    • English (Native, British)

    • Arabic (Fus-ha)

Ambitions

I remember a while back looking at my options and potential paths for the web3 space, in opposition to my current career in cybersecurity, which is web2 mainly, and focusses on management of assets within a business regarding vulnerabilities. I came across the Spearbit's Github repo that broke down the different levels of the positions they have, and this gave me a starting point of where to aim, for what I consider, entry level.

• Spearbit Roles

  • Junior Security Researcher (JSR)

  • Associate Security Researcher (ASR)

  • Security Researcher (SR)

  • Lead Security Researcher (LSR)

• Promotion Flow

  • JSR to ASR

  • ASR to SR

  • SR to LSR

My aim was to become proficient enough in this space to at least be considered a JSR. In doing so, I would then be in a position to apply for the role with places like Spearbit, or at least have a reference of my level for other opportunities they may arise.

Fast forward a little, the aim is still to be proficient, but, after experiencing audit competitions, the maturity of vulnerabilities and the shift of the severity from Critical to a low in under a year, I started to believe just meeting the expectation would be suitable; but wouldn't separate me from the rest.

After a Twitter/X post from Dacian (great chap!) for a role of LSR at Cyfrin (not applying, lol), I started to have a think about what I needed, and how I could prove it to potential employees. I dropped Dacian a DM too just to get a bit of insight into what I could do, without becoming a content creatooor all over Twitter/X, I'm not one for that sort of attention, nor did I really enjoy reading threads of regurgitated knowledge, good for those that have done it, but as many things do, it gets old and "seen it before" quite quickly. You could argue the same is with blogs, however, for me, this is an easier way to point someone to what you have been doing, what you know, and how you've applied what you've learnt; without having to link to tweets lost in your Twitter/X history.

So to wrap up! I want to use this space to share learnings from competitions, vulnerabilities I've reported and/or read about, and anything web3 or coding related that I do as I venture deeper into security research - emphasis on research, since this really is my main interest, I research, I try, I win and fail, but in general, I have a passion for learning things, problem solving, and usually sharing amongst family and friends (sometimes to blank faces and "that's nice" smiles), but hopefully now I can widen my reach to a bigger audience, and put myself in the window for the next role that comes up in the future.