1. Intro
2. The need
3. Initial prompt and resulting specifications
4. Getting the files
5. Code shakedown / Steel-thread
6. Building up with functionality
7. Next level
8. Conclusion
9. Appendix – The specification

Intro

This is a natural continuation of my previous post, “Chat GPT for Coding.” As I mentioned there, coding is an excellent use case for ChatGPT. Even if some productivity uplifts may be overstated, the fact that some errors in code can be quickly identified by a failing program offers a swift feedback loop, which is generally a good thing.

Bill Gurley, a prominent VC, wrote about this the other day:

Has anyone generated an AI task utility/efficiency/productivity score? It strikes me in coding this would be a 10/10. Other tasks seem lower down. I have seen lists of jobs that are impacted but I’m focused more on the utility against specific tasks. I’d be shocked if this wasn’t…

— Bill Gurley (@bgurley) April 14, 2023

Then I saw this from Mckay Wrigley, who I’ve been following recently, particularly regarding his ChatbotUI project. This project is a ChatGPT lookalike using OpenAI APIs in the background, providing a better UI to interact with (Chat)GPT models. I’d like for OpenAI to either incorporate the project, acqui-hire it, or develop equivalent functionality (but preferably any of the first two options).

Prompt tricks pay off.

In this clip I show you how to *drastically* improve ChatGPT’s outputs for software generation.

Compare the default prompt to my specialized prompt – night and day.

Full prompt below! pic.twitter.com/kKP6hi8JQp

— Mckay Wrigley (@mckaywrigley) April 11, 2023

So I decided to put this in practice.

The need

The problem I wanted to solve was the following: I have a few Raspberry Pis at home, and occasionally one of them encounters an error. I usually try to fix them by either SSH-ing or connecting them to a monitor (especially if the problem is related to networking or SSH).

In this case, I was trying to solve an issue with the SSH client of a Raspberry Pi Zero with DietPi as its OS (I have a project I will write about at some point). At some point, I wanted to use ChatGPT to troubleshoot my issues and display specific command outputs or the content of certain files, including logs. Since I was connected via command-line only, I wanted a way to quickly send that output to a more capable computer with a faster browser and discovered the application Netcat.

Netcat is an old-school Unix command (according to Wiki, the latest stable release is from 2006) that allows you to send text (probably bytes in general) to another computer via the network. The other computer needs to be listening for Netcat input on a given port, and you can send messages from the client to that server. The usual way to do this is to pipe a command like "ls | netcat 192.168.1.55 12345". This code will execute the “ls” command on the client computer and send the output to a computer with the IP 192.168.1.55 via port 12345. If the other computer is running Netcat as a server on that port, it will receive the messages.

I was curious to see if I could have the server present the received text in a more accessible format that I could quickly copy and paste into ChatGPT. So, I thought about asking GPT for help, using Mckay’s approach (with some modifications as needed or according to my personal preferences). I generally liked the approach and will likely try it a few more times.

Initial prompt and resulting specifications

I mostly used Mckay’s approach here:

You are a world class software engineer.

I need you to draft a technical software spec for building the following:

• I want a server running on linux that will work as a netcat server. The server will be built with Flask and needs to have the capability of being listening constantly to input. If it’s able to identify via netcat the receiving ip, it’s going to incorporate it in its own modelling
• that server will render into a nicely formatted terminal. It should look a bit retro, but with modern design principles.
• the server will be deployed with Docker
• use env config files for URLs, ports, api keys, secrets, etc
• create a repository with instructions on how to build and deploy docker image, appropriate gitignores, etc
• as a user of the website that is on the server host, I should be able to apply some filters to the received data: ip (if available), received time. I should have an easy way to select some text and copy and paste it
• in terms of the use, the client will be the standard netcat app from a linux computer

Think through how you would build it step by step.

Then, respond with the complete spec as a well-organized markdown file.

I will then reply with “build,” and you will proceed to implement the exact spec, writing all of the code needed.

I will periodically interject with “continue” to prompt you to keep going.

Continue until complete.

I am quite skeptical that you need things like “you are a world classs software engineering” or tell GPT to “think through”, but I mostly followed the recipe.

The output specification was quite good (see in the appendix), although it initially rejected the “build” command to create the files, so I had to coax it a bit. I’ll try to find wording to get what I wanted on the first try.

Me: build

GPT: As an AI language model, I am unable to directly build or execute code. However, I can provide you with guidance and code snippets to help you build the project based on the provided

Me: Please proceed to implement the exact spec, writing all of the code needed. I will periodically interject with “continue” to prompt you to keep going. Continue until complete.

GPT: I understand your request, and I will provide you with code snippets and guidance for each step of the implementation. Since I cannot execute the code, you will need to build and test the project yourself.

Getting the files

Several files were required for the project, and ChatGPT showed them to me one by one with their content. As I wanted to push for efficiencies (a friend just called me lazy), I wanted a single terminal command that I could execute to create the files and folders for me. So, I tried this:

Me: Can you give me a script that I can just copy and paste in a unix terminal and will create each of the files you mentioned with the appropriate content

GPT: Sure, here’s a script that will create the necessary files and directories with the specified content. Make sure you’re in the project’s root directory and then copy and paste the following script into your Unix terminal:

bashCopy codemkdir templates && \
echo 'FROM python:3.9-slim

WORKDIR /app...

I was able to click on the “copy” button that appears when ChatGPT provides code, paste it in my terminal, and the files were there with the right content. However, due to output length limitations, it only reached step 3.5 of the specification.

I wanted GPT to continue from that step onwards, but I struggled and had to iterate about 4-5 times (my current approach is to stop the text generation and ask a new question, not rewrite the last one, I should try a different approach as well to avoid consuming the context window), and this is what worked in the end:

You are helping me with a project. You already gave me several files and I already have the following:
├── app.py
├── Dockerfile
├── docker-compose.yml
├── requirements.txt
├── .env.example
├── .gitignore
└── templates/
└──── index.html

I need you to help me with the rest of the files. I will give you the full spec for context, but I only need you to provide the files for step 3.5 and onwards, no need to provide the existing files for the previous steps. Please give me a single script that I will copy and paste once in a unix terminal and the output of the execution will create each of the files you suggested with the appropriate content, I shouldn’t have to put your output in an sh file or do multiple copy/pastes

{project specification created earlier by GPT, see appendix}

You can probably tell some of the previous “failures” were by looking at the specificity of some my requirements.

Code shakedown / Steel-thread

After that, the code was sort of ready to run, but I had some early issues which I managed to get GPT to help me with by looking into the errors. However, quite soon, GPT had forgotten about the context, so I had to use this prompt:

you are helping me with a project. You provided me with some code. I managed to do some debugging and when sending data through netcat, I can see in the python console that data is flowing, but the webpage I have open does not. I can see the code flow that only “handle client” is invoked

<relevant code>

{project specification}

So, I started by stating that we were already working and that the code was provided by GPT. Again, I am not sure how useful this is, although I feel this is more important than telling GPT that it’s a great programmer. My thinking is that by telling GPT that we have been working together and that code has already been provided, it should be less likely to decline to help or provide generic feedback that doesn’t include code.

Then, I explained the issue I was having and the content of the file I thought the problem was in. I would have included error logs as well in this step if I had any.

Finally, I included the project specification again, but I am not sure if ChatGPT manages to incorporate all of it. Even if it only incorporates the beginning, I believe it will help to get a better answer.

At this stage, and after minor iterations, the project was broadly working (from a backend perspective), but the styling and generally the front end wasn’t really nice or doing much.

Building up with functionality

I then asked:

ok, this now works as expected. Please help me with the styling and functionality of the website:

3.6. Terminal Interface
Design and implement a retro-style terminal interface using HTML, CSS, and JavaScript.
Use modern design principles to create a responsive and accessible user interface.
Implement real-time data streaming between the server and the terminal interface using WebSockets.
3.7. Data Filtering and Copy-Pasting
Implement the ability to filter data in the terminal interface based on IP and timestamp.
Add a feature for easily selecting, copying, and pasting text within the terminal interface.

I already have
templates – index.html
static-css-styles.css
static-js-app.js

I would like to use the tailwind.css framework for this

I was explicit that the code was working (the previous messages were about troubleshooting that), I made the request (styling and website functionality), and then I used the two sections of the specification to reinforce what I wanted.

Then, I reminded GPT of the current front-end file structure (even if those files mostly were not doing much).

I asked GPT to use the Tailwind framework for CSS.

With this prompt, functionality started to emerge, although I had a few things to wrangle (copying and pasting output from the web development console from any main browser), and in particular, GPT wasn’t great at pointing to a good CDN link for Tailwind and the JavaScript’s Socket.IO library that was needed for the project, so I had to do some manual intervention.

Then, I asked for some improvements which were not part of the original spec (or had been lost/forgotten).

This works great and looks very well.

I want to make some changes now on the layout (and that implies some changes in the workings).

Each message received through netcat should have it’s own cell in the HTML. Also, We should see the data in a table format, on the first left-most column, metadata: ip, timestamp, on the 2nd, and much wider column, all the content.

There should be a max height of each row, and if the content is larger havinga scroll only for that cell.

Each cell with input from netcat (not metadata) should have a visible hovering button that will allow to add the content of the cell to the clipboard

A bit more wrangling emerged, as I couldn’t see the data flowing properly, but without much info provided (just describing the behaviour), ChatGPT identified that there was a mismatch in the structure of the data sent form the backend to the front end, and that was quickly solved.

I added some new improvements and corrections

this is working now. Things that need improvement

Along the first load of the page a new row is generated with undefined, this is not how it should be.

The rows don’t have a maximum height and scroll bar

we lost the old school terminal format

the left column should be much more narrow than the right one

A funny thing is that as I paste these prompts while typing this post, I notice some typos and other writing errors that GPT is able to work around quite well anyway. I do imagine that by making those mistakes, you probabilistically reduce the quality of the output, but I haven’t read much about it.

Next level

Docker had been one of the original requirements, and I had the Dockerfile, but I hadn’t tested it so far.

At this point, I had already abandoned the initial structure. At some point, I also hit the limit on ChatGPT4, so I started to ask for small improvements like code comments using ChatGPT 3.5 in a different chat thread.

Anyway, getting it to work in Docker was a bit trickier, even though the solution was very simple (and I probably had faced the same issue in the past). The troubleshooting with GPT wasn’t very easy, and my hypothesis is that the Socket.IO library and Netcat are not that common, so they are not heavily represented in the training corpus of GPT.

One sort of “failure mode” (or “suboptimal mode”) occurred when GPT suggested trying something without providing example commands, so it’s something I should get better at including in prompting from the start.

As a funny aftermath, the day after I was on a queue doing some errands and thought about a new feature which was to have a simple unix program that you can set up once in the client computer and will just invoke netcat {IP} {PORT}, so I created “wnc”, so that you can just do COMMAND | wnc or wnc COMMAND, which is quicker to type and you don’t need to remember the ip.

So I continued my chat session from my mobile. Of course I wasn’t testing the code, but I still iterated when thinking through the problem and thinking through potential challenges when using or deploying it. For instance, maybe the IP changes from time to time, so I wanted a command to update that saved IP. Also to use a file to store default IP/PORT, but being able to pass through different ones if needed, also to pass all the arguments that netcat supports.

Then I thought about having a simple installer with default values. I came home later, and I already had a couple of scripts to try. I had to iterate a bit but I finally got one version working (at least on my machines!).

Conclusion

I will definitely keep trying this structured prompt strategy. In some point it could even be passed to an “agent” and orchestrate the different actions including running some of the code. I still think there are parts where I would have to steer, as some requirements only appeared when trying the code. Thinking about agile incremental functionality, I could see the agent doing those loops and not a fully fledged application.

One thing that occurred to me was how much it would have cost if, instead of using ChatGPT, I had used the API.

In total, my back-and-forth amounted to 32,000 “words” and 231,000 characters. Normally, words-to-tokens is 3:4, but as there was code involved, it’s different. I’ve just counted the tokens with some python code (based on the great video from Andrej Karpathy: Let’s build GPT: from scratch, in code, spelled out.), and there are 70,000 tokens.

If I had only used GPT-4 (8k tokens context), assuming 5c/1k token (actually, it’s 3c for input and 6c for output), the session would have cost me $3.5. I still believe there’s room for improvement regarding tokens that are spaces, especially in the input, and perhaps in the output, maybe by adding an extra step to decode.

If I had used GPT-3.5, it would have cost 0,14$.

With my current usage pattern and from a purely economic perspective, it’s not immediately clear whether it makes more sense to pay for ChatGPT Plus or simply pay for API usage alone. However, I’m not sure if you can actually access the GPT-4 API without Plus. Regardless, I’m still eager to test the plugins exclusive to Plus users, and I suspect they’ll continue to release new features primarily for Plus subscribers, so I’m not considering unsubscribing.

All in all, coding and working on personal projects have become significantly more enjoyable than before. The main reasons, I believe, are the shortened feedback loop and experiencing fewer instances of getting stuck. These factors are closely tied to the increased speed and higher throughput.

Notes

• The post was lightly edited for language with ChatGPT
• The featured image of the post was generated with Midjourney (I caved-in and signed up to it) using the prompt: “a robot is pair-programming with a human coder. They are working on a very wide screen. They are in a coffeeshop. It’s around sunset time, it’s autum, and they are in NYC. High quality photo with an artistic flare“
• The excerpt of the post was generated as well with input from ChatGPT

Appendix – The specification

Netcat Server Technical Specification

Table of Contents

1. Overview
2. System Architecture
3. Implementation Details
   1. Server Setup
   2. Flask Application
   3. Docker Deployment
   4. Environment Configuration
   5. Repository Structure
   6. Terminal Interface
   7. Data Filtering and Copy-Pasting
4. Dependencies

1. Overview

This document describes the technical specification for building a netcat server running on Linux, built with Flask, which renders received data in a retro-style terminal interface. The server will be deployed with Docker, and environment configuration files will be used for managing sensitive information. The web application will provide users with filtering options based on IP and received time, as well as the ability to copy and paste text easily. The client will be the standard netcat app from a Linux computer.

2. System Architecture

The system will consist of:

1. A Flask application that acts as the netcat server, listening for input and rendering the terminal interface.
2. A Docker container for easy deployment and configuration.
3. A repository containing the source code, Docker configuration, environment files, and instructions for building and deploying the Docker image.

3. Implementation Details

3.1. Server Setup

1. Create a new Flask application using Python.
2. Implement a socket listener to receive data via netcat, capturing the sender’s IP address.
3. Store the received data and its associated metadata (IP address and timestamp) in a data structure, such as a list of dictionaries.

3.2. Flask Application

1. Implement a route to serve the terminal interface.
2. Create a WebSocket endpoint for real-time data streaming between the server and the terminal interface.
3. Implement the logic for filtering data based on IP and timestamp.

3.3. Docker Deployment

1. Create a Dockerfile containing the necessary instructions to build the Docker image, including installing dependencies and configuring the Flask application.
2. Add a docker-compose.yml file for easy deployment and configuration management.

3.4. Environment Configuration

1. Use environment variables for managing sensitive information, such as URLs, ports, API keys, and secrets.
2. Create a .env.example file containing placeholder values for required environment variables, to be used as a reference for creating the actual .env file.

3.5. Repository Structure

1. Create a repository containing the Flask application source code, Dockerfile, docker-compose.yml, .env.example, and .gitignore files.
2. Add a README.md file with instructions for building and deploying the Docker image.

3.6. Terminal Interface

1. Design and implement a retro-style terminal interface using HTML, CSS, and JavaScript.
2. Use modern design principles to create a responsive and accessible user interface.
3. Implement real-time data streaming between the server and the terminal interface using WebSockets.

3.7. Data Filtering and Copy-Pasting

1. Implement the ability to filter data in the terminal interface based on IP and timestamp.
2. Add a feature for easily selecting, copying, and pasting text within the terminal interface.

4. Dependencies

1. Flask (Python web framework)
2. Docker (containerization platform)
3. JavaScript libraries for WebSocket communication and terminal UI implementation (e.g., Socket.IO and xterm.js)