I remain skeptical about using LLMs in an autonomous, agentic manner. In my experience, for software development tasks, they are most useful when employed in a chat-driven development manner, where humans guide their behavior and work with them step by step. The Answer.ai team recently published a post about Devin, sharing multiple tasks where their AI agent failed. Devin is a collaborative AI teammate built to help ambitious engineering teams achieve more. According to Answer.ai blog post, out of 20 tasks they gave to Devin, it failed at 14 tasks, succeeded in 3, and 3 were inconclusive. These results are quite disappointing. They have shared the complete task list in the appendix for anyone to try.
Many of the tasks they mentioned seemed achievable using AI assistants like Claude or ChatGPT, so I decided to experiment with Claude to complete one of them. I’ve increasingly been using Claude for coding-related tasks and have a paid subscription. This experiment uses Claude 3.5 Sonnet.
One task in the “Create a new project” category involves building a minimal FastHTML HTMX application. The task description was somewhat confusing – the project name asks for a “bulk upload example app” but the description requests a “bulk edit example.” I decided to follow the description and create a FastHTML bulk edit example.
From their documentation:
- htmx (also stylized as HTMX) is an open-source front-end JavaScript library that extends HTML with custom attributes that enable the use of AJAX directly in HTML with a hypermedia-driven approach.
- FastHTML is a new next-generation web framework for fast, scalable web applications with minimal, compact code.
HTMX is a relatively mature project, with its first release in November 2020 and the latest in October 2024. Claude 3.5 Sonnet, trained on data until April 2024, has knowledge about HTMX. However, FastHTML is newer, released in July/August 2024, so Claude won’t have direct knowledge about building web apps using FastHTML.
AI Assistants support external knowledge input, so I provided both HTMX and FastHTML documentation to Claude. FastHTML offers a 55-page document at https://docs.fastht.ml/llms-ctx.txt covering how to build FastHTML web apps. For HTMX, I provided the bulk update example page https://htmx.org/examples/bulk-update/ exported as a PDF document.
Let’s see if Claude can create a minimal app with this documentation. I started with a simple prompt: “Read the attached HTMX and FastHTML documentation and create a minimal bulk edit FastHTML example application.”
Claude took a few seconds to ponder and read the document and generated the following code.
from fasthtml.common import *
from dataclasses import dataclass
# Create a database and define the User model
db = database('users.db')
if 'users' not in db.t:
db.t.users.create(name=str, email=str, active=bool, pk='email')
User = db.t.users.dataclass()
# Add some sample data if table is empty
users = db.t.users
if len(users()) == 0:
sample_users = [
dict(name="Joe Smith", email="joe@smith.org", active=False),
dict(name="Angie MacDowell", email="angie@macdowell.org", active=True),
dict(name="Fuqua Tarkenton", email="fuqua@tarkenton.org", active=False),
dict(name="Kim Yee", email="kim@yee.org", active=True)
]
for user in sample_users:
users.insert(user)
# Set up the FastHTML app
app, rt = fast_app()
# Define toast styling
toast_style = Style("""
#toast.htmx-settling {
opacity: 100;
}
#toast {
background: #E1F0DA;
opacity: 0;
transition: opacity 3s ease-out;
padding: 1rem;
border-radius: 4px;
margin: 1rem 0;
}
""")
app = FastHTML(hdrs=(picolink, toast_style))
rt = app.route
@rt("/")
def get():
# Create checkbox inputs for each user
user_rows = []
for user in users(order_by='name'):
user_rows.append(
Tr(
Td(user.name),
Td(user.email),
Td(CheckboxX(id=f"active:{user.email}",
checked=user.active,
name=f"active:{user.email}"))
)
)
# Create the main form with table
form = Form(
Table(
Thead(
Tr(
Th("Name"),
Th("Email"),
Th("Active")
)
),
Tbody(*user_rows)
),
Button("Update Selected", type="submit", cls="contrast"),
Div(id="toast"),
hx_post="/update",
hx_swap="innerHTML settle:3s",
hx_target="#toast"
)
return Titled("Bulk User Update",
H3("Select Users to Update Status"),
form)
@rt("/update")
def post(request):
# Get form data
form_data = parse_form(request)
# Process each user
updated = 0
for key, value in form_data.items():
if key.startswith("active:"):
email = key.split(":", 1)[1]
is_active = value == "on"
users.update({"active": is_active}, email)
updated += 1
# Return toast message
return Div(f"Updated {updated} users", id="toast")
serve()
It recreated the HTMX example in FastHTML. By the looks of it this looked fine but we should always test and verify LLM output. The best part is Claude along with the code explaination gave steps to run the app.
- Install FastHTML:
pip install python-fasthtml - Save the code to a file (e.g.
bulk_edit.py) - Run with:
python bulk_edit.py - Visit http://localhost:5001 in your browser
After following the steps and running the app, the web page rendered fine and showed 4 records pulled from sqlite database. However, I encountered an error when trying to update. Welcome to the world of AI-generated code! The error was:
File "~/tmp/fasthtml-bulk-edit/bulk-edit.py", line 88, in post
for key, value in form_data.items():
^^^^^^^^^^^^^^^
AttributeError: 'coroutine' object has no attribute 'items'
When I reported the error to Claude, it realized it had forgotten to use await with the parse_form method and needed to change the /update API to use async.
@rt("/update")
async def post(request):
# Get form data
form_data = await parse_form(request)
# Process each user
updated = 0
for key, value in form_data.items():
if key.startswith("active:"):
email = key.split(":", 1)[1]
is_active = value == "on"
users.update({"active": is_active}, email)
updated += 1
# Return toast message
return Div(f"Updated {updated} users", id="toast")
After this change, the error disappeared, but I noticed it was always setting all users to an inactive state. It took me a few minutes to discover that the form was sending values as 1 instead of on. This occurs because SQLite stores booleans as 0 (False) or 1 (True).
After addressing this issue, the application worked correctly, though the toast message wasn’t visible. I prompted Claude again for help with this issue as shown below.
So, I again prompted Claude as shown below.
It changed the code to the code shown below.
from fasthtml.common import *
from dataclasses import dataclass
# Define toast styling
toast_style = Style("""
#toast {
background: #E1F0DA;
opacity: 1;
padding: 1rem;
border-radius: 4px;
margin: 1rem 0;
transition: opacity 0.5s ease-out;
}
#toast.htmx-swapping {
opacity: 0;
}
#toast:empty {
display: none;
}
""")
app = FastHTML(hdrs=(picolink, toast_style))
rt = app.route
@rt("/")
def get():
# Create checkbox inputs for each user
user_rows = []
for user in users(order_by='name'):
user_rows.append(
Tr(
Td(user.name),
Td(user.email),
Td(CheckboxX(id=f"active:{user.email}",
checked=user.active,
name=f"active:{user.email}"))
)
)
# Create the main form with table
form = Form(
Table(
Thead(
Tr(
Th("Name"),
Th("Email"),
Th("Active")
)
),
Tbody(*user_rows)
),
Button("Update Selected", type="submit", cls="contrast"),
Div(id="toast"),
hx_post="/update",
hx_swap="innerHTML settle:3s",
hx_target="#toast"
)
return Titled("Bulk User Update",
H3("Select Users to Update Status"),
form)
@rt("/update")
async def post(request):
# Same as before
# Return toast message with more detail
message = "✓ " + (f"Updated {updated} user{'s' if updated != 1 else ''}" if updated > 0 else "No users selected")
return Div(message, id="toast")
serve()
After making the above change I could see the message.
There was one final issue: it was updating all users regardless of their previous state. When I pointed this out to Claude, it modified the update logic appropriately.
Claude said “You’re absolutely right. We should only update users where the active state has actually changed. Let me modify the update logic.” 🙂
It changed the update endpoint to as shown below.
@rt("/update")
async def post(request):
# Get form data
form_data = await parse_form(request)
# Process each user
updated = 0
# Create a map of current active states
current_states = {user.email: user.active for user in users()}
# Get submitted states from form
submitted_states = {
key.split(":", 1)[1]: value == "1"
for key, value in form_data.items()
if key.startswith("active:")
}
# Only update users where state has changed
for email, new_state in submitted_states.items():
if email in current_states and current_states[email] != new_state:
users.update({"active": new_state}, email)
updated += 1
# Return toast message with more detail
message = "✓ " + (f"Updated {updated} user{'s' if updated != 1 else ''}" if updated > 0 else "No users selected")
return Div(message, id="toast")
After making this change I could run the application end to end . It is functionally equivalent to the HTMX example.
In this post I am making an assumption that Answer.ai team also wanted to achieve the same end outcome.
Conclusion
Can Claude do it in a single shot? No.
Working with Claude to complete the task took about an hour. As someone who doesn’t regularly use FastHTML and HTMX, it would have likely taken me longer to implement this independently.
I find value in learning from these mistakes and prefer the chat-driven approach to software development where you’re an active participant. While there’s merit to the agentic approach – an agent could have handled all the back-and-forth I did with Claude – these were relatively simple prompts focused on solving specific problems to move forward. It’s unclear why Devin was unable to follow these steps.
I plan to try other tasks and possibly write about those experiences as well.
Discover more from Shekhar Gulati
Subscribe to get the latest posts sent to your email.
