Update: All the individual day’s posts have been collected into this single post. All previous posts have been removed to reduce clutter on this site.
Few weeks back I was introduced to a coding challenge called #100DaysOfCode. All you have to do is work on some piece of coding project which is not your work project everyday for 100 days. Though I liked the idea I didn’t commit to taking up the challenge. I thought I could start working on a side project or an open source project whenever I wanted to.
But deep down I knew it was not true. Saying “I can work on a side project whenever I want to” is just like me saying to myself “I can lose weight whenever I want to”. It will never get done unless you start it. Remember the best time to plant a tree was 20 years ago and the second best time is today.
We all like to believe that we are rockstar programmers and we can sit through a 24 hour no-sleep coding session and build our next big thing. But in reality, Programming is a muscle that needs constant training and regular exercise to keep it in shape. It is a skill which will improve when you regularly do it and will degrade when you don’t do it.
So I decided to cut the BS and sit down and start the challenge today. The original blog post about the challenge says the detailed rules about the challenge. But since this is a personal challenge, I decide my own rules.
- Even though I prefer to release my code using an Open Source License, I will start the project as a private project and after it reaches some critical mass, I will decide to release it to the public.
- I will try to post everyday about the project and the status of the project here and/or a public issue tracker.
- Most of the projects will be very very small – some might be single page web apps or command line utilities. I don’t want to bite more than I can chew – at least in the initial stages of the challenge.
Day 1
Everyday at work, the first meeting (of the numerous ones) we have is the backend team’s daily standup meeting. This is an important meeting because this is the only time and place where we get to discuss what we would be working on to build/improve the main platform/product of the company.
We also discuss on what pending delivery items are there and what is in each team member’s todo list. There are other standup meetings about what we have for day’s delivery and that doesn’t exactly follow a template of a standup meeting.
Each member of the team gets to be that week’s scrum master or in other words, the person who collects all the discussed points and sends an email to the team and other delivery managers. This isn’t a strict scrum style standup meeting, but more of a “what are all my todo items today?” meeting. There are a few problems with this:
- Even though the minutes of the meeting is captured on email, it is not the proper way to log what was done and what is to be done today.
- Remembering to send the email to the developers in the To field and to the managers in the CC field, is hard.
- We don’t track the “done” and the “blocked on” parts properly.
But I hear you asking how all these can be easily captured if we use a proper agile issue tracker. But making the team use a proper issue tracker with all the agile features is a high friction task. Right now this 10 minute meeting works fine for the team and I wanted to capture and log the meeting better.
This is my first project for the 100 Days of Code challenge.
Nillu: A simple web application to log and email the daily standup meetings to everyone involved.
It is a simple web app which has a matrix of developers vs (done, todo, blocked) items. The scrum master fills in the details for each person and updates it. The details are stored in a DB and also emailed to everyone else on the team including managers.
Language and framework
I prefer to program using python and I decided to use Flask as the web framework for this. Since this is a maximum of one or two pages, I didn’t want a heavy weight Django framework. And for my personal web applications I prefer flask. I do want to develop it in python 3 as I want to move away from python 2.
I have built a basic schema for the DB with two models – User and Entry. The user will have a role – developer or non-developer. Only developers will have entries against their name. But the email will be sent to all users. Developers will be sent in the To field and the non-developers will be CC’ed. Similarly each entry could be one of the following: done, todo, blocking. And each entry has an user and date associated with it.
The last time I used flask was way back when it was in version 0.7 and currently in 0.12 lot of new things has come up. There is a separate flask command to run your app. You can add custom commands to the flask command line tool, which is useful for initialising the DB.
I am tracking the issues and features in gitlab. The code is private for now, but I might open source it later.
Lot of interesting changes and lot of new things to learn. Lets see what I can do tomorrow.
Day 2
Initially when I thought of building the Daily Standup Meeting Logger, I decided to give one single master password and whoever wants to access it, will just type the password to login.
- But whenever someone leaves the team, you have to reset the password.
- If the password is leaked, because someone decided to write it down on a post it, you have to reset the password.
- And giving different set of permissions for developers vs non-developers with a single password is hard.
Since I already have a users table with all types of users in the system, I decided to add a password field to it. I hate having to handle user’s authentication and would prefer to not have a password based authentication at all. But for the v0.1 milestone that I have set for Sunday (16 April), I decided to go with the traditional route of having a email/password combo.
Lot of people make mistakes when storing passwords. I don’t want to reinvent the wheel, so I just use bcrypt for hashing the passwords. Flask-Bcrypt is a very easy to use extension which handles salting and hashing the passwords.
Even though an user system is available, the only way to add new users is through the flask shell command line tool or insert queries in the DB directly. I think for v0.1, I can live without an UI for user management.
Second part of what I did today was to slap on a Bootstrap base template for the project so that the UI looks pretty. Bootstrap is my preferred way to get out of designing a good user interface.
Though the base template is put in place, I have integrated only the login template with bootstrap. I haven’t even written the views for displaying or updating the entries.
Tomorrow I will be concentrating on adding/updating/viewing the entries.
Day 3
Yesterday I said I would be spending all of today on getting the entries page done. I did work on the date parser part of the entries view function, but I didn’t fully implement the views function yet.
Date Parser
The idea is you can pass different types of dates in the URL and see/edit the entries for that date. For example: /entry/2017/04/15/ would show the entries for 15th of April, 2017. You can also goto /entry/today/ and /entry/yesterday/ as shortcuts for the corresponding days.
For tomorrow’s v0.1 milestone, I think I will implement just these types of dates. But for the next milestone (next week) I want to display all the entries for a month (/entry/2017/04/) and for an year (/entry/2017/). This makes it easy to generate reports for entire month or year.
Test cases
While the date parser was easy to implement, I spent the most of my time writing unit test cases.
And Remember Kids..
Writing code: 1hrs
Fixing bugs: 3hrsWriting code: 1hrs
Writing tests: 1hrs
Fixing bugs: 10mins— Mahdi Yusuf (@myusuf3) January 9, 2016
Even though I spent most of today writing test cases, it would help a lot when I refactor the code base. Test cases is something that I haven’t given much attention to in my projects at work. But I do know that there is a much higher probability of catching and fixing bugs if you have proper test cases.
One example of such a bug was in the date parser function I just wrote. I was doing a check to see if the date was a future date as I don’t want to edit the future entries before the day. The code was pretty simple.
if date_obj - today > 0:
raise FutureDateException("Trying to edit the future.")
The bug is that date_obj - today returns a timedelta object and you can’t really compare it with int. I wouldn’t have caught this bug till I finished the views function and manually entered different types of dates. But since I wrote a test case to cover this line I found out the bug and handled it properly.
Testing Web Apps
Usually testing web apps are not that easy and that is one major reason I hate writing test cases. But flask has few helpful ways to make it easy. But I wanted to test whether a URL redirects properly. For that I ended up using the Flask-Testing extension. If you are writing flask apps, I would suggest using Flask-Testing.
Continuous Integration
Since I use Gitlab for hosting my code, I ended up setting up a Pipeline to run tests automatically whenever I push any code. I use Gitlab-CI for it. It is a bit different compared to the Github+Travis combo I used to use earlier. But the advantage of Gitlab-CI is you can specify what docker image to use to run your builds/tests. And I always felt that even with my sparing builds, travis was overloaded and had lot of downtime. Gitlab-CI seems to be quite fast in spinning up jobs as soon as a git push is done.
Moving to PostgreSQL
I was using SQLite for my development and tests, but I knew I had to move to PostgreSQL soon as I will be deploying on it. I didn’t want any surprises when I made the switch. Using an ORM does abstract out most of the pieces, but I did get couple of surprises.
- bcrypt’s hash function returns a byte object and SQLite doesn’t have any problem in storing a byte in a text column. In fact SQLite wouldn’t even complain about most data types. But Postgres kept saying that I was trying to store more than varchar(120) even though the hash value was just 60 characters long. Then I remembered that for python 3, I had to do a .decode(‘utf-8’) to convert it into a str type. Weird error message, but quite easy to google and figure out.
- Postgresql has it’s own Enum datatype and whenever you create a enum type, you have to give a name for it. In SQLite there is no Enum data type and there was no error even if you didn’t give a name. But when I moved to postgres, I had to pass a name param to the Enum.All I had to do was
db.Enum('done', 'todo', 'blocking', name='entry_types')
Other than these two gotchas, there were no other issues in the database change.
Docker
Btw, I didn’t want to install postgresql on my Mac OS and ended up running a docker container for the DB. The commands to run the DB server was:
docker run --name nillu-pg -e POSTGRES_PASSWORD=foobar -p5432:5432 -d postgres
and the command to run the psql prompt was
docker run -it --rm --link nillu-pg:postgres postgres psql -h postgres -U postgres
For tomorrow, I definitely need to display the entry page and the edit screen for “today”. And I also need to send an email to the users. Those are the three things that are definitely needed for the app to be usable on Monday.
Day 4
Today was a slow day as I had to spend some time with family given most of the long weekend was spent on coding. But I did get to completing a working Entry page. Though not pretty, it conveys the information to the user.
Right now for each day it displays the entries for each user. If a particular day doesn’t have any entry, it redirects to the edit page for that date. The edit page is almost similar to the entry page, but with text areas instead of displaying the entry. (Btw, why is text area still so fugly?)
I did want to build the entry forms dynamically for each user in the system, but decided to do it in jinja templates instead. I would like to do it through WTForms so that I can do server side validations and add a consistent way to include the CSRF field.
For today’s milestone, I need to save the entry from the edit screen and have to send an email after saving the entry.
After that, for the next week’s milestone, I will put in extra checks like preventing users to not edit the history, displaying the entries using markdown, displaying entries for multiple dates, etc.
Update: I had set a milestone for v0.1 and I had to finish it. The last parts to be completed were editing and saving an edit for a day’s entry. And to send an email whenever a day’s entry was created. To send emails, I used the Flask-Mail extension which is as easy as plugging in a few configs and creating a Message object.
Bonus: The emails are also HTML emails so I have added basic tags to highlight specific words.
Day 5
Being a Monday, I was busy with my work projects. But I managed to put in a very simple feature. The ability to display a month or even an years’ entries in 1 page.
By going to /entry/2017/04 you can see all the entries from 2017-04-01 till 2017-04-30 (or till today if the current month isn’t complete yet. Similarly visiting /entry/2017/ will show all entries for the entire year.
For the month view, I need to find out how many days a particular month has. I can spend a few days to put in the right conditions to calculate this or I can use the calendarstandard library package in python. There is a function called monthrange which returns how many days a month has.
_, num_days_in_month = calendar.monthrange(year, month)
This one line is all it takes to do complex calendar arithmetic without any mistakes. With just this and a few more lines for handling year, I was able to generate SQLAlchemy queries to get all entries which is between two dates.
And since I had refactored all the processing of the rows to separate functions, the change took practically 5 minutes to code. Remember kids, write modular code. Separate out your code into methods and functions.
I will be travelling for most part of tomorrow, so don’t know what I will be able to finish tomorrow. I also need to deploy the app on Heroku maybe?
Day 6
Today I was too tired after driving for 12 hours. I couldn’t think too much and ended up cleaning up the templates. Removed background colours and made the screen a tad bit cleaner to see.
I also put in a logout button and abstracted out common template pieces to a base template file.
Tomorrow I will probably add in views for adding/editing users. And I am still thinking whether to add an edit screen for entries. Should we be allowed to edit history or not?
Day 7
Today I am working on views to add an user. A very simple form which just collects the user’s Name, Email, password and role. Checks if the email address already exists. If not creates an entry.
I also wanted to display the user details after saving. But I think I wouldn’t have time to do it. Instead I want to concentrate on deploying a version on Heroku tonight. If I get it deployed before tomorrow’s standup meeting, we can use this to send the emails instead of manually composing it in gmail.
Day 8
Today was an important milestone – I finally deployed the app to Heroku and today’s standup meeting was logged using the app. Emails went out to both developers and non-developers and it was beautiful to see the app working so smoothly.
But of course the deployment didn’t go without hiccups. I was trying to deploy it using gunicorn and by following the official flask documentation on deploying to gunicorn, my Procfile was supposed to be
web: gunicorn nillu:app
But even after many combinations I couldn’t get it to work as gunicorn kept complaining about not able to import nillu.app.
Finally I had to create a wsgi.py file which had these lines:
from nillu import app as application
if __name__ == '__main__':
application.run()
And my Procfile had just
web: gunicorn wsgi
This is the combination that worked.
When I received the emails today morning, I noticed that the user names were sorted in a different order than the /entry/ page. I wanted to maintain an alphabetical order for the names and managed to do it in the templates by sending a user order variable.
For the emails I used the default random ordering of the dict which caused this inconsistency. I had to pass the same user order to the email template generation program and the bug was fixed.
But I did find a few more bugs like if I don’t fill in any entry for a user, it still creates three empty entry rows. This should not happen. And the email should also not include users with empty entries. I will try to fix it later maybe.
But tomorrow I do want to beautify the entries a little bit by allowing markdown text (a subset of markdown).
Day 9
Working late at night after office work means, I can’t put in my best effort on this challenge. So I wanted to change my schedule into coding in the morning as soon as I wake up. But yesterday, after writing the post, I continued and worked on supporting markdown for the entry text and adding Jinja support for email templates.
Flask-Markdown and Bleach
Flask-Markdown does a pretty easy job of integrating markdown in your flask app. But I wanted to support only a subset of the tags, like ordered and unordered list, bold, italics, etc. I don’t want a h1 tag included in an entry.
To strip out such tags and allow only a specified list of tags, there is a python package called bleach. You specify which tags are allowed and it encodes it into HTML entities or even strips out those tags.
So I had to pass the entry’s text through a markdown filter and then through a bleach filter. I also used Markdown package’s nl2br extension instead of my custom template filter. Less number of code means less bugs.
Jinja Email Templates
For the emails, I was using normal python formatted strings. I was constructing the string using complex loops and lists. I wanted to use Jinja templates as I already use that for the HTML pages. Replacing with Jinja means there is a single template string/file and I can store it in a config table in a database. For now I have it embedded in the views.py file. But later I will move it to a file or DB.
Beautified Text Area
HTML Text areas are always an eyesore. The width and height are fixed and doesn’t adjust based on the window width. Bootstrap makes it easy to beautify it. All I had to do was add a form-control class to the textarea.
Fixing Bugs Live
Today during the call, I typed out all the text and pressed the save button waiting for a beautiful email in my inbox. Instead I saw an error screen and couldn’t understand why. All my changes were simple and not in the actual business logic. Why an error suddenly.
Remember how I was trying to beautify the textarea? I added one more feature where I set a jinja variable so that I can reuse it for the name of the textarea and the label tag. The way I constructed the variable was wrong and it caused all the text areas to not have a name.
I made two mistakes:
- combine two fixes in one commit. One commit should always be for fixing 1 bug/issue. Don’t combine multiple fixes into one commit. Hard to debug and hard to revert back too.
- Didn’t have test cases for saving entries via the HTML templates. I had basic test cases which tested the views, but no test cases to add a day’s entry via the HTML templates.
So I had to rush and fix the code during the call and it was slightly panicking.
This is a lesson that I think will have to be etched into my brain (and many other developers’ brains).
BTW, I will be travelling for the next 3 days and may not have a good internet connection. I will have to try to commit something and try to finish the milestone over the weekend.
Day 10
Today I am in a place that makes me feel that my old 28kbps dial-up modem is a super-fast fibre internet. This hotel has so many power cuts, Air Conditioner compressor heating issues, hot spicy food that causes abdominal heating issues, etc. I don’t mind any of those personally, except bad internet. Even hotspot is spotty with just Edge connection. I can’t believe I am paying so much for this shit.
Trying to look at some documentation? Sorry.
Want to access the issue tracker? Fat chance.
Trying to commit and push some code? No way in hell.
Want to debug a live production issue? Why don’t you go back to civilization and try again?
But I did get some work done. I finished the view function to add a user. I also finished views to display a list of all users and details about a single user.
Right now the details are very sparse, but I will be displaying a user’s done/todo/blocking tasks for the past N days in this screen. That will be useful to see how a developer has been working.
But for today, this is all I could complete and push.
Day 11
I think I might restrict edits to only today’s entries instead of being able to edit any arbitrary day’s entries. But that is for a future release.
But for milestone v0.2, I would have to probably remove couple of issues and mark the milestone as complete. Next week, when I go back to mainland, I will add more features.
Day 12
Thinking of the next important feature that would be asked to make others to use the app is the Password Change screen. Right now, we have 8 users on the app and it was I who set everyone’s password. But soon they would want to change it to something more secure.
So I quickly wrote a view function which will display a password change form and will also change it if the password and confirm_password fields match. It also logs the user out on successful password change. All of this was made easy because of Flask-Login.
Tomorrow I will be back to my home and would have a much better internet and more time to program.
Day 13
Today I spent most of the time refactoring the entry view function. Moving the save functionality into it’s own separate function.
This refactoring is needed because I have to add a way to get all entries that were added between two dates. The entry view function is becoming too big and I definitely need a cleanup for implementing this new feature better.
I have been thinking hard about how best to create the URL for this and since URL. Currently the URL structure is something like /entry/YYYY/MM/DD/ and you can remove the DD and MM part to show entries for a month and an year respectively. But I want the user to be able to see all entries from a date till another date.
Option 1:
Use an URL like /entry/YYYY/MM/DD/to/YYYY/MM/DD/. While this does make it easy to remember and type, it does make writing code to handle all the different combinations hard. What is the user wants to put in just /entry/YYYY/MM/to/YYYY/MM ? And what if he gives date in from and no date in the to section?
Option 2:
Accept the from and to as GET params – like /entry/?from=YYYY-MM-DD&to=YYYY-MM-DD. This makes it easier to code and will introduce a lot lesser bugs. But the downside is bad URL design.
These are the two options I can think of and I think I will go with the second option for it’s ease of implementation.
Day 14
Today I completed the custom date ranges that I talked about in yesterday’s post. Instead of just displaying a day/month/year, you can choose a from and a to date range and all entries that were created in that range will be displayed.
I ended up using GET parameter to get the date ranges. You have to form the URL as /entry/?from=2017-04-01&to=2017-04-26 to display all entries from 1st of April till 26th of April (both inclusive).
Day 15
Since I try to make most of the interaction with the web app through the URL, I wanted to add an easier way to get the last N entries. I added a new view function which displays the last N entries.
The user has to visit /entry/last/N/ to display the N days’ entries. This is in fact quite simple to implement after I have implemented the custom date range filters yesterday. I just calculate the from and to date based on the last N dates and I redirect the user to the right filters.
The second feature I added was to send an email to all the users based on the filter you have selected. Previously the app used to send emails only when a new day’s entry was created.
Now at the end of every entry page, be it a single day’s entry or a month or a year or any custom date ranges, there is an Email button which will send the report to all the subscribed users in the system.
For now the implementation is simple as I reuse the email function and I pass all the entries for the given query.
I might change it later to not email all users and instead to let the user specify an email address.
I do want to clean up the templates and fix some timezone related bugs, But I think the app is in a stable state that I will not be concentrating on this daily. Instead I will be working on a new project that I want to build using Golang – so that I can start using golang as my primary programming language.
Tomorrow I will be doing a few more tests, some basic cleanup, deploying of latest code to heroku and making the code repository public. From Saturday, I will be working on my golang project.
Day 16
Yesterday when I added the feature to email users multiple day’s entries, there was one small bug that I missed. The subject like still showed the first entry’s date. I fixed it by making sure that if we are emailing a single day’s entry, it showed one date and if we are emailing more than a day’s entry, it will show the from and to dates.
hOther than that bug fix and a few docstrings, I deployed all the changes from past 3-4 days to heroku. Even though the changes are simple, I didn’t want to debug a silly error when someone is editing the entries in the morning.
Another major change in the project is making it a public repository. Previously the code was private, but since the code is now decent enough and gets the work done, I am making it public. Code reviews, pull requests are welcome. I usually prefer BSD 3 clause license for such kind of small scripts or web apps and I am releasing it under the same license.
Now that it is stable enough for daily use, I will not be adding new features to this daily. I will be working on it when I get some free time or want to improve the design. But what will I do for the rest of the 84 days of the challenge?
The whole point of the challenge is to push yourself to learn new stuff and me building yet another web app is too un-challenging (if thats even a word). So I decided to concentrate on sharpening my golang and start building some tools or services using it.
What changes? Nothing! I will have to brush up on my golang skills and decide on a simple project to work on. I will be logging my progress in evernote instead of posting about it daily in this blog. I do want to make sure to post here at least twice a week with detailed progress. But my daily public progress tracker will be on twitter (@cnu).
Day 17
It has been a while since my last #100DaysOfCode update. I haven’t been exactly working every day on these challenges, but I did some work towards it for the past 10 non-consecutive days.
First to give some context on the project I had originally chosen to work on – a key-value store. Wow! So original, right? But I did want to add a multi-layered backend that store the actual data.
So I ended up looking at the different key value stores (especially those that were written in Go). I read through lot of source code, papers, blog posts and understood how different stores are implemented. I think I need some more time to build the KV store I have in mind.
The only real coding (other than work related code) I did was a PR to BradFitz’ groupcache project.
So even though I was reading and learning stuff, I am going to count the past 10 days as just one day in my 100 Days of Code challenge, because as the name suggests, it is a code challenge.
Two lessons that I learnt with this big break:
- Momentum. In whatever projects that you choose, always maintain the momentum. Keep churning code and making small improvements. Once the habit is formed, it becomes pretty easy.
- Small > Big. Begin with smaller projects than big grand projects. Even a small one-line change that you do today is much better than the 1000 line commit that you aspire to write some day in the future. Live in the present and work towards the next day.
With these learnings, the project I am going to work on is still a Key-Value store. But this is very simple no-frills store. It will work on memcache protocol (or a subset) and would be able to store the data to a local disk. No clusters, no multiple layers, none of that sorts.
I know there is memcached (the disk backed memcache store), but this is a way for me to get started on golang. I think implementing a key-value store is the “hello world” equivalent of learning a new language.
In tomorrow’s post I will be explaining more about the store, the need for it and the backend I will be using.
Due to various reasons (my laziness being the primary), I couldn’t continue this. I might restart this challenge or a similar challenge later.