This post has been brewing for a while over here at the Bitlancer headquarters. Over time, we’ve begun to notice patterns among the requests we receive from clients and prospective clients alike. More often than not, we’re seeing the same questions being asked and we’re recommending/implementing the same solutions.
Originally, we thought this might be a simple side effect of rapid development and various fastest to market strategies. If everyone is following similar methodologies, of course we’ll see repetitive bottlenecks, roadblocks, and challenges. While we still believe that’s part of the cause, we also believe that there are steps start-ups can take early on to make their lives easier down the road, allowing them to save money and avoid wasting time.
Without further ado, we bring you the top 10 technical things start-ups could do early on (in under an hour!) to save themselves time and money down the road:
- Backups: This one is simple, but often low on priority lists. Before implementing other suggestions, ensure that you can recover from hardware failure, software failure, human error, a security breach, or a malicious former employee. If you only have an hour, setup a script that automatically copies your work every hour to Rackspace Cloud Files/Amazon S3 or commits to git/subversion. If you use relational databases like MySQL, please remember that even if you have a slave, you’re only protected from two of the five possibilities above.
- Define System Standards and Automate from the get-go: We realize you need to move fast in the beginning. Time is critical. But basic automation is easy. While jump start scripts work well, they don’t easily allow updates to infrastructure after-the-fact. The fix? Take a few hours and implement puppet or chef, even if in a not-so-efficient way, right from the beginning. It may not be clean, and it may not be perfect, but it will be a step in the right direction that will at least define some loose standards.
- Database Standards: In less than 10 minutes, you can define database standards that will undoubtedly improve developer efficiency AND reduce clutter. Pick a solid user prefix, and stick to it. Name your tables a certain way. Put a prefix in front of indexes if you so choose. If you’re going to give application users ALL privileges, at least limit it to a specific database. Documenting this procedure in a shared document or wiki wouldn’t be very difficult.
- Dedicated Usernames: It is very common in startup environment to see shared usernames: developers have the tendency to want to move fast, so they spin up and access servers with shared root logins. This notion can be applied to databases, too: instead of using individual application accounts, engineers may use shared database credentials to simplify the number of configuration files and accounts that need to be managed. As time goes on, and the number of applications increase, implementing unique usernames per user or application becomes exceedingly difficult. Therefore, it’s important to define a simple, consistent username standard (perhaps the first initial of one’s first name, all of their last name, or one’s full name) from the beginning and stick with it. For database users, consider “app_” for application users and “sys_” for system/process users. Demonstrating this sort of simple consistency from the beginning will save many headaches down the road.
- Always Assume Two+ Servers: We see this all the time. Early-start developers build their application as if the database, web servers, caching tools, etc. will always be on the same server. This can be true in the beginning, but the idea doesn’t scale! Eventually, if your application grows, you will have dedicated database instances, webserver instances, and caching instances.. At that point, the most common method of accessing resources will be via the network, so it is important to build your applications using the “network” by default. Avoid accessing the file system directly from your applications. Avoid storing files on the file system that need to be used by the application - you may have more than one webserver down the road! Store these things in distributed databases or object stores like Rackspace Cloud Files or Amazon S3 that can easily be accessed via a network protocol like HTTP. Also avoid developing for vertical scalability! It’s important to think horizontal from the beginning.
- Simple Security: You don’t have to think about security as a #1 priority from the get-go. If you just started a company, obviously you want to move fast. However, it’s important to keep in mind that some security is easy and does NOT always “slow things down.” Some basic thoughts are to: set a root password on your MySQL servers if one isn’t set by default, which is easy to do and will avoid bad habits (like always logging in as root because it’s the only user that has no password set); Use iptables or security groups (if your cloud provider or cloud stack supports them); and put some basic rules into play, such as only allowing ports 80 or 443 (etc.) on your webservers - you’ll thank yourself later!
- Clean Up Process: You’re going to have logs that eat up your server disk space and cause outages. Indexes inside your database hosts are going to become fragmented and unoptimized. Tables will need to be cleaned up from time to time. Automating all of this from the beginning may be hard, and we definitely don’t suggest you spend a ton of time on this from the beginning - after all, this entire list is only supposed to take one hour per item! But it’s important to tackle the low hanging fruit when it comes to clean up: implement something like logrotate or newsyslog in the very beginning, so there’s at least an easy way accessible to rotate logs as needed. To save costs, write a cronjob that actually deletes old backups, snapshots, and unused cloud instances, for example. It’s easy to enable nightly backups, but don’t forget they need to be cleaned up and rotated, too.
- Monitoring and Trending Systems: If you visit a farm in the middle of nowhere, it is likely that you will notice a strong odor. After a day or two, you won’t notice the smell for a while, and eventually, you won’t even remember it smells to begin with. This process is known as desensitization, and can happen with monitoring and alerting just as easily. If you send too much email, the messages will get filtered. If you have too many false alarm alerts, no one will react anymore. If you have too many fire drills, no one will think twice when the real fire hits. You get the picture. Monitor error logs and basic server metrics from the start. Keep business rules and metrics out of your server and basic service monitoring. Make sure every alert you send out is actionable. If alerts aren’t actionable, get rid of them, or simply log them for later analysis. And no matter how “devopsy” you are, be sure to have some sort of on-call or ownership system in place.
- Avoid Technology Politics: There are three types of people in start-ups: those who want to use a technology because it’s the latest and greatest, those who want to use the latest technology because it’s the right tool for the job, and those who are indifferent. Be on the lookout for all three types and avoid technology politics. Use the best tool for the job, and remember that just because there’s a member of the team who may be able to PROVE a technology is the correct choice, if no one else knows how to use it long-term, and that member leaves the company, it could end up causing headaches for others. Avoid using technology just because it’s the latest and greatest buzzword. No matter what technologies you choose to use, be sure to use them in the correct way.