Category Archives: Development

Building an Offline Program with Online Syncing

As I become more and more experienced with software development, I am very quickly discovering the limitations of the web. Having a desktop application that constantly needs to retrieve large amounts of data from the Internet is very, very slow. Sometimes, when network speeds are spotty, a simple interaction can take up to 30 seconds, which is definitely unacceptable for the end-user.

In order to solve this problem, I decided to restrict myself from looking at online solutions. I wanted to create a solution on my own.

First, I wanted to create a robust system for managing a database for an offline program. I looked into several different services, including Microsoft’s SQL Stack and a local MongoDB server. However, I found that most of these solutions were either too difficult to learn (I was used to setting up a MySQL server using cPanel and phpMyAdmin) or that they over-complicated the issue. Then, I came across SQLite, which happened to be exactly what I was looking for. It was a lightweight, speedy, local replica of MySQL. After spending about an hour reading the documentation, I realized that it was also super easy to setup. I began working on a library that would make working with a local SQLite database super easy.

Thus, SQLiteDatabaseManager was born. The project was meant to house functionality that would be used in all of my projects, and here are some of the key features that were packed into it:

  • The ability to quickly create a local database
  • The ability to quickly insert, delete, and update records knowing the table names, the fields, and the new values.
  • The ability to have a small table-verification system that allowed the user to define table schemas and check the existing SQLite tables against these schemas for integrity verification.
  • The ability to retry operations when they fail (the fail could be a result of a spontaneous locked file and a retry could fix everything)

SQLiteDatabaseManager worked well because all you needed to do was create a new instance of it, and everything worked out of the box. A new SQLite database was setup and managed by the library. After rigorous testing, this worked perfectly. Of course, the library still needs major work and many more features, but it does what it was intended to do at the moment. My offline data management system worked.

Now the problem was not just managing data offline, but getting it to sync to an online counter-part when a network was available (In a perfect world, all operations would be done on the local SQLite database (fast) and then they would be replicated on the duplicate remote MySQL database (slower) on a background thread). The idea was simple.
Local SQL and Online SQLThe SQLite database and the MySQL database would communicate with eachother via a simple REST API. The databases would be identical and the ID numbers created by the SQLite database would simply be moved to the MySQL database.

 

Creating the REST API for the remote MySQL portion of this project was rather simple. Relying on the methods of the LAMP stack, I used a set of PHP files that would receive GET, POST and DELETE requests so that data could be managed and manipulated fairly easily. Using PHP’s json_encode() function also proved very useful for creating readable responses.

Thus, with the creation of a working SQLite local database and a working and accessible remote MySQL database, I went ahead with the first iteration of implementing this idea. So far, I have run into a few problems.

 

An easily-solvable problem arose when thinking about how to handle many, many clients. Since there was only one remote MySQL database and many, many instances of the local SQLite database, it was a pain tracking when the local client’s changes actually had to be applied or if the remote version was actually newer. This problem was solved by creating a dateAltered field in both the remote and local databases that would contain the last date of modification of the data that it pertained to. Thus, the data with the latest dateAltered value was put permanently into the remote MySQL database.

But what about when there are many, many clients creating information. Surely, some of the client’s SQLite databases will assign ID numbers that coincide with other clients’ SQLite databases. Thus, when information is uploaded to the remote server, different pieces of information may have the same ID numbers. But since all data is modifiable, how is this detectable? This is a problem that I have not yet worked out a solution to since the first implementation of this design has a one-client one-server relationship. However, at the moment, I am hypothesizing that each data-entry contain a unique hash of its original state. Thus, database operations can be performed by referencing the ID of the element, but conflict-resolutions will use this hashed code. Thus, we will always be able to track if a certain piece of information has been modified from its original form or not. Thus, when uploading to the remote server, the server will manage some conflict resolutions and return the ID number of the object. The local clients will then reassign the ID number of the object to that ID number in order to match with the remote MySQL database.

 

I will continue to work on this project in the coming weeks, and I plan on making some detailed blog posts about it, so stay tuned! Eventually, I hope to have this process wrapped up in an easy-to-install and easy-to-implement library. We will see how that turns out.

 

Home Automation

If you have seen the latest addition to my code page, it is my first project that deals with hardware other than standard computers. It deals with the Chromecast. Take a look at the project here.

One of the goals of this project is to make life with roommates easier. In my opinion, it’s the modern alternative to keeping a grocery list pinned to the fridge. I want to make an app in which everyone can add tasks to a task list and everyone can see them just as easily as they appear on the television screen.

This, I believe, is my first step into the realm of “home automation”. Ever since I started toying around with making the tv screen display my things when I want them displayed, I have begun thinking about the possibilities of home automation. One of the things that I would really like to happen is a system in which I tap my phone on an NFC tag and everything turns on. The TV turns on and displays tasks that I need to complete. My computer turns on and becomes ready to use. Coffee is brewed. I imagine a place where I am truly the master of the house.

As I gain more and more knowledge in software development, this is exactly what I will be experimenting with. Right now, I am working on getting my Chromecast to display a task list that is editable from anywhere. The concept is really simple and pretty straightforward, but not knowing most of the technologies behind it makes it challenging. Basically, the following process needs to happen:

“Launcher App” –> Chromecast App –> Node.js App –> Data from MongoDB

This launcher app needs to be launched manually. However, as long as the launcher app exists as a mobile app, I can make it launch when the phone detects a specific NFC tag. Adding tasks is a little more straight forward.

AngularJS –> ExpressJS –> Node.js –> MongoDB

Also known as the MEAN stack, this is the process that I plan to use to power the backend of the task management. As the project progresses, hopefully I will be smart enough to make it into a deployable package so that anyone with a Chromecast and roommates has the ability to setup a communal living room task list.

During the development of this small system, I’ve been listening to a lot of older songs, especially those featured on Radio New Vegas. Here’s one of my favorites:

Visualizing my Operating System Use

In my work, home, and play life, I constantly switch operating systems. If I want to play games, I use Windows. If I want to get some hardcore development done, I use Linux. If I want to casually browse the internet, I use my mac. But the question remains: How often do I use each operating system?

The Tool

Being a software developer, I thought that the best approach to answering this question would be to create a tool to do it for me. I would call it OSTracker (Short for Operating Systems Tracker – An extremely advanced acronym that definitely required this parenthetical explanation). Originally, the plan was to have a service that could run on any operating system that tracked how long I was on the computer. These metrics would then be uploaded to a centralized remote repository and would be parsed and analyzed by some sort of scripting language, that would then display the results in a semi-pretty format.

In order to create something that could easily be run on any operating system that I would pickup, I thought of creating the service in Java. The service could then be called with the following calling conventions:

java ostracker <login/logoff>

This would be extremely useful because this would also allow me to easily port the project from Linux to Windows, Android, or any other modern operating system that I may come across.

The Unfortunate Lack of Direction

Unfortunately, my laziness got the best of me. After all, I did not need to worry about having a constantly running service, as all I needed to keep track of was the name of the operating system, the time of the event, and whether or not the event was a log off or a log on. Since all I needed to do was push these metrics once, I decided that it would be better if everything was handled on the remote repository itself.

Thus, I created a PHP Servlet that would process all events. The servlet is pretty simple, it essentially accepts the aforementioned details as parameters and plops them into a file on the server if they should be there.

How does the servlet know if the information should be there? Well, if the previously tracked action is one that makes sense in relation to the new action. For instance,

  • You just logged off Windows, your last action was logging off Windows. This pair of events doesn’t really make sense.
  • You just logged off Windows, your last action was logging on Windows. This pair of events makes perfect sense.
  • You just logged onto Linux, your last action was logging off of Windows. Again, this makes perfect sense. I just switched operating systems.

Now, I wanted to give the system the ability to track when I was on many operating systems at the same time (assuming multiple computers, I suppose), so the servlet handles that case well.

  • You just logged onto Linux, your last action was logging onto Windows. No problem.

The datafile is arranged in a format of <operatingSystem><time><action> and different events are delineated with a newline character. Thus, the PHP file just has to use all the information from the last event in order to determine if the next event is able to be processed. If so, a 1 is returned. If not, a 0 is returned. Here’s the code:

$actions = explode("\n", file_get_contents($filename)); 
$actions_count = count($actions) - 2; 
$lastAction = $actions[$actions_count]; 
 
if (strstr($lastAction, $os) == FALSE || strstr($lastAction, $action) == FALSE)
{ 
  $resp = file_put_contents($filename, "$os$time$logaction\n");
  echo 1; 
} 
else 
{ 
  echo 0; 
}

As you can see, it’s extremely simple.

The Client-Side

Linux

Now that I had something that was running on the server, I needed something to actually call the PHP file in order to push the metrics onto the server. Again, I considered writing this in Java as it would provide me with the ease of portability. However, laziness got the best of me. I decided to write proprietary scripts.

For UNIX, I wrote a bash script. The script is extremely simple. In order to account for a possible lack of internet connection, I decided to essentially “spam” the script until I got a proper response from the server. The PHP file is navigated to using the curl utility (in quiet mode, so that I don’t notice any spam whatsoever).

Again, nothing too hefty at all. All of the meat is handled by the curl utility.

#!/bin/bash

time=`date +%s`
action=$1

response=`curl -s# ${url}?system=${operatingsystem}&time=${time}&action=${action}
while [[ ${response} != 1 ]]; do
        response=`curl -s# ${url}system=${operatingsystem}&time=${time}&action=${action}
        if [[ ${resonse} == 0 ]]; then
                exit 0
        fi
        printf ${response}
done
exit 0

Windows

On Windows, it is essentially the same story. In order to recover from a lack of internet connection, I decided to plop the whole thing in a while loop so that it ran until it found a response. Not sure whether or not having a broken pipe would throw an exception, I also wrapped the whole thing in a try/catch that when caught simply looped back to the beginning of the method.

All of this was implemented in C#.Net and uses a WebClient to do the heavylifting and post the data to the server. The code is almost identical to that of the BASH script:

class Program
{
 static void Main(string[] args)
 {
 
 try
 {
 string action = "logout";
 WebClient wc = new WebClient();
 string time = ConvertToUnixTimestamp(DateTime.Now).ToString();
 string resp = wc.DownloadString(url + "?system=windows&time=" + time + "&action=" + action);
 while (int.Parse(resp) != 1)
 resp = wc.DownloadString(url + "?system=windows&time=" + time + "&action=" + action);
 }
 catch (Exception e)
 {
 Main(args);
 }
 
}

public static double ConvertToUnixTimestamp(DateTime date)
{
 DateTime origin = new DateTime(1970, 1, 1, 0, 0, 0, 0);
 TimeSpan diff = date.ToUniversalTime() - origin;
 return Math.Floor(diff.TotalSeconds);
}
}

Getting the Scripts To Run

Getting the scripts to run at the right time was probably the most challenging part. I didn’t have to worry about running the scripts many, many times since the PHP servlet took care of any situations that didn’t make sense (If we logged in twice, per se); however, automatically executing the scripts when the computer is turned on or off is a daunting task.

Linux

In order to get this to work on Linux, I called the script from my /etc/rc.local file, which launched the script on the computer’s start with no problems whatsoever.

Shutdown was a different story, however. Since during shutdown the network adapter is shut down very early, I had to make sure that my script would run before anything else in the shutdown process. I learned that in order to get something to run on shutdown, there is a small process to go through:

1) Copy your script to /etc/init.d
2) Make a symlink of it in /etc/rc0.d
3) Make a symlink of it in /etc/rc6.d

Then, within rc?.d, the symlinks are actually executed in ASCIIbetical order, so in order to make sure that my scripts executed first, I made sure to insert ‘A01’ at the beginning of the symlink names. I think that did the trick, since everything works properly!

Windows

Getting things to automatically start in Windows is a bit trickier. There were some suggestions on the internet to use the Group Policies Editor in order to add my programs as Startup/Shutdown scripts; however, these did not work for me for whatever reason.

To solve the startup issue, I made a shortcut to my script’s executable and plopped it into the startup folder in order to make sure that it got launched when Windows started.

Shutdown, however, is a bit more complicated. Since there is no way that  I know of to control Windows shutdown process, I decided to make a simple wrapper script for shutting down my computer. Instead of simply shutting it down, it calls my OSTracker script and then tells the computer to shut down. A easy, two line, batch script. The only caveat? It’s sitting on my desktop and I need to be sure to double click it whenever I need to shutdown my Windows machine. Shouldn’t be too hard to remember!

The Display

Now comes the fun part. How do I display this information to the public? I decided to go with a line graph. Although it’s probably not the best choice, it looks cool, and with a ‘1’ representing logged in and a ‘0’ representing logged off, it’s definitely easy to decipher a line graph like such.

In order to implement the graph, I used Chart.js. As a bonus, Chart comes with some pretty cool animations, so it made my page look just a tad nicer than I had originally intended.

Basically, the JavaScript page makes use of regular expressions to find the operating system, time, and action taken on each line of the statistics file. Nothing too interesting, so I’ll just skip the ramble.

The Product

As you could probably deduce if you actually read any of my rambling, this system is extremely hacked together. Nothing was really planned out and it took me about 6 hours to complete. However, this is my first spur-of-the-moment project in quite some time. It feels great.

Here are the results: http://brandonio21.com/OSTrackerStats/

A Rant About Startup Growth

A bit of background

I very recently began working at a software development company that has grown very, very fast. Its profits have grown tremendously, its employee count has grown, its project count has grown, and even the amount of buildings it owns has grown. From the outsiders perspective, this is a great thing. When talking to those that are in charge of the business side of things (sales, management, hiring, etc..), everything is going just as according to plan; however, the tech side of the company definitely has a different story to tell. The company has grown so rapidly, in fact, that the technology and routines of the company cannot keep up with the ever-changing environment that the company lives in.

A Quick Example

Take, for instance, version control. This is probably my biggest gripe about how the company operates. The company consists of many, many teams doing many, many different things. None of these teams can agree on a single method of version control. Each team does something different with their own little twist. Many teams use an internal clone of GitHub, Many others use Perforce. This doesn’t seem so bad at first, however, with each side of version control comes its own integration of code review, submittal processes, and checkin/checkout procedures. There have been several occasions where I needed to grab code from another team that was in GitHub and attempt to sync it with my code that is stored in Perforce. It’s a logistical nightmare.

It doesn’t stop there, though. Each method of version control has its own story. I won’t go too far in depth, but let’s just say that the goal of the company was once something along the lines of “We need to push code that works“. Now, the company is leaning more towards a philosophy of “We need to push code that works and is efficient, modular, and understandable“. This makes fixing bugs on older products or adding features to older products a logistical nightmare.

A Resolution?

One of the problems that I have found with Perforce is that its local repository management is more than complicated. There have been several times where I quickly want to branch a directory full of code so that I can play around with a  possible new feature, but be able to quickly switch back to the main devline of code. With Perforce, this is much more complicated than a simple git checkout -b feature.

So, in order to solve this problem, I have resorted to keeping all of my copies of the company’s source code within my own local git repositories. I then sync all of the Perforce checkouts into a branch and then merge that branch into all of my feature branches, resolving all conflicts along the way.

The Message

Although this post seems to have gone nowhere at all and made no constructive points whatsoever (that was definitely not the purpose of the post), it did point out something that I never expected. The internal infrastructure of large companies can be a serious mess. This being my first position at a large software development company, I honestly did not expect things to be as messy as my code directory on my home computer. But it’s also much, much worse. I believe that the problem arose from the fact that the company simply expanded very, very fast and that the internal tech employees were unable to adapt to newer tools quickly enough.

What does this mean? It means that now developers are stuck at a crossroads of bad naming conventions and commented-out blocks of code. They’re forced to use project-specific version control and they’re constantly deciding on which of the hundreds of branches they should push their code to.

Honestly, in situations like this, I think it is best for the company to stop working on new features for a small while, and fix their current infrastructure disaster.

I Have Finally Discovered the Meaning of Cookies

Everyone that browses the web has heard the term cookies thrown around every once in a while. Whether it is originally defined as data, evil little programs that hackers use, annoying things that cause popups, everyone has heard of cookies.

I, too, was once one of the people that thought they knew what cookies were. I thought that it was data that the WebBrowser stored to keep track of things. For example, remembered usernames and passwords and addresses were stored in cookies. Cookies were my friend and sometimes I needed to clear them in order to get a website to function properly.

I was only half-right, however. The other day I was assigned the task of making a JavaScript page that automatically refreshed itself every 5 seconds. Easy enough. I created the HTML layout, added in a few dynamic Java calls to get some data, and then added in the JavaScript method that would refresh the page. For reference, the method looked a little something like this:

function StartTime(){
        setTimeout("RefreshPage()",5000);
}
    
function RefreshPage(){
        if(document.Refresh.auto.checked)
        window.location.reload();
}

 

All was well and the page functioned as it should. That is, until I needed to add a toggle switch that would hide or display a <div> HTML tag. That’s when things got really tricky. The problem was not building the toggle button or the JavaScript function that changed the style=display property of the tag, but rather, after making this, I realized that every time the page was refreshed, the toggle would be flipped again.

So how could the webpage memorize the user’s choice on whether or not the toggle should be flipped? My initial thought was to make something like a global boolean variable that kept track of the state. Something like so:

<%! boolean hideDiv = false; %>

Then the thought occurred to me. What if many, many users are visiting the page at one time and they are all toggling this variable? Since the page is being rendered on the server, a toggling of the variable means that all users are affected by any toggle that one user makes. So how could I store the user’s preference in their browser locally without affecting any other user?

A-ha! So that’s what cookies are for! After some research, a website’s cookies are essentially stored in a semi-colon delineated string that contains many key-value pairs. For example, a cookie string could look like:

fruit=apples;vegetable=corn

This information is stored locally by the user and is stored specifically for the website that stores the cookies. The website can also get the cookies as well. So, in order to solve my problem, I created a cookie that tracked whether or not the div needed to be displayed. Something like:

displayDiv=false

This cookie was then saved every time the user pressed the toggle button (with its respective value, of course) and was retrieved every time the page needed to refresh.

Cookies are surprisingly fast, too. I expected the page load times to slow down significantly since data was being retrieved from the disk, but it didn’t seem to have much slowdown. Of course, this could be because we are running machines with huge amounts of memory and processing power and all we are trying to retrieve is a few bytes of text (don’t quote me on that — that’s not the exact filesize whatsoever).

So, until very, very recently I did not know exactly what cookies were. Hopefully this helps you discover their true meaning. In short,

cookies are a set of key-value pairs, specific to a website, that are saved and managed by the user’s browser. They come in a semi-colon delineated list and have the ability to expire after a certain amount of time. They are used to store data locally that is usually specific to the user.

For those that are curious, my implementation of saving and getting cookies looked a little something like this:

function setCookie(cname, cvalue) {
    var d = new Date();
    d.setTime(d.getTime() + (50*24*60*60*1000));
    var expires = "expires="+d.toGMTString();
    document.cookie = cname + "=" + cvalue + "; " + expires;
  }

  function getCookie(cname) {
    var name = cname + "=";
    var ca = document.cookie.split(';');
    for (var i=0; i<ca.length; i++) {
      var c = ca[i];
      while (c.charAt(0)==' ') c = c.substring(1);
      if (c.indexOf(name) != -1) 
      {
        if (c.substring(name.length, c.length).contains("="))
          return c.substring(name.length, c.length).split("=")[1]
        else
          return c.substring(name.length, c.length)
      }
    }
    return "";
  }

Note that by default, my setCookie method sets the cookie to expire after 50 days.